Preference Rating and the case for subjective preference

[richard12511]

There is an issue that is not addressed in the research: long term preference. When testing bright speakers for example, at first they sure sound more detailed and "better." But listen for a while and that effect becomes grating and not preferred. I have the same issue with some processing effects like simulates surround and such. They sound wonderful first when the stereo presentation comes all around. But over time I get tired of the effect and go back to 2 channel. It is true also that the more you are trained, the less you tolerate variations as the research found.

All this said, because there is no standard in music production as far as tonality, the content may very well demand different overall equalization.

I have a lot of experience with blind taste testing beers. I have a little over 1500 blinds in my database, and it took me several hundred blinds to realize that I was mainly picking the beer with "bolder" flavors, and not necessary the beer I preferred to drink the full bottle of.

 

[goldark]

Also, I don't buy the Harman research where they allowed listeners to tone control to preference. I think listeners were preferring their own tweaks due to self bias. I bet if you took the tweak of the neutral speaker they "preferred" and blinded it against the unaltered version the next day, they would prefer the unaltered version.

If that was actually the case, then the generalizations by Sean Olive couldn't have been made:

" The younger and less experienced listeners on average preferred more bass and treble in their headphones compared to the older, more experienced listeners. Female listeners on average preferred about 1 dB bass and 2 dB treble than their male counterparts. Listeners over 55 years preferred less bass and more treble than the younger listeners "

If listeners were just randomly preferring the sound they tweaked, then the data would be more of a random sample of preferences and tweaks, not the trends in the data being shown. In practice, this is consistent with my anecdotal experiences of young listeners loving their boosted bass and treble while older listeners with hearing loss, prefer even more boosted treble but less bass. The discrepancy between male and female listeners does surprise me, however. I wonder what accounts for that. The tweaks also varied by music being played as well. So users were re-tweaking to their liking track by track, which suggests some orderly methodology of trial-and-error instead of automatically preferring something because they, themselves, tweaked it.

 

[aarons915]

Help me understand the preference rating listener preference correlation. When they say that the bass controlled score has a .99 correlation with listener preference, what does this mean?

What percentage of listeners would prefer a 8.0 speaker over a 6.0 speaker? How about a 6.5 speaker over a 6.0 speaker?

There were 2 separate studies where they did this, the 1st was 13 bookshelf speakers that had relatively similar bass performance. The 2nd set of speakers had bookshelf and tower speakers mixed in and also took place over many months with different groups of listeners, these are the main reasons often cited for the lower .86 correlation in the 2nd study, even though that is still very good. Many people claim these tests are percentages but they are really correlations, meaning the preference rating from the algorithm correlates to the subjective rating from the blind listeners.

I don't know the exact percentage but it would be very unlikely to prefer a speaker 2 preference ratings lower than another, I think it would have to be closer to .5 for it to be a more likely scenario.

 

[Sancus]

I don't know the exact percentage but it would be very unlikely to prefer a speaker 2 preference ratings lower than another, I think it would have to be closer to .5 for it to be a more likely scenario.

Page 11 of the paper states the residual error from the model is 0.8 preference ratings, so it would need to be +/- 0.8. It's been suggested that this should be stated more frequently in the places where the ratings are listed.

 

[richard12511]

If listeners were just randomly preferring the sound they tweaked, then the data would be more of a random sample of preferences and tweaks, not the trends in the data being shown.

It's not random. They were preferring the tweaked sound because they tweaked it.

 

[richard12511]

Page 11 of the paper states the residual error from the model is 0.8 preference ratings, so it would need to be +/- 0.8. It's been suggested that this should be stated more frequently in the places where the ratings are listed.

Page 11 of the paper states the residual error from the model is 0.8 preference ratings, so it would need to be +/- 0.8. It's been suggested that this should be stated more frequently in the places where the ratings are listed.

Does this mean 7.4 speaker would be preferred over a 6.5 speaker by 99% of listeners?

 

[aarons915]

It's not random. They were preferring the tweaked sound because they tweaked it.

I didn't think of that before, it's definitely a possibility. Either way, arguing that some people prefer different than neutral is odd unless you believe that the original vocals or instruments should sound different than they do. I personally think the best speakers are ones that don't get in the way of the music, they should simply reproduce it accurately, not change it in any way.

 

[goldark]

It's not random. They were preferring the tweaked sound because they tweaked it.

I'm saying the data would be more random, and Dr. Olive wouldn't be able to make generalizations like young people like more bass and old people like more treble.

What you're surmising is that young people actually only prefer boosted bass because they, themselves, boosted it, not that they actually like the sound more. And old people don't actually like the sound of boosted treble; they just say they prefer it because it was they who adjusted it.

That is a ridiculous statement.

 

[richard12511]

For example, because it is greater than 0.8, 99% of listeners would prefer the Pioneer BS22 over the Ascend Sierra 2 at a level matched volume that the Pioneer can handle?

 

[richard12511]

I'm saying the data would be more random, and Dr. Olive wouldn't be able to make generalizations like young people like more bass and old people like more treble.

What you're surmising is that young people actually only prefer boosted bass because they, themselves, boosted it, not that they actually like the sound more. And old people don't actually like the sound of boosted treble; they just say they prefer it because it was they who adjusted it.

That is a ridiculous statement.

No, I'm not saying that people with different hearing capabilities(old vs young) don't have different preferences. I'm saying saying they prefer their own tweaks due to bias.

 

[richard12511]

It does make sense that old people with HF hearing loss would prefer a brighter curve, but I think younger folks will still prefer a more neutral curve.

 

[goldark]

It does make sense that old people with HF hearing loss would prefer a brighter curve, but I think younger folks will still prefer a more neutral curve.

My anecdotal experience says that's not the case at all. I've had to give rides to younger highschool aged siblings and all of them have increased the bass on their music from my default settings.

No, I'm not saying that people with different hearing capabilities(old vs young) don't have different preferences. I'm saying saying they prefer their own tweaks due to bias.

So in my previous, example, they can't possibly actually prefer the sound; they only prefer it because they, themselves, twisted the bass knob up.

That is just not logical.

 

[richard12511]

My anecdotal experience says that's not the case at all. I've had to give rides to younger highschool aged siblings and all of them have increased the bass on their music from my default settings.



So in my previous, example, they can't possibly actually prefer the sound; they only prefer it because they, themselves, twisted the bass knob up.

That is just not logical.

I'm not sure we're understanding each other.

I'm saying the test is flawed from the outset, and therefore meaningless. You can't let listeners tone control themselves, as it introduces a bias.

Someone else needs to be in charge of the tone control. Have someone else raise the treble by 2db. Compare the tone controlled speaker against the default speaker(present them as different speakers to the listener) and then collect preference ratings.

My guess is that you would end up with a different correlation.

 

[goldark]

I'm not sure we're understanding each other.

I'm saying the test is flawed from the outset, and therefore meaningless. You can't let listeners tone control themselves, as it introduces a bias.

Someone else needs to be in charge of the tone control. Have someone else raise the treble by 2db. Compare the tone controlled speaker against the default speaker(present them as different speakers to the listener) and then collect preference ratings.

My guess is that you would end up with a different correlation.

At the risk of appealing to authority, I'm sure Dr. Olive will be upset that richard12511 has deemed his research meaningless

I understand the point you're making but with the data clearly showing preference trends that can be grouped neatly by both age and gender out of 249 participants - bias didn't affect the data in a significant way.

Unless you're saying all young people are biased the same way, all old people are biased in the same way, men are biased in the same way, and women are biased in the same way - as it pertains to how they like tweaking by the same general amount without actually preferring the sound of their own tweaks, of course.

 

[Sancus]

Does this mean 7.4 speaker would be preferred over a 6.5 speaker by 99% of listeners?

I'm not sure, I don't know much about correlation coefficients and linear regressions. Based on the graph of measured vs predicted scores, I see differences of more than that, although not often, so it's possible that that's 1 std deviation, which would be 68% and then 95% would require +/- 1.6.

E: On further examination I think this is correct, and the "residual error" he's talking about is the RMS error. In Test One it's specified as 0.26 preference rating, and that one had a very high correlation.

So if you want 95% confidence you need +/- 1.6.

 

[richard12511]

I'm not sure, I don't know much about correlation coefficients and linear regressions. Based on the graph of measured vs predicted scores, I see differences of more than that, although not often, so it's possible that that's 1 std deviation, which would be 68% and then 95% would require +/- 1.6.

View attachment 52880

E: On further examination I think this is correct, and the "residual error" he's talking about is the RMS error. In Test One it's specified as 0.26 preference rating, and that one had a very high correlation.

So if you want 95% confidence you need +/- 1.6.

Thanks. It's clear to me that those Pioneers are very well constructed.

 

[richard12511]

At the risk of appealing to authority, I'm sure Dr. Olive will be upset that richard12511 has deemed his research meaningless

I understand the point you're making but with the data clearly showing preference trends that can be grouped neatly by both age and gender out of 249 participants - bias didn't affect the data in a significant way.

Unless you're saying all young people are biased the same way, all old people are biased in the same way, men are biased in the same way, and women are biased in the same way - as it pertains to how they like tweaking by the same general amount without actually preferring the sound of their own tweaks, of course.

I doubt Sean Olive cares. I think he's contributed more than almost anyone to the science of audio reproduction.

I'm not saying anything you're trying to strawman in your last paragraph. All I'm saying is that letting people know what adjustments are being made prior to listening introduces an element of bias. Do you disagree?

It's like saying "You're about to listen to a speaker that has the treble boosted by 2db", but even worse.

 

[Sancus]

Thanks. It's clear to me that those Pioneers are very well constructed.

Yes, and with respect to the "with subwoofer" score it's important to remember that it's throwing out part of a correlated model, we can only assume that it's only throwing out the bass extension, and not anything unknown that might have correlated with that. A point that has been made in the other thread.

So if you're taking the preference rating with a grain of salt, you really have to take the 'with sub' rating with an even bigger grain. Especially since that score probably creates additional dissonance if people are referring to their experiences listening to some of the cheaper bookshelves WITHOUT a sub.

 

[Xulonn]

At age 78, I am probably one of the oldest "regulars" here at ASR, and my hearing ability has survived remarkably well - up to 10kHz in both ears. Participating at ASR has revived my interest in audio - particularly psychoacoustics and room treatment. But I have also upgraded my system a bit, and currently I am watching for a particularly model of small floor-standing loudspeakers to show up on the used market (Canada and USA). They will replace my excellent Paradigm Atom v6 monitors, but I will wait until I find a pair and order thembefore discussing them.

While I wait for my new speakers to become available, I have been researching and thinking about achieving a Toole "preferable" curve - and other curves to compare with the "preferred" one. Starting with a fairly neutral pair of loudspeakers, the two possibilities are either DSP, or a 3-5 band equalizer, and a bit of room treatment included with either one.

Schitt has an excellent, well reviewed four band EQ - the $150 Loki - that I could use to roll off the treble if necessary. However, I am leaning towards the $325 MiniDSP OpenDRC-DI, which is is a "higher-quality" "all digital solution" unit that works only in the digital domain. (@Sal1950 uses one and likes it.) Unlike the much less expensive $205 MiniDSP 2x4 HD with its internal ADC and DAC (which did not measure too well), the OpenDRC-DI would leave the DAC duties to my Topping DX7s DAC and its XLR connection to my Classé amplifier, which I prefer.

With the ability to store up to four DSP filter profiles in the OpenDRC-DI, I could play with different profiles - just for fun - because I am no longer seeking reach audio nirvana, and just want to have some fun.

Although the OpenDRC-DI does not have USB audio in, there are a number of USB to AES/EBU or coax interfaces available, several of which have been tested - and/or discussed here at ASR. (I prefer USB for a cleaner signal from a computer.)

I also own a tube amp - a 6.5wpc Musical Paradise model MP-310 Mk3, an EL34 single-ended unit, but I bought it for sentimental reasons - not with the expectation of "incredible" sound. I will also play with REW and a UMIK-1 microphone for a bit more objective evaluation.

 

[Blumlein 88]

I'm not sure, I don't know much about correlation coefficients and linear regressions. Based on the graph of measured vs predicted scores, I see differences of more than that, although not often, so it's possible that that's 1 std deviation, which would be 68% and then 95% would require +/- 1.6.

View attachment 52880

E: On further examination I think this is correct, and the "residual error" he's talking about is the RMS error. In Test One it's specified as 0.26 preference rating, and that one had a very high correlation.

So if you want 95% confidence you need +/- 1.6.

That isn't really re-assuring is it? If a speaker scores a 5, you'll need above 6.6 or below 3.4 to be 95% sure of which you'll prefer.