Preference Rating and the case for subjective preference

[goldark]

I mean in comparisons of different speakers, neutral was most preferred. If something colored was more preferred, it would have been more preferred...right?

They were preferred to non-neutral speakers with other issues besides frequency response (such as directivity as Sean Olive states). Also, there is only a finite number of speakers that can be used in an experiment - the likelihood that a speaker matches up exactly with one's highest preference is near zero, but we still all have a general preference towards neutrality, thus the neutral speakers still won out. However, based on the results of the 2013 study, the hierarchy in preference is not just non-neutral < neutral, but non-neutral < neutral < neutral with user-adjusted bass/treble. It seems intuitive, but often subjective preferences are met with comments about expectation bias and invalidating the preference because it wasn't done under double-blind, scientifically rigorous conditions because preferences in sound have been already scientifically determined. The mere existence of a listener liking the results of his/her tone control adjustments more than dead-neutral shows it's not - preference is still ultimately subjective within the confines of a design being "well behaved."

I almost think adjustment of tone controls actually adds a biasing factor. People think they want a little more treble, so they twist the treble knob a bit and then "oh yeah, yeah that's more like it!"

Possible, but at this point, merely speculation and conjecture. I prefer to take the results of Sean Olive's paper at face value. In an experiment, how do we tell a subject to adjust something to his/her liking and actually know for sure they like it better? We just have to take their word for it.

I posted this as more than just a thought exercise to show that different people like different sound (within neutrality parameters). I wanted to argue that "preference rating" is a misnomer when it should really be called something else that's more accurate to what it is: a rating in how a design adheres to engineering principles that came from past Harman experiments from which some misleading conclusions can be drawn (that neutrality is the king of preferences, for example, when the 2013 Harman study shows this isn't really the case)

 

[QMuse]

well...it might. I don't know how the room was treated. Or how the speakers were positioned. Or how the measurements were taken.

Sidewalls and ceiling were professionaly treated with panels. Wooden floor was covered with thick rug. Speakers were positioned in a 3m wide symmetric triangle with 3rd point being LP. Measurement was taken from LP using moving microphone method with RTA pink noise, shown with 1/6 smoothing.

Graph is showing normal room modes which will happen in any room. Freqeuncies will vary depending on room geometry.

 

[Jon AA]

I posted this as more than just a thought exercise to show that different people like different sound (within neutrality parameters). I wanted to argue that "preference rating" is a misnomer when it should really be called something else that's more accurate to what it is: a rating in how a design adheres to engineering principles that came from past Harman experiments from which some misleading conclusions can be drawn (that neutrality is the king of preferences, for example, when the 2013 Harman study shows this isn't really the case)

In the end, when you look at the average preferred room curves, there really is no conflict. In the speaker preference tests, listeners were listening in a room--they were listening to the in-room curves of the speakers while Harman was comparing the preferences to the anechoic measurements. When looking at room curves, it should be no surprise that a speaker that measures neutral anechoically, will generally produce an in-room curve closely matching the in-room curve that most prefer.

As has been noted, the largest variable is the bass--the in-room response will vary depending upon speaker placement and everybody agrees that EQing that area smooth (subs are helpful) and to taste is the proper thing to do.

If your point is merely about there being some variation in the data--well of course there is. There always will be. What is it you think we should do with that? Give speakers equal ratings even though maybe 90% will prefer one over the other because that other 10% should carry equal weight?

 

[aarons915]

I read this study a bit more in depth and it's clear that there is a difference in preference between loudspeakers and headphones and that is what the study is addressing. Here is a key quote that shows this:

The experiment was repeated for both loudspeaker and headphone playback conditions to determine how closely the two results matched. The average preferred bass and treble levels were 4.8 dB and -4.4 dB, respectively for headphone playback, and 6.6 dB and -2.4 dB for
loudspeaker reproduction. In other words, listeners on average preferred about 2 dB less bass and treble when listening to the same music tracks over headphones compared to loudspeakers.

So this study isn't challenging previous work on preference in loudspeakers. Looks like JonAA beat me to the punch.

 

[tuga]

When looking at room curves, it should be no surprise that a speaker that measures neutral anechoically, will generally produce an in-room curve closely matching the in-room curve that most prefer.

Are you referring to on-axis neutral?

 

[tuga]

If your point is merely about there being some variation in the data--well of course there is. There always will be. What is it you think we should do with that? Give speakers equal ratings even though maybe 90% will prefer one over the other because that other 10% should carry equal weight?

How did you arrive to that figure of 90%?

 

[Jon AA]

Yes, on-axis neutral. Clearly stated in the chart posted. The 90% was a hypothetical. Asking where it came from is missing the point.

 

[goldark]

I read this study a bit more in depth and it's clear that there is a difference in preference between loudspeakers and headphones and that is what the study is addressing. Here is a key quote that shows this:

The experiment was repeated for both loudspeaker and headphone playback conditions to determine how closely the two results matched. The average preferred bass and treble levels were 4.8 dB and -4.4 dB, respectively for headphone playback, and 6.6 dB and -2.4 dB for
loudspeaker reproduction. In other words, listeners on average preferred about 2 dB less bass and treble when listening to the same music tracks over headphones compared to loudspeakers.

So this study isn't challenging previous work on preference in loudspeakers. Looks like JonAA beat me to the punch.

This is why I also linked to comments on his blog: https://seanolive.blogspot.com/2015/11/factors-that-influence-listeners.html

When asked if this contradicted previous studies that age or experience did not factor in listener preferences, Dr. Olive says the following: "The early studies involved comparison of different speakers that varied more than bass and treble balance. Some speakers had resonances that produced serious colorations, distortions, differences in directivity. The headphone study basically takes a flat neutral headphone and asks people to adjust the bass and treble. That's where experience and age seem to take over. The same holds true for loudspeakers when we did a similar experience [sic]"

Obviously he meant "experiment" instead of "experience."

Furthermore, "This study does confirm a previous one where we found untrained listeners adjusted the bass and treble of a loudspeaker and headphone higher than trained listeners. That study only had 3 untrained listeners, which is why we did this much larger study. "

He also says, "You could certainly characterize the younger and less experienced listeners as preferring more boom and tizz in their headphones. However, the more experienced and older listeners prefer the treble setting to simulate what they would hear from an accurate loudspeaker in a room. In terms of bass the more experienced listeners like 4-5 dB bass boost which matches what they prefer from an accurate loudspeaker in a room. "

The conclusion drawn here is certainly different than past studies which said that across age/experience/etc that listener preference remained the same.

The differences between the preferences for headphones and speakers isn't relevant to my point; the fact that there exists significant variation in preferences for 'both' loudspeakers and headphones implies listener preference is more subjective in nature than previously established. In fact so much so that the phrase "significant variations" was used - hardly a 90%/10% breakdown that was characterized by the other poster.

 

[MZKM]

It shows how a speaker with superior anechoic and spinorama measurements measures in a treated room without EQ.

It proves that you can't get anything similar to spinorama predicted in-room response without EQ.

Toole shows that the predicted in-room response tracks very well above the room’s transition frequency.

The Spinorama for the F208 shows a rise at 3kHz for the Sound Power and Early Reflections graph, and we see that rise in the measurement you posted.

You can safely group preference ratings with ~1.00 point difference together, as that’s the statistical average. So, if a speaker that scores a 5.25 gets louder and looks nicer than one that scores a 6.0, then by all means choose the former.

We need some way to be able to sort through the good and the bad, and since relying on Amir‘s own subjective opinions (and him trying to recall how a previous speaker from a year ago sounded) is faulty, having some scoring mechanism is need, and since the preference rating has decent validity, it’s a good option.

Yes, it has its faults, so you should indeed inform yourself with how to read and analyze the measurements. Besides log-spacing, there is no frequency weighting, a peak at 3kHz will get treated similarly to a peak at 10kHz, and even if they score the same, you should know that the latter would sound better.

If we want, I can add a Notes tab to each measurement list to state my opinions on the measurements, such as a speaker sounding worse than the score leads on due to a 3kHz peak.

 

[aarons915]

The conclusion drawn here is certainly different than past studies which said that across age/experience/etc that listener preference remained the same.

The differences between the preferences for headphones and speakers isn't relevant to my point; the fact that there exists significant variation in preferences for 'both' loudspeakers and headphones implies listener preference is more subjective in nature than previously established. In fact so much so that the phrase "significant variations" was used - hardly a 90%/10% breakdown that was characterized by the other poster.

He addresses some of that in the study, he mentions that older listeners could prefer more treble due to age-induced hearing loss and that the female population was very small. I get your point but either way you still want to start with a neutral speaker and EQ to your tastes. Many of the passive speakers that have been measured so far are not just tonally shifted toward the bass or treble, but they have many resonances, peaks, dips, etc.

Having said that, I'm not saying the preference algorithm is perfect, if you've seen the double blind listening test of my LS50 vs Revel M105, you'll see that me and my girlfriend preferred the LS50 over the M105 even though the M105 measures much better. I still believe in the Science though because the M105 was still an amazing speaker and I could be happy with it as well.

 

[goldark]

He addresses some of that in the study, he mentions that older listeners could prefer more treble due to age-induced hearing loss and that the female population was very small. I get your point but either way you still want to start with a neutral speaker and EQ to your tastes. Many of the passive speakers that have been measured so far are not just tonally shifted toward the bass or treble, but they have many resonances, peaks, dips, etc.

Having said that, I'm not saying the preference algorithm is perfect, if you've seen the double blind listening test of my LS50 vs Revel M105, you'll see that me and my girlfriend preferred the LS50 over the M105 even though the M105 measures much better. I still believe in the Science though because the M105 was still an amazing speaker and I could be happy with it as well.

Agreed. A well behaved speaker with good directivity and generally flat response should be the baseline (and would be more receptive to EQ/tone controls), so the engineering targets are still useful. User subjective preferences take over from there. Your blind testing results of the speaker with worse measurements (at least according to the preference rating) just goes to prove the point I was trying to make. If a speaker is "neutral enough," listener preference becomes subjective from there. And the "preference rating" is a misnomer that causes confusion and wrong conclusions to be drawn.

 

[tuga]

He addresses some of that in the study, he mentions that older listeners could prefer more treble due to age-induced hearing loss and that the female population was very small.

(in the Youtube presentation) He also attributes the Germans' preference for slightly less bass to age, but many measurements I've seen show German speakers typically have a somewhat exaggerated top and/or a the frequency response curve tilting upwards from bass to treble.

 

[Sancus]

Having said that, I'm not saying the preference algorithm is perfect, if you've seen the double blind listening test of my LS50 vs Revel M105, you'll see that me and my girlfriend preferred the LS50 over the M105 even though the M105 measures much better. I still believe in the Science though because the M105 was still an amazing speaker and I could be happy with it as well.

There definitely seems to be more to the story of vertical dispersion/reflections than the spinorama research addresses, at least in some types of rooms.

 

[amirm]

Here is the link to the white paper "Listener Preferences for In-Room Loudspeaker and Headphone Target Responses": http://www.aes.org/e-lib/browse.cfm?elib=17042 The abstract itself says the following: "There were significant variations in the preferred bass and treble levels due to differences in individual taste and listener training. "

The conclusion is that under the overarching umbrella of being "neutral enough," listener preferences take over, and from there, there is "significant variation" - this is the definition of subjective preferences.

There is an issue that is not addressed in the research: long term preference. When testing bright speakers for example, at first they sure sound more detailed and "better." But listen for a while and that effect becomes grating and not preferred. I have the same issue with some processing effects like simulates surround and such. They sound wonderful first when the stereo presentation comes all around. But over time I get tired of the effect and go back to 2 channel. It is true also that the more you are trained, the less you tolerate variations as the research found.

All this said, because there is no standard in music production as far as tonality, the content may very well demand different overall equalization.

 

[tuga]

All this said, because there is no standard in music production as far as tonality, the content may very well demand different overall equalization.

This is usually my argument regarding "coloured" systems. There's no universal solution because the "colouration" requirements vary from recording to recording.

 

[goldark]

This is usually my argument regarding "coloured" systems. There's no universal solution because the "colouration" requirements vary from recording to recording.

Floyd Toole refers to this as the "circle of confusion." Basically everything from the studio/room acoustics, recording mic, monitors/headphones used by the engineer along with the audio engineer's own hearing/preferences are inaccurate in their own way before it even reaches our imperfect headphones/speakers with our imperfect hearing.

 

[thewas]

Toole shows that the predicted in-room response tracks very well above the room’s transition frequency.

Can confirm that, it also worked surprisingly well for me despite my current acoustically not optimal room, see here https://www.audiosciencereview.com/...udio-monitor-review.10897/page-31#post-318895

 

[aarons915]

Agreed. A well behaved speaker with good directivity and generally flat response should be the baseline (and would be more receptive to EQ/tone controls), so the engineering targets are still useful. User subjective preferences take over from there. Your blind testing results of the speaker with worse measurements (at least according to the preference rating) just goes to prove the point I was trying to make. If a speaker is "neutral enough," listener preference becomes subjective from there. And the "preference rating" is a misnomer that causes confusion and wrong conclusions to be drawn.

Yes but it would also be nice to see a Klippel of the M105 to make sure it really does measure that well, not that I've seen any Harman measurements that didn't match a 3rd party but it would be nice to know for sure. I personally don't care much about the preference rating, I prefer to look at the graph myself and draw conclusions and to gauge whether any problems are easy to EQ or not.

 

[richard12511]

Help me understand the preference rating listener preference correlation. When they say that the bass controlled score has a .99 correlation with listener preference, what does this mean?

What percentage of listeners would prefer a 8.0 speaker over a 6.0 speaker? How about a 6.5 speaker over a 6.0 speaker?

 

[richard12511]

Also, I don't buy the Harman research where they allowed listeners to tone control to preference. I think listeners were preferring their own tweaks due to self bias. I bet if you took the tweak of the neutral speaker they "preferred" and blinded it against the unaltered version the next day, they would prefer the unaltered version.