I cannot trust the Harman speaker preference score

[Frgirard]

Is that a language barrier or just you being a dick? Different people assigning different meanings and values to things has nothing to do with nihilism at all. Nihilism denies values and meaning altogether (more or less).

We speak about the Harman preference score. Process supposed to be objective. The objective applies to everyone and is not variable in geometry. This implies a hierarchy of values.

Rhetoric is nothingness. You are rhetoric

 

[IPunchCholla]

We have left nuances of language and entered into miss-understanding what is written. I start with saying “I cannot trust the Harman speaker preference score”. It’s what I think. Check the meaning of IMHO.

Why some people can’t accept that others may have different opinions?

You’re right you started there. But you did not end there. You said you can not trust the Harman QUALITY score. You couched it as opinion (using IMHO etc.). You also couched it in the language of proof. Presenting “evidence“ to validate the opinion. So the water was muddied by a misnomer at the start and then this waiving of “proof” and “evidence” around.

If your categorically saying that you don’t like it. Fine. I have no problem with that. If you are saying you find no value in it. Also fine. I take exception where the argument (not necessarily, but maybe yours) is made that it has no value at all (objectively or to others). I take exception at the claim of proof, even for your own opinion as your ”evidence” doesn’t prove anything. You can post as many different examples of similarly scoring speakers as you want. Proof, and really evidence, needs, at a minimum, a second step, showing that people (in some significant manner) prefer those speakers differently. You don’t seem to like that it is your job as the claimant to provide that proof. Fine, don’t make or retract the claim. This is also a matter of respecting the labor of those who produced the scoring system.

As for why I am not letting this go, even after I said I would, is people tend to use “opinion” as cover for stating false facts. They then tend to also hide under claims of misunderstanding or irrelevant gotchaisms (again claims of fact) in order to deflect. I do not believe I am misunderstanding anything. Nor do I worship the score because it was produced by computers. I’m still here responding because nuance is important, not because I don’t value it.

To summarize:

You don’t trust it. Fine.
You don’t find value in it. Fine.
You think it is weird that speakers with obvious differences (to you) score similarly. Fine.
That proves your opinion valid. Nope. Opinions don’t need proof and logically, the presenting of those scores proves nothing. It is reasonable to point this out.
That others shouldn’t use it or it is objectively meaningless. Those are arguing words.
This thread has slipped into the last several times, and your own language has implied it on occasion. So here I am. Again.

The Harmin score is literally an example of how meaning is created. It is an answer to what do we mean by a good speaker. Is it the only answer or the best answer? No. If it is even a good answer is debatable. But how that number came to be is the currently accepted definition of scientific meaning and creating it took an enormous amount of work. It should be attacked. It should be tested. That is how it will be made even more meaningful. But those attacks should be meaningful themselves.

You offered an interesting observation. Thank you.

 

[sarumbear]

You’re right you started there. But you did not end there. You said you can not trust the Harman QUALITY score.

I used the wrong word, which I later corrected and never used it again. Why do you make it a big issue? (Typing in caps?)

To summarize:

You don’t trust it. Fine.
You don’t find value in it. Fine.
You think it is weird that speakers with obvious differences (to you) score similarly. Fine.

Everything is fine by you but I deserved a negative response.

You offered an interesting observation. Thank you.

Finally you understood what I said in my OP. You are welcome.

 

[Sean Olive]

My naivety then. I thought Harman tested lots of people and then used their preferences to calculate a target curve (of likely preference).

Yes, that is exactly how the model was developed. Controlled listening tests on many speakers were conducted to measure listener preferences. Objective measurements of the loudspeakers were then used to model and predict the preferences.

 

[Sean Olive]

The more I look into it the less I can trust the Harman speaker quality score. IMHO it is a totally meaningless metric. I know the background, I read all the papers even before Harman was involved, and their patent. However, it works so badly that IMHO it is a meaningless metric.

Here are some scores that I took from this database to prove why.

The KEF Reference 2C Meta: 5.6
Sonos Roam: 5.5
JBL M2: 5.1

The 17cm x 6cm smart-speaker Sonos Roam scores just shy of a real reference of a speaker from KEF, while a JBL flagship that weighs 60kg scores less than that smart-speaker.

Do I have a case or not?

The model does not take into account distortion and max SPL. That is clearly a limitation stated in the AES paper. The model only considers frequency response defined by the Standard Method of Measurement for In-Home Loudspeakers. ANSI/CTA-2034-A.

I can give you many examples where the weight, price and size of the speaker is a poor predictor of how it will be rated in a listening test.

 

[Sean Olive]

You can’t assess preference listening to a single speaker. Unless you only listen to mono recordings over a single speaker…

The majority of the sample seemed to have preferred the wider-directivity Harman-designed? Rega over the Kef and by a larger margin the Quad when listening in mono but the 3 speakers rated equally well in stereo.
I disagree with the testing methodology and with the interpretation of the data.

You most certainly can assess preference listening to a single speaker. We've been doing it for 4 decades. The question is whether the results can be extrapolated to stereo and multichannel layouts. I think we've shown that if a speaker does well in mono, it will generally do well in stereo and multichannel setups.

What we've shown is that as you increase the number of channels listeners become less sensitive or critical to the off-axis performance. So mono tests produce more discriminating responses. Does that mean our tests are too sensitive to colorations compared to typical stereo-multichannel setups? Maybe. But it depends on the context and the recording. Many movies and music recordings have single instruments or voice panned and even soloed to a single speaker: Movies, TV news, and music are perfect real-life example of a mono test being replicated in your home.

 

[Sean Olive]

How many of us have gone to a dealer and asked to listen to a single speaker?
“Ah, this one sounds better, I’ll have two to take away, please…”
Or brought a single speaker home to test?
It’s absurd.

How many people go to audio dealers these days to hear a demo , period? Based on the number of stores closing I would say fewer every day.
But if they did, they would be smart to audition/compare them in mono rather than pairs because the research indicates we are more critical of certain flaws in the speaker when hearing one source versus multiple sources.

Dealers, on the other hand, probably want to demo in 7.1.4H if they want to sell lots of speakers and amps.

 

[Sean Olive]

I have not heard of any of these companies or even Harman using the score for any speaker design. Where have you read this?

What companies are designing to is the fundamentals of flax on axis and smooth directivity. That is very sound and ideal. But it doesn't equate to computing the score and trying to up the number.

We don't calculate the scores as a routine part of loudspeaker design, but engineers know that if the spinorama measurements meet certain criteria (as defined in the model) they will yield high preference ratings in listening tests. Other parameters (distortion, sensitivity, directivity, Max SPL) are also taken into consideration in the design.

 

[AdamG]

How many people go to audio dealers these days to hear a demo , period? Based on the number of stores closing I would say fewer every day.
But if they did, they would be smart to audition/compare them in mono rather than pairs because the research indicates we are more critical of certain flaws in the speaker when hearing one source versus multiple sources.

Dealers, on the other hand, probably want to demo in 7.1.4H if they want to sell lots of speakers and amps.

Maybe stating the obvious. But this back confirms why Amir does his Test and Evaluation’s normally in just Mono (One speaker) set up. Good points Sean, thank you for participating Sir.

 

[Duke]

How many people go to audio dealers these days to hear a demo , period? Based on the number of stores closing I would say fewer every day.
But if they did, they would be smart to audition/compare them in mono rather than pairs because the research indicates we are more critical of certain flaws in the speaker when hearing one source versus multiple sources.

I understand that loudspeaker colorations are easier to hear in mono. But does spatial quality preference in mono reliably predict spatial quality preference in stereo?

According to a paper written by Wolfgang Klippel and cited by Floyd Toole in his book (third edition, pages 185-186), the “feeling of space” makes a 50% contribution to "naturalness” (realism and accuracy), and a 70%(!) contribution to "pleasantness" (general satisfaction or preference). Here is how Toole sums it up (page 186):

"Sensations of sound quality and spaciousness contribute equally to impressions of "naturalness", and spatial quality dominated the impression of "pleasantness". Therefore whether one is a picky purist or a relaxed recreational listener, the impression of space is a significant factor."

The point of the preceding two paragraphs being, spatial quality matters a lot.

In the study posted by @tuga in post number 53, going from mono to stereo there is more movement in the spatial quality scores than in the sound quality scores, with which speakers rank first and second trading places. Now admittedly this is a very small sample size, but one of the conclusions one might draw from it (in the absence of additional data, which is the situation for most of us) is that preference in mono is a less reliable predictor of spatial quality preference than it is of sound quality preference.

What are your thoughts on the reliability of spatial quality preference in mono as a predictor of spatial quality preference in stereo?

 

[Sean Olive]

I am duplicating information but the score is not a test. It is the output of a formulae that is statistically came about. A very good research was done more than two decades ago at NRC of Canada and a large set of preference data was collected. Then some clever scientist created a formulae that will replicate the same preference score from that data. They wrote a paper about it but their funding was cut they and went to Harman and continued their research there. Finally they obtained a patent for the formulae for Harman.

It is a bizarre system. It is trying to extract a subjective score from objective data. The patent title is "Method for predicting loudspeaker preference." There is almost no scientific proof that it works as it is never tested with tests similar to NRC's. I expect Harman is continuing tests but in a much smaller scale no doubt. Hence, score's validity was never proven beyond doubt.

If the examples I gave you are to go by score is not reliable. Higher values reflect good speakers and lower values bad lines, but in between it fails miserably. That is why I cannot trust the Harman speaker preference score.

Do we need a score to tell us that the sound of a US$40,000 KEF Blade 1 will be preferred over a US$400 KEF T101?

Not to mention that we may be fooled by the scores of the following speakers even though the score contradicts Revel marketing. Two speakers similar in price and and size and from the same manufacturer within the Harman group.

Revel Studio: 7.3 vs Revel Salon2: 6.3

I worked at the National Research Council with Floyd Toole from 1985-1993 and then joined him at Harman in 1993 where I've been ever since. The "clever scientist who created a formula" is me

Toole wrote the seminal papers about loudspeaker measurements and listener preferences in 1985-86. Those tests continued at HARMAN in 1993. Predicting subjective data from objective measurements is not so "bizzare." Many examples of it exist in the world of sensory measurements. Klippel did his PhD on the topic (1990s), Moore & Tam (headphones) and many others.

The data from the model is based on loudspeaker tests done at HARMAN -- not NRC tests. So I'm not sure why it's necessary to prove the model works based on NRC tests. However, since the loudspeaker measurements and listening tests are refined versions of what we did at NRC there are very strong correlations and agreement. Speakers that score well at NRC also score well at HARMAN in spite of differences in listening rooms, listeners, equipment and a speaker mover that we have at HARMAN but not at NRC. The only common link is the scientists conducting the experiments and similar methodology. This tells me that the relationship between subjective and objective measurements is quite robust.

Again, you seem to be obsessed with price being a predictor of loudspeaker. Again, I have data showing a $400 speaker being preferred to a $10,000 speaker in a blind listening test. Do they have the same predicted score? No. If they do, then you have to consider whether the SPL output and distortion will satisfy your application.

 

[Pdxwayne]

I worked at the National Research Council with Floyd Toole from 1985-1993 and then joined him at Harman in 1993 where I've been ever since. The "clever scientist who created a formula" is me

Toole wrote the seminal papers about loudspeaker measurements and listener preferences in 1985-86. Those tests continued at HARMAN in 1993. Predicting subjective data from objective measurements is not so "bizzare." Many examples of it exist in the world of sensory measurements. Klippel did his PhD on the topic (1990s), Moore & Tam (headphones) and many others.

The data from the model is based on loudspeaker tests done at HARMAN -- not NRC tests. So I'm not sure why it's necessary to prove the model based on NRC tests. However, since the loudspeaker measurements and listening tests are refined versions of what we did at NRC there are very strong correlations and agreement. Speakers that score well at NRC also score well at HARMAN based in spite of the fact that the listening rooms, listeners, equipment and speaker mover are not common between both places. The only common link is the scientists conducting the experiments.

Again, you seem to be obsessed with price being a predictor of loudspeaker. Again, I have data showing a $400 speaker being preferred to a $10,000 speaker in a blind listening test. Do they have the same predicted score? No. If they do, then you have to consider whether the SPL output and distortion will satisfy your application.

Since you are here, may I know where I can find out more about the blind tests details?

For example:
Room size
Distance of listeners to speakers
Average SPL level
Music used
Etc.

Thanks!

 

[tuga]

What we've shown is that as you increase the number of channels listeners become less sensitive or critical to the off-axis performance. So mono tests produce more discriminating responses.

I don’t disagree with that.
What I am questioning is how you correlate more or less sensitivity to off-axis performance with preference, particularly when most people will listen to 2-channel stereo.
How can you associate wider directivity with better sounding is listening in stereo drastically reduces how wider directivity speakers rate in comparison with narrow directivity ones?

 

[tuga]

Many movies and music recordings have single instruments or voice panned and even soloed to a single speaker: Movies, TV news, and music are perfect real-life example of a mono test being replicated in your home.

I have maybe 3,000 albums, but only a few stereo mixes will have an instrument playing from a single channel (and will probably sound ghastly e.g The Beatles).
I don’t use my system for video so I am not qualified to comment on cinema.

 

[Sancus]

I don’t use my system for video so I am not qualified to comment on cinema.

It is pretty normal for a single speaker(obviously the center does this, but also the surround speakers sometimes) to be the only one playing a vocal line or some sound effect in multi-channel film soundtracks.

Also, multi-channel music is a thing and an increasingly important thing considering the massive amount of effort going towards Atmos music production now. In multi-channel music it is more common for a single speaker to be playing vocals or some instrument, or even for vocals to be panned across multiple speakers(Yello's Point album pans vocals 360 degrees around the listener and one speaker is primary for most of the pan).

So, I do think mono performance has some value standing alone.

 

[Sean Olive]

I understand that loudspeaker colorations are easier to hear in mono. But does spatial quality preference in mono reliably predict spatial quality preference in stereo?

According to a paper written by Wolfgang Klippel and cited by Floyd Toole in his book (third edition, pages 185-186), the “feeling of space” makes a 50% contribution to "naturalness” (realism and accuracy), and a 70%(!) contribution to "pleasantness" (general satisfaction or preference). Here is how Toole sums it up (page 186):

"Sensations of sound quality and spaciousness contribute equally to impressions of "naturalness", and spatial quality dominated the impression of "pleasantness". Therefore whether one is a picky purist or a relaxed recreational listener, the impression of space is a significant factor."

The point of the preceding two paragraphs being, spatial quality matters a lot.

In the study posted by @tuga in post number 53, going from mono to stereo there is more movement in the spatial quality scores than in the sound quality scores, with which speakers rank first and second trading places. Now admittedly this is a very small sample size, but one of the conclusions one might draw from it (in the absence of additional data, which is the situation for most of us) is that preference in mono is a less reliable predictor of spatial preference than it is of sound quality preference.

What are your thoughts on the reliability of spatial quality preference in mono as a predictor of spatial quality preference in stereo?

If we ignore frequency response (which does affect spatial perception) which aspect of loudspeaker performance would give different spatial ratings in mono versus stereo? The only one I know is directivity, and most of the research on speaker directivity has not come up with definitive guidelines. It's difficult to manipulate DI while not manipulating other parameters.

This literature review of speaker directivity research in 2009 tells us there are no clear cut winners. DI doesn't seem to affect localization (due to the precedence effect), and most of the effects we see seem to be attributed to how the speaker was interacting with the room acoustics. As rooms became more acoustically dead the differences in DI may become less noticeable. For example, wider dispersion speakers have the potential to provide stronger lateral reflections in the right rooms than say directional speakers and this will increase image width and spaciousness. These speakers also seem to do well in both mono and stereo. But for immersive audio setups (e.g. 7.1 and 7.1.4) this likely matters less since the height and surround/rears contribute more to the perceived sense of space than the off-axis reflected sounds.

So, I think more careful research is needed before we can say for sure. The effects seem to be very contextual depending on the number of speaker channels, how the recordings were made, and room acoustics. Naive listeners seem to weigh spaciousness higher than timbre compared to trained/professional listeners.

In these experiments, did they carefully control everything in the loudspeaker, besides the directivity? I don't think compromising the sound quality of a loudspeaker for better spatial quality is a good tradeoff, in spite of what Klippel's 1990 research suggest -- especially in 2022 where we have immersive audio loudspeaker arrays. At any point in a recording, movie or broadcast there will be a mono speaker activated that will reveal any spectral defects in the loudspeaker.

AES E-Library » Effects of Loudspeaker Directivity on Perceived Sound Quality - A Review of Existing Studies

The directivity of a loudspeaker system is often regarded as a prominent factor in the overall subjective quality of the reproduced sound experience. Much literature is available on the topic, and currently a broad field of opinion exists amongst designers. This paper provides an overview of the...

www.aes.org

 

[krabapple]

I looked up indoctrination - the process of teaching a person or group to accept a set of beliefs uncritically

But that doesn't mean that challenges to critiques must indicate that the challenger is 'indoctrinated'.

For example, a critique could simply be based on faulty information or understanding. Pointing that out isn't a sure sign of 'indoctrination'.

By the same token, being a critic isn't a sure sign you're a crank.

My anecdotal experience, though, is that online 'mavericks' or 'contrarians' tend to over-rate their grasp of some 'truth' they they can see but the deluded masses (not excepting 'masses' of scientists) have missed. The internet has simply inflamed a phenomenon that already existed (see for example, evolution denialism well before there was an internet). There's a reason most such warrior-savants end up being just cranks. But that's no fun, so we (as a society) also tend to valorize the relative few who turn out to be right.

 

[Sean Olive]

I don’t disagree with that.
What I am questioning is how you correlate more or less sensitivity to off-axis performance with preference, particularly when most people will listen to 2-channel stereo.
How can you associate wider directivity with better sounding is listening in stereo drastically reduces how wider directivity speakers rate in comparison with narrow directivity ones?

Y

Since you are here, may I know where I can find out more about the blind tests details?

For example:
Room size
Distance of listeners to speakers
Average SPL level
Music used
Etc.

Thanks!

Most of this research is available at AES e-library, and much of it Floyd's book Sound Reproduction. Some of it is in by blog (www.seanolive.blogspot.com)

 

[Sean Olive]

I don’t disagree with that.
What I am questioning is how you correlate more or less sensitivity to off-axis performance with preference, particularly when most people will listen to 2-channel stereo.
How can you associate wider directivity with better sounding is listening in stereo drastically reduces how wider directivity speakers rate in comparison with narrow directivity ones?

Not sure I understand the first question: I am saying that the off-axis coloration is more audible in mono than stereo. If it sounds uncolored in mono it will sound uncolored in stereo. If it sounds colored in mono, it may sound less colored in stereo until an instrument is hard panned to one of the speakers. The audibility of the coloration will increase in more reflective rooms.

The second question I don't understand what you mean.

 

[tuga]

How many people go to audio dealers these days to hear a demo , period? Based on the number of stores closing I would say fewer every day.
But if they did, they would be smart to audition/compare them in mono rather than pairs because the research indicates we are more critical of certain flaws in the speaker when hearing one source versus multiple sources.

Dealers, on the other hand, probably want to demo in 7.1.4H if they want to sell lots of speakers and amps.

You've been testing for 40 years though.

In regard to 'mono rather than pairs' being 'more critical of certain flaws in the speaker when hearing one source versus multiple sources', Harman research seems to indicate that untrained listeners have very little ability to identify flaws. On top of that some flaws sound pleasing (to some people). And that is what preference is about, choosing a "presentation" one prefers.