I cannot trust the Harman speaker preference score

[sarumbear]

The more I look into it the less I can trust the Harman speaker quality score. IMHO it is a totally meaningless metric. I know the background, I read all the papers even before Harman was involved, and their patent. However, it works so badly that IMHO it is a meaningless metric.

Here are some scores that I took from this database to prove why.

The KEF Reference 2C Meta: 5.6
Sonos Roam: 5.5
JBL M2: 5.1

The 17cm x 6cm smart-speaker Sonos Roam scores just shy of a real reference of a speaker from KEF, while a JBL flagship that weighs 60kg scores less than that smart-speaker.

Do I have a case or not?

----
Edit: I copied below some more comparisons I posted later in the thread.

Revel Ultima2 Salon2: 6
Google Nest Audio: 5.5

Apple Homepod: 4.9
Bowers & Wilkins 803 D3: 4.8

 

[voodooless]

It’s not about quality, but preference.

But yes, as with SINAD, it’s just one of many parameters to judge by. I’d rather look at the graphs myself to see what objective performance is like.

 

[sarumbear]

It’s not about quality, but preference.

I have amended but I think it is simply semantics. As price is not part of the equation, what your prefer will be the higher quality, wouldn't it?

In my example above will anyone prefer the Sonos smart-speaker over the JBL M2? (Again, speaker size is not part of the equation.) Or will people have difficulty in selecting between the KEF and Sonos on a blind test?

 

[voodooless]

I have amended but I think it is simply semantics. As price is not part of the equation, what your prefer will be the higher quality, wouldn't it?

It’s not about what I prefer, but what apparently on average is preferred.

In my example above will anyone prefer the Sonos smart-speaker over the JBL M2? (Again speaker size is not part of the equation.)

Well, maximum output is one of the obvious criticisms. One can come up with many more quite easily.

 

[sarumbear]

It’s not about what I prefer, but what apparently on average is preferred.

Tests may have been correct but the formulae was written by a person or group of persons. It is derived from the tests but who can guarantee that it is working correctly? I showed that, on this occasion at least, the formula is failing. If a formula fails we should question its validity.

 

[Chester]

I thought the Sonos sounded good, but now I know it doesn’t weigh 60kg, it sounds awful.

 

[Xulonn]

It’s not about quality, but preference.

LOL! My exact though after seeing the word "quality".

Do I trust it? Yes, of course. It is a carefully designed and administered testing technique. The results have been challenged, but I have seen no evidence that the testing method was invalid.

Would I use it to determine which speakers I should buy? Of course not! But it is a factor to consider. My "preference" might be a bit different from the groups who were tested, and as @voodooless indicated, they did not test all attributes of the speakers for listening to all types of music at levels from very soft to extremely loud.

 

[sarumbear]

I thought the Sonos sounded good, but now I know it doesn’t weigh 60kg, it sounds awful.

You will like this

Apple Homepod: 4.9
Bowers & Wilkins 803 D3: 4.8

 

[sarumbear]

Do I trust it? Yes, of course. It is a carefully designed and administered testing technique. The results have been challenged, but I have seen no evidence that the testing method was invalid.

First of all, allow me to correct. Score is not a testing method nor a technique. It is the output of a calculation. You enter the test values, the formula outputs a number. If the formulae is not correct the results will be wrong. So far nobody tested the formula by creating the same test and see if the scores match to what the formulae output.

The researchers involved are the best in the world but even the best can fail. When we see silly results like smart speakers scoring better than reference speakers shouldn't we question? The anomalies are many, not just a one off.

 

[Chester]

One thing I probably agree with you on: All these preference curves based on what is most popular: I can understand the Apple, Bose, Sony’s of the world really caring about it, as if it’s the most preferable curve for humans, that’s going to equate to greater chance of sales.

Why are we, as people who clearly care more than the average, and have a greater desire for sound as it was meant to sound, care about these curves that sound good to people who care less about sound?

I don’t know the answer by the way, just something I’ve always wondered and your thread seemed an appropriate place to mention it

 

[voodooless]

So far nobody tested the formula by creating the same test and see if the scores match to what the formulae output.

Why not? It seems to me that this should be tribal trivial to unit test.

 

[sarumbear]

One thing I probably agree with you on: All these preference curves based on what is most popular: I can understand the Apple, Bose, Sony’s of the world really caring about it, as if it’s the most preferable curve for humans, that’s going to equate to greater chance of sales.

One of us is mistaken. The score is the result of a formulae. Not on personal or historical tests.

 

[sarumbear]

Why not? It seems to me that this should be tribal to unit test.

What do you mean by "tribal"?

 

[tuga]

IMHO it is totally meaningless metric.

Agreed.

 

[voodooless]

What do you mean by "tribal"?

Damn autocorrect should read: trivial

 

[sarumbear]

Damn autocorrect should read: trivial

However, the test are not trivial to replicate. We are talking blind tests with an audience made of a diverse demographic. It is a huge undertaking that was done at The National Research Council (NRC) which was backed by the Canadian government. Nobody has resources like that anymore.

 

[Chester]

One of us is mistaken. The score is the result of a formulae. Not on personal or historical tests.

Interesting, which formula? I thought it was researched based?

 

[sarumbear]

Interesting, which formula? I thought it was researched based?

It is based on research. You test speakers and enter the results into a formulae, which outputs a value. That is the preference score. It is derived from tests and based on research but it is a calculation.

Look at a speaker test on ASR. @amirm test the speaker, then someone makes the calculation from those tests and posts the score.

 

[voodooless]

However, the test are not trivial to replicate. We are talking blind tests with an audience made of a diverse demographic. It is a huge undertaking that was done at The National Research Council (NRC) which was backed by the Canadian government. Nobody has resources like that anymore.

That’s a little bit different from what I meant, but a very valid point. Besides, who even knows how preferences changed over the years? There might even be cultural differences over continents? Even the average way houses are build will probably influence the preference for a speaker. How much of the preference score is contingent on the particular room the test were conducted in?

 

[tuga]

You can’t assess preference listening to a single speaker. Unless you only listen to mono recordings over a single speaker…

The majority of the sample seemed to have preferred the wider-directivity Harman-designed? Rega over the Kef and by a larger margin the Quad when listening in mono but the 3 speakers rated equally well in stereo.
I disagree with the testing methodology and with the interpretation of the data.