I cannot trust the Harman speaker preference score

[TimVG]

Can you explain what you mean by the oblique angles are missing?

Well the measurement method calls for a full horizontal and vertical measurement set, which are then weighted to represent a value on a sphere. However in reality, the oblique angles also contribute to the radiation, and often feature more drastic differences than simply weighting the straight angles. In the M2 for instance the waveguide itself features a different shape in the oblique angles than it does in the straight planes, so without a doubt differences in frequency response will occur in these angles compared to the weighted straight angles.

 

[abdo123]

Well the measurement method calls for a full horizontal and vertical measurement set, which are then weighted to represent a value on a sphere. However in reality, the oblique angles also contribute to the radiation, and often feature more drastic differences than simply weighting the straight angles. In the M2 for instance the waveguide itself features a different shape in the oblique angles than it does in the straight planes, so without a doubt differences in frequency response will occur in these angles compared to the weighted straight angles.

View attachment 190729

If i understood you correctly, you're saying that the NFS brings up more flaws by generating the full sound field of the speaker which (due to the unique geometry of the waveguide) is more accurate than the reductive route of measuring specific angles in a typical anechoic room setting / measurement?

I still don't understand what does that have to do with the resonances at 200Hz to 500Hz weighing the score down, the part of the spectrum handled by the horn looks flawless in both measurements.

 

[TimVG]

If i understood you correctly, you're saying that the NFS brings up more flaws by generating the full sound field of the speaker which (due to the unique geometry of the waveguide) is more accurate than the reductive route of measuring specific angles in a typical anechoic room setting / measurement?

I still don't understand what does that have to do with the resonances at 200Hz to 500Hz weighing the score down, the part of the spectrum handled by the horn looks flawless in both measurements.

Yes, that's what they mean when they say the NFS has higher resolution - it has nothing to do with smoothing
I've no idea about tje resonances - perhaps the official spinorama was done with a slightly different cabinet. If I remember correctly Erin concluded these were cabinet resonances - perhaps it was a cost cutting measure. I don't believe at the time Harman, or anyone, would have thought regular people would be crazy enough to buy 100k measurement robots to verify. Just speculating.

 

[sarumbear]

do not understand the limitations of science and statistics-based testing, and extrapolate limited conclusions to be claimed hard truths.

I am duplicating information but the score is not a test. It is the output of a formulae that is statistically came about. A very good research was done more than two decades ago at NRC of Canada and a large set of preference data was collected. Then some clever scientist created a formulae that will replicate the same preference score from that data. They wrote a paper about it but their funding was cut they and went to Harman and continued their research there. Finally they obtained a patent for the formulae for Harman.

It is a bizarre system. It is trying to extract a subjective score from objective data. The patent title is "Method for predicting loudspeaker preference." There is almost no scientific proof that it works as it is never tested with tests similar to NRC's. I expect Harman is continuing tests but in a much smaller scale no doubt. Hence, score's validity was never proven beyond doubt.

If the examples I gave you are to go by score is not reliable. Higher values reflect good speakers and lower values bad lines, but in between it fails miserably. That is why I cannot trust the Harman speaker preference score.

Do we need a score to tell us that the sound of a US$40,000 KEF Blade 1 will be preferred over a US$400 KEF T101?

Not to mention that we may be fooled by the scores of the following speakers even though the score contradicts Revel marketing. Two speakers similar in price and and size and from the same manufacturer within the Harman group.

Revel Studio: 7.3 vs Revel Salon2: 6.3

 

[FrantzM]

Hi

I don't know enough to judge the validity of the method. In the absence of something that would help us separate the wheat from the chaff... We take it and it has some use . I think however that it is an incomplete metric. We have to look for it; but as a final determinant? We are not yet there. The OP examples are cases in point. Do we really expect the SONOS to actually be preferred to the JBL M2, by a statistically significant group of persons? If that is the case that would be a big fail from Harman/JBL: That is their own study and that is supposed to be one of their flagships.
In the same vein the LSR 308 from Harman at $250 would be superior to the LSR 708p from Harman at $1895... Something is not computing ...

Thus, I continue to (often) be befuddled by that score ...
For that, I have defined some thresholds: a no sub preference score of at least 5. I will always use subs in any system I have. Same with SINAD, past 75 dB, I am good to go.. knowing that my subjective audiophile friends in spite of their (often) very expensive analog chain, cannot go over 60 dB ...

Peace.

 

[sarumbear]

While there may be good sounding speakers that don't score as expected, it is fair to say that the scoring system does not let any bad-sounding or ill-preferred speakers near the top. If you are honest with yourself and can acknowledge this, then you also must admit the methodology is more "useful" than flawed- which is far from useless.

Google Nest Audio: 5.5
Sonos Roam: 5.1
Apple Homepod: 4.9

So far the score list goes from -3.3 to 8.1 Will you consider 5.5 "near the top"?

 

[Frgirard]

I thought for a moment that the subject would be approached objectively but now it all comes down to the price.

On the side in a meeting, a CD player manufacturer said that if he made a $200 Hdg sound player, no one would believe him. At 2000 $ he would become credible.

The price depends on what the buyer is willing to pay.
The JBL M2 is an industrial accident in disguise.

 

[Absolute]

It's meaningless in the grander scheme, but can indicate certain traits. It's limited in usage to speakers with normal dispersion characteristics, though.

I think it's time to develop a Spinorama version 2 (with Olive score version 2 to match) to more accurately predict quality. Greater emphasis on far off-axis horizontally will likely create a bigger gap between seemingly similar measuring -and scoring - speakers .

Bass spl output should also be a factor because that's the main difference between a toy like Sonos and a proper audiophile speaker.

If anyone here on ASR think the score alone decides quality, then the site is failing spectacularly in educating people.

 

[sarumbear]

If anyone here on ASR think the score alone decides quality, then the site is failing spectacularly in educating people.

It is the only metric that is filterable on the review list other than price. Do you expect people not use that on their decisions? It is the only ASR rating!

 

[pierre]

I have not heard of any of these companies or even Harman using the score for any speaker design. Where have you read this?

What companies are designing to is the fundamentals of flax on axis and smooth directivity. That is very sound and ideal. But it doesn't equate to computing the score and trying to up the number.

We are telling the same thing. The score is strongly correlated to flat on axis and smooth directivity. Improving both parameters improves the score.

 

[sarumbear]

Dr. Olive himself said he would not trust a score difference below one point to be significant.

I wonder if he would trust the following score differences?

Revel F328be: 6.6 vs Sonos Roam: 5.5 vs JBL 4312SE: 4.5

KEF R5: 5.9 vs Apple Homepod: 4.9 vs JBL 4309: 3.8

 

[Absolute]

It is the only metric that is filterable on the review list other than price. Do you expect people not use that on their decisions? It is the only ASR rating!

If the aim with that list is to provide a filtered list with better to worse, then we need to provide additional categories to sort by rating.
Like full-range, bass-limited, near-field etc so that the list at least is usable for something.

But I would prefer if we would find a way to change the PIR factor in the metric in some way that doesn't unjustly degrade speakers with different dispersion characteristics, like constant directivity designs or omni.

I expect people with limited interest in the technical aspect to take the easiest path to find the answers they are looking for. But I also expect a community like ASR to provide sensible and usable information to those people.

As it is, can we with a straight face tell people to just buy a speaker with a good Olive score and be done with it for all eternity? Misleading at best and therefore useless unless you know the limits of the usability of the score.

Imo, of course.

 

[abdo123]

I wonder if he would trust the following score differences?

Revel F328be: 6.6 vs Sonos Roam: 5.5 vs JBL 4312SE: 4.5

KEF R5: 5.9 vs Apple Homepod: 4.9 vs JBL 4309: 3.8

I don’t see why not. Unless low frequency extension and resonances are inconsequential now.

 

[sarumbear]

I don’t see why not. Unless low frequency extension and resonances are inconsequential now.

Shall I understand that you meant any person will score a higher preference for a Sonos Roam or an Apple Homepod than a JBL 4312SE in a blind test?

 

[Absolute]

Shall I understand that you meant any person will prefer a Sonos Roam or an Apple Homepod over a JBL 4312SE in a blind test?

At 50 dB, it's possible

 

[F1308]

What do you mean by "tribal"?

As soon as many individuals gather, you get a tribu. Now you discover they nearly have the same audio preferences and you go offering tribal loudspeakers that aproach the plotted curves like no other.
As for the weight, try with bamboo...

 

[Kervel]

I think OP is asking to much of the research. Toole and others converted spinorama data in just 4 simple variables (LFX, PIR smoothness etc) that can best predict user preferences. At ASR, for each newly measured speaker, we then do an out of sample prediction to get the score (forecasting). Then OP arbitrarily picks a few speakers where the score seems too high or too low, as compared to common sense.

Any model has outliers, that's no reason to call it useless.

 

[abdo123]

Shall I understand that you meant any person will score a higher preference for a Sonos Roam or an Apple Homepod than a JBL 4312SE in a blind test?

No. Not ANY person, but rather a statistically significant percentage of a group of people in a blind test.

That’s how statstics work.

 

[Waxx]

The problem with this kind of "preferences metric" is that you just can't include all factors. First of all, what sound good is in a large way subjective (altough a flat response, low distortion and even dispertion certainly helps), and for many not only sound is important, but also looks (it need to fit the decor), and maybe also status factors or just practical things like size, how to use it and cost. I often do advice people on buying equipment, and sound is very often not the main factor once a certain quality is reached, looks and practicality is. That is one of the factors why the LS50 is so popular, it looks good, is easy to drive and it sounds more than decent. You can get better sound for cheaper, but not with those other factors in play. Genelec is not that popular altough it sounds as one of the most neutral, partly because it looks very bland and cheap (while it isn't), partly also because the sound is fatiguing for some (including me) for long listening. But the former is a way more important factor in deciding not to buy Genelec in most cases.

 

[abdo123]

Then OP arbitrarily picks a few speakers where the score seems too high or too low, as compared to common sense.

Honestly you would think the opinion ‘big and expensive = good, small and smart = bad’ would be everywhere else but in this forum but here we are.