I cannot trust the Harman speaker preference score

[Newman]

To take multi-channel seriously seems to involve something along the lines of the setup shown in the photo below (from an earlier post in this thread). Apart from the overall expense, is that really something that is viable for the majority of consumers? It is for those that may have a specialized room for home theatre applications, but modifying a lounge room to accommodate such a setup just isn't viable for many consumers. However, the latter can still use a good stereo sound system to listen to music, which they have been successfully doing via 2-channel audio for quite a while now, and at a relatively moderate total cost.
View attachment 192280

Have you seen how much a lot of people are spending on 2-channel?? How about a multi-thousand-dollar cartridge, and then everything else priced to scale. And a lot of people do that.

If you are happy to acknowledge the above, then I too acknowledge your point. It’s a long-discussed point.

However, no, I don’t mean that room that Sean Olive posted. When I say taking MCH seriously, I don’t mean bleeding-edge as a minimum: I mean taking it seriously enough to let go of 2-channel obsession and over-attachment to false ideas like “it’s what the artists intended” (hint: no it isn’t).

Floyd Toole recently confirmed in discussion with me that even 5-channel MCH can bring a heap of goodness to the table, so I’m suggesting as a cohort we at least take it seriously enough to go there. With recently published spinorama measurements of affordable speakers, those with modest budgets who thought MCH meant something they can’t afford, can now pick up speakers for a couple or few hundred dollars that can do all the surround duties and have very nice FR and DI and other basics. And they only need 3 of them to jump from 2-channel to 5-channel.

Personally, I think there is another level (pun intended) in MCH World beyond the ‘classic’ 5-or-7-channel MCH, with the introduction of Atmos and the like since 2016 for home theatre. Let’s also go there when we are ready (I am ready: been 5-channel for over ten years, ready to upgrade), and remember very decent speakers are available at every price point to match budgets.

cheers

 

[Bjorn]

This is a good point, and one that I have made before.

Harman has produced good research over the years, and whatever quibbles I have with the methodology used in some of the studies I can understand that simplification was necessary for practical reasons.
But there's no escaping the fact that oversimplification will produce distorted view.

Yes. That's unvoidable. As a speaker designer and of passive designs with "no boundary compensation" you have to choose a low frequency tuning based on a compromise. Some does this with close proximity to the walls (i.e. several Audiphysiscs speakers) and others with placement further away from the front wall and corners.

And with dipoles you generally need more distance to the front wall and dipoles tends to measure flatter with placement close to side walls. With the dipole you also need to take into consideration the specular reflections above the Schoeder frequency.

One could also debate that the room and placement of speakers with good distance to side wall only gives answers to the same type of room. It's very likely that a narrower room with short distance to side wall would favourize speakers with a low beamwidth. After all, specular reflections that arrive early are detrimental.

So these researchers don't give us all the nuances.

My main concern with the Spinorama though is the fact that it gives high scores to speakers that:
- Doesn't have a constant horizontal directivity low enough in frequency (for example only down to 1000 Hz area)
- Has generally poor uniform vertical directivity and suffers from vertical lobing in the most sensitive area
- Doesn't minimize floor and ceiling reflections or floor bounce
- Suffers from thermal compression with the result of frequency deviations at certain levels
- Have high intermodulation distortion

That's why I much prefer seing a whole set of full measurements, so I can interpret them for myself rather looking at some score which can be quite misleading.

 

[thewas]

Controlled blind stereo listening comparisons of the Lexicon SL-1 in its various modes would arguably be the Holy Grail of comparing different radiation patterns. Was this ever done, and if so, do you recall the results?

I remember reading and article or watching a video some years ago where the B&O senior engineer mentioned such comparison with their Beolab 90 which is similar to the SL-1 and was released before it, here few article about it and its different radiation width modes:

B&O Tech: What is “Beam Width Control”?

#34 in a series of articles about the technology behind Bang & Olufsen loudspeakers A little background: Distance Perception in “Real Life” Go to the middle of a snow-covered…

www.tonmeister.ca

B&O Tech: Shark Fins and the birth of Beam Width Control

#37 in a series of articles about the technology behind Bang & Olufsen loudspeakers In the last posting, I described a little experiment that we did where we could easily be accused of asking a…

www.tonmeister.ca

B&O Tech: BeoLab 90 – Behind the scenes

#41 in a series of articles about the technology behind Bang & Olufsen loudspeakers Most loudspeaker projects at Bang & Olufsen are conceived in the design or the Product Definition …

www.tonmeister.ca

Of course would also really appreciate if Harman did some blind testing on their SL-1 radiation modes.

 

[Digby]

You shouldn't focus on preference scores. It was only intended to help naive people interpret spinorama measurements and reduce it to a single number. Buy a loudspeaker based on its spinoroma measurements and how loud it can play without distortion for your application.

If you have the time, and don't mind, could you write a (as far as possible, non-technical) blog post explaining how the preference score should be understood and applied, as there seems to be some confusion here as to what it represents and how it should be understood/interpreted.

I erroneously thought it was a universal "apples to oranges" score (in that you could compare a small and large loudspeaker, cheap and expensive and the score would hold water). However, from what you've said it is not and is only to be used as a rough guide within certain defined limits.

I would like to better understand the degree of resolution the score represents (it has been used, by some, to grade speakers as universally better or worse, but given what you said it seems wrong to be doing so).

The problem is, those who are not experts in the field would appreciate "one score to rule them all", so to speak, but from what you've said the preference score is not that and cannot be used to compare apples (Sonos) to oranges (M2).

It seems there are no shortcuts/free lunches in this respect.

 

[Bjorn]

Controlled blind stereo listening comparisons of the Lexicon SL-1 in its various modes would arguably be the Holy Grail of comparing different radiation patterns. Was this ever done, and if so, do you recall the results?

In what type of room? What room dimension, with which seating distance to the back wall? With which placement of the speaker? With treatment (and if so with what type of treatment), little or no treatment?

I'm afraid the result of such a study would basically only be applicaple to the specific room it was tested in. With the right treatment and positioning you can achieve very similar degree of spaciousness and other aspects with almost any speaker directivity.

 

[thorvat]

With the right treatment you can achieve very similar degree of spaciousness and other aspects with almost any speaker directivity.

Huh.. So speaker's spaciousness/directivity can be corrected with the "right room treatment"? Is that your personal opinion or a claim that you can back up with some research?

Btw, how exactly do you define "spaciousness"? Can you measure it or is it a subjective cathegory?

 

[sarumbear]

So a 5.4 speaker and a 5.5 speaker would be one whole point apart?

I don’t see the usefulness in that.

I don't see the usefulness of pretending a score that is not designed to be used, used as it is now, either. We are fooling ourselves. If the person whose name on the patent says anything more than one point is irrelevant, especially the decimal point cannot work what we are doing is fallacy at its best.

 

[sarumbear]

I agree with @abdo123: it is better to give people the information (including the decimal point) and teach them how to use the information.

If what you are giving is information. Due to the error baked into the scoring system, anything less than a point is noise. There is not enough resolution in the system. It is not me who is stipulating, Dr. Olive has said it many times.

It is similar to how many people are fooling themselves with Hi-Res recordings and are buying noise.

 

[HooStat]

If what you are giving is information. Due to the error baked into the scoring system, anything less than a point is noise. There is not enough resolution in the system. It is not me who is stipulating, Dr. Olive has said it many times.

It is similar to how many people are fooling themselves with Hi-Res recordings and are buying noise.

As was already explained, 5.4 would get a 5 and 5.6 would get a 6. Those scores would then "1 point apart" and would be "different" using the 1-point approach to determining whether speakers are different. Having them at 5.4 and 5.6 tells the user that they are very similar. Rounding the result just trades one problem for another.

What I would like to see is a confidence interval around the estimates, but we don't have the original model to get the proper standard errors from the model coefficients. If we had the original model, we would estimate scores with confidence intervals and I think everyone would be happier.

 

[sarumbear]

Having them at 5.4 and 5.6 tells the user that they are very similar.

But we don't know that. The error baked in the score means we cannot be sure. For all we know the scores can be the other way around. The speaker that scored 5.4 can well be better than the one scored 5.6. The system does not have enough resolution to differentiate such close numbers.

It is better that we compare speakers with 1 full point difference so that we "may" have an understanding. Currently we are fooling ourselves. This is not me saying it, read what Dr. Olive said on the subject, on this very thread!

Rounding the result just trades one problem for another.

I can't see what that problem will be.

 

[Vuki]

Even the good doctor told us not to use it but we still discuss it.

 

[abdo123]

I can't see what that problem will be.

You would literally increase the error from almost two points to almost three points.

 

[sarumbear]

You would literally increase the error from almost two points to almost three points.

Maybe we should test if what you said is true. Is there way I can download the speaker review list data on ASR so that we can see what happens to the scores when their resolution is reduced to +/- 1 points and rounded?

 

[krabapple]

No doubt, but irrelevant. My hypothetical example was a speaker that scores very well by Harman criteria in mono, perhaps by virtue of cabinet sounds compensating for peaks and dips. Such errors aren't distinguishable from driver output in mono measurements, and will result in a good score in mono, but a poor stereo performance. There were dozens of such speakers in the old days, and there probably still are. Pleasant in mono, not so much in stereo.

Sorry, explain this to me. A speaker has 'peaks and dips' in its FR that were/are NOT measurable because they just happen to be compensated for by 'cabinet sounds'? And there were 'dozens' of such models in the old days? Is this your claim?

 

[fineMen]

The problem is, those who are not experts in the field would appreciate "one score to rule them all", so to speak, but from what you've said the preference score is not that and cannot be used to compare ...

It is better that we compare speakers with 1 full point difference so that we "may" have an understanding.

Nope even further!

The decision of a naive (sic!) customer for some model will be a multi-dimensional one. Cost, looks, footprint, less naive bass and spl capabilities and so forth will play a role.

The score doesn't help here. At least not the naive customer. How to trade e/g cost for score points? Is 0.5 on the scale worth double the price? Is the distance between a score 5.5 and 6.7 the same as between 8.1 and 9.3? Numerically, yes, but in comparison to other criteria than the scope itself?

You see, better to not be naive. Re-reiterated, way better to allow for some equalising at home to make it fit, add some sub in case, done and for sure superior.

 

[Sancus]

Even the good doctor told us not to use it but we still discuss it.

Didn't actually read his posts, now did you? It's just ridiculous at this point.

 

[krabapple]

I found it useful. I just disagree with the points mentioned, which have mostly to do with how preference is assessed.
We are debating the usefulness of the Preference Rating for the consumer. I have no use for it myself, but would like to suggest that it needs perfecting in order not to be misleading for the general public.



But aren't the majority listening with earbuds or a monaural Bluetooth virtual assistants?

Also, when most recordings are put out in 2-channel, one must really enjoy space-effects a lot in order to put up with the fuzzy imaging and the phasey? distortion of upmix processing.

Which I don't hear, and I use upmixing routinely.

IIRC you don't really have much experience with upmixers.

Is multi-channel really a goal for many in the US? In the UK, and Europe in general I suppose, sitting rooms are too small for 5.1, let alone immersive audio.

I'm using it in a 13 x 14 ft room. I've used it in even smaller rooms than that. Small enough?

Not to mention that a room will look like a low-budget sci-fi set, unless you are fortunate enough to own a dedicated room, again unlikely in Europe...

A lot of people here seem to be using two desk mounted speakers speakers that they use when working on a computer. So, how LCD (lowest common denominator) do you want to aim?

 

[sarumbear]

Didn't actually read his posts, now did you? It's just ridiculous at this point.

However, the good doctor did say there is no point in using the decimal point and that score cannot be trusted for values less than one.

 

[krabapple]

The perceived effects of widening the images and soundstage spaciousness work best when the off axis response doesn't have peaks and oddities to draw attention to themselves in the room instead of letting you hear thru the room to the recordings. Sorry, you just keep backing up one step at at time to not agree with the data.

It's goalpost shifting...all the ways down.

 

[Sancus]

However, the good doctor did say there is no point in using the decimal point and that score cannot be trusted for values less than one.

Correct, and I completely agree with him. That's how I've always used the score, and I've made countless posts telling people to ignore small differences. However, removing the decimal points from the calculated results without making the differences between scores confusing is tough. Simple rounding doesn't do the job.

What I would like to see is a confidence interval around the estimates, but we don't have the original model to get the proper standard errors from the model coefficients. If we had the original model, we would estimate scores with confidence intervals and I think everyone would be happier.

We do have the original data points for both listener tests and calculated results though. Dunno how much that helps, I'm not a statistician