I cannot trust the Harman speaker preference score

[Sean Olive]

@Sean Olive It is so refreshing to hear someone with your credentials say that Stereo is outdated and irrelevant today.

Sometimes Stereo really seems like the hill this industry wants to die on, with all its faults.



I just posted this on Twitter the other day, after coming across this. AES paper.

"Research shows a more emotional response to music listened in 3D format (9.1 channel) compared to surround (5.1) and stereo (2-channel). So, if you are still listening to stereo you are depriving yourself of a more emotional music experience. "

AES E-Library » Musical Emotions Evoked by 3D Audio

Do 3D audio formats enhance the quality or intensity of musical emotions? In this study, listeners were presented with classical musical excerpts in Stereo, 5.1 Surround, and Auro-3D 9.1 playback formats. Participants were asked to report their emotional states on the Geneva Emotional Music...

www.aes.org

 

[Inner Space]

A speaker with off axis peaks and dips scores very poorly by Harman criteria.

No doubt, but irrelevant. My hypothetical example was a speaker that scores very well by Harman criteria in mono, perhaps by virtue of cabinet sounds compensating for peaks and dips. Such errors aren't distinguishable from driver output in mono measurements, and will result in a good score in mono, but a poor stereo performance. There were dozens of such speakers in the old days, and there probably still are. Pleasant in mono, not so much in stereo.

 

[Andrej]

The "Olive score" is based on frequency response. Frequency response is the single most important predictor of sound quality. Distortion is only important if its well above audible threshold. Compression is essentially distortion.

In most of our listening tests distortion was not a factor with the home speakers we tested in our listening room at the average level ( 80 dB-weighted SPL). Even with the small 5-6 inch bookshelf speakers, distortion was not a factor.

Most loudspeaker researchers (Toole, Klippel) would agree that frequency response is the biggest predictor of sound quality.

However, distortion can be a significant factor in small powered smart speakers. They usually have dynamic electronic limiters that protect them from overload by rolling off the bass as the excursion /power limits are approached. The sound quality is often dependent on the volume setting. I would say, the predictive model is the least useful for these types of loudspeakers.

I do not have much first hand knowledge in this area, but given the current availability of gadgets which can get most speakers to have virtually identical frequency response (Dirac and such) and in a well treated room to minimize reflections, do all modern speakers become indistinguishable?

What is left is distortion? Something else? Is that the only thing to look for in selecting speakers (plus low frequency extension, I guess)?

What is particularly troubling to me, due to the Fletcher-Munson family of curves, every speaker is "optimal" at only one volume level. At 80 dB, I would expect that the preferred frequency response would have a significant boost in low frequencies, and a little in the high frequencies. Not the same as it would be at 60 dB or 100 dB. For a more meaningful comparison, I'd adjust every speaker with Dirac (full bandwidth), with an "optimal" target curve for the desired output level, which would then remove the dominant differentiator, which can be mostly eliminated anyway, with very little effort.

Finally, none of this has a direct relationship to "fidelity". But very important when trying to make listeners (customers) happy.

Does this argument make sense to you?

 

[TurtlePaul]

I do not have much first hand knowledge in this area, but given the current availability of gadgets which can get most speakers to have virtually identical frequency response (Dirac and such) and in a well treated room to minimize reflections, do all modern speakers become indistinguishable?

No, I think there is some research showing that we perceive direct sound and early reflections diferently than total sound power. If you EQ total sound power to be right your brain will still know the on-axis is wrong.

 

[Blumlein 88]

No doubt, but irrelevant. My hypothetical example was a speaker that scores very well by Harman criteria in mono, perhaps by virtue of cabinet sounds compensating for peaks and dips. Such errors aren't distinguishable from driver output in mono measurements, and will result in a good score in mono, but a poor stereo performance. There were dozens of such speakers in the old days, and there probably still are. Pleasant in mono, not so much in stereo.

I don't believe you can manage to get a good spin-o-rama the way you are thinking. It will show up there. If forget the details of how they decided on how closely to do the spin data, but it wasn't something they just picked and ran with. Every ten degrees all around both axis and broken into at least 1/20th octave data.

 

[Andrej]

No, I think there is some research showing that we perceive direct sound and early reflections diferently than total sound power. If you EQ total sound power to be right your brain will still know the on-axis is wrong.

How does that change as you add more and more sound absorption treatments? Ultimately, in a hypothetical perfect anechoic chamber it would make no difference. The question I have is how close do you have to get to an anechoic chamber to fool your brain?

Are you saying that direct vs indirect sound ratio is the next best predictor of speaker preference after frequency response? The original claim by Sean Olive was: "Frequency response is the single most important predictor of sound quality." Surely Dirac can make speakers close to equal in terms of frequency response? Perhaps if Dirac was applied in an anechoic chamber? It could even emphasize perceptual effects of differences in reflected sound?

 

[TurtlePaul]

How does that change as you add more and more sound absorption treatments?

Most sound absorption materials absorb much more highs than bass, so eventually woth too much (but not 2 foot think panels like an anechoic) you end up with a dark sounding room. I guess you could EQ that back to flat, bit why not buy neutral sounding speakers to begin with?

 

[Andrej]

Most sound absorption materials absorb much more highs than bass, so eventually woth too much (but not 2 foot think panels like an anechoic) you end up with a dark sounding room. I guess you could EQ that back to flat, bit why not buy neutral sounding speakers to begin with?

Because any reflections are in fact noise. Whether we like the sound of reflections or not, they will always (perhaps insignificantly) mask the original recording. In my limited experience, adding absorption removes the fog reflections introduce when, for example, recreating the ambiance of the original recording. Certainly, adding more and more absorption has kept improving the clarity of reproduction in my experience. And I have never craved for reflected sound when listening through a good pair of headphones.

All speakers are pretty much omni-directional below, say, 500Hz (equivalent reflections) and it does not take impossible absorption to do a very effective job above ~500Hz (or evan much lower frequencies), significantly removing the effect of reflections. I would also argue that rooms with a lot of absorption are the natural ones, and the typical "living room"-like listening rooms are the opposite of what you called dark. It is all about what you define as the reference.
Having tried both, I do not have any doubt as to where my preferences lie. It also makes speaker selection much easier, as you can remove the need for a particular frequency response (with Dirac, etc.) or directivity characteristic.
Just my personal preference for achieving high fidelity reproduction I enjoy

But to get back on topic, the value of perceptual experiments as done in the past, when comparing various speakers, as I have said in previous messages, are of dubious value for ranking the "quality" of loudspeakers. All I am trying to do is to propose changes which might remove some of my objections. And hoping that somebody will come with more and better changes.

 

[Holmz]

Because any reflections are in fact noise. …
…

I don’t think ^that ^ is right?
Reflections that are the same as the direct sound, give us a sense of the room.
So they are like echos.

Most sound absorption materials absorb much more highs than bass, so eventually woth too much (but not 2 foot think panels like an anechoic) you end up with a dark sounding room. I guess you could EQ that back to flat, bit why not buy neutral sounding speakers to begin with?

Ah, no, I don’t think that is correct.
I thought that the preference was for the direct sound to be the same tonality as the reflected stuff?
(I may not have paid enough attention though…)

So EQ’ing would jack up the direct sound.

 

[tuga]

I'm old enough to have experienced mono as the majority option, and then the transition to majority stereo, and your question could be answered in two particulars - if a pair of less-neutral speakers has better pair matching than a pair of individually more-neutral speakers, its stereo illusion will be better; and if the more-neutral mono speaker achieved its apparently neutral spectral balance with the help of cabinet talk here and there, perhaps fortuitously filling in FR dips, which phenomenon can't be detected by on- or off-axis measurements, then pairing it with a similar speaker will produce a poor stereo illusion. Good mono usually scales to good stereo, but not quite always.

A speaker with off axis peaks and dips scores very poorly by Harman criteria.

But may be preferred nonetheless for the reason explained above (or a different one related to psychoacoustics)?

 

[tuga]

I don’t think ^that ^ is right?
Reflections that are the same as the direct sound, give us a sense of the room.
So they are like echos.

They give us a sense of the listening room, but we are trying to listen to the recording (or recorded space/spatial effect), so they are indeed distortion (inducing).

Stereo is an illusion. As such it requires from the listener a level of abstraction, but also some degree of enhancement seem to be favoured by many or we would all be listening to ultra-accurate equipment over headphones to remove any trace of euphonic distortion and room interference.

Room interference in the mids and treble is just as much distortion as below Schroeder. But the perceived effects (widening of images and soundstage, spaciousness, envelopment) sound nice(r), at least to some people.

 

[tuga]

Because any reflections are in fact noise. Whether we like the sound of reflections or not, they will always (perhaps insignificantly) mask the original recording. In my limited experience, adding absorption removes the fog reflections introduce when, for example, recreating the ambiance of the original recording. Certainly, adding more and more absorption has kept improving the clarity of reproduction in my experience. And I have never craved for reflected sound when listening through a good pair of headphones.

I don’t think ^that ^ is right?
Reflections that are the same as the direct sound, give us a sense of the room.
So they are like echos.

They give us a sense of the listening room, but we are trying to listen to the recording (or recorded space/spatial effect), so they are indeed distortion (reflections overlaid on the recording).

Stereo is an illusion. As such it requires from the listener a level of abstraction, but also some degree of enhancement seem to be favoured by many or we would all be listening to ultra-accurate equipment over headphones to remove any trace of euphonic distortion and room interference.

Room interference in the mids and treble is just as much distortion as below Schroeder. But the perceived effects (widening of images and soundstage, spaciousness, envelopment) sound nice(r), at least to some people.

 

[Blumlein 88]

But may be preferred nonetheless for the reason explained above (or a different one related to psychoacoustics)?

Which is back to saying "I can imagine" with nothing reasonable to back it up. You have off axis peaks and dips, they are heard differently than on axis and the spins take that into account. When you have those off axis issues the score is lower, and the design is seen to be less likely to sound neutral. Then there is the leap of logic, that hey, maybe, I can imagine, that it might sound preferred. Yet the data from Harman's tests is no it won't be preferred in all likelihood.

 

[tuga]

Well, please feel free to ignore the research. Others have found it useful. I've done several of these mono vs stereo comparison over the years and found stereo produces little new insight into listener preference, and usually adds more noise. Mono tests are simply more reliable and sensitive methods to assess the performance of a loudspeaker.

I found it useful. I just disagree with the points mentioned, which have mostly to do with how preference is assessed.
We are debating the usefulness of the Preference Rating for the consumer. I have no use for it myself, but would like to suggest that it needs perfecting in order not to be misleading for the general public.

Stereo is 1950s, and the research focus now is immersive audio.

But aren't the majority listening with earbuds or a monaural Bluetooth virtual assistants?

Also, when most recordings are put out in 2-channel, one must really enjoy space-effects a lot in order to put up with the fuzzy imaging and the phasey? distortion of upmix processing.

Is multi-channel really a goal for many in the US? In the UK, and Europe in general I suppose, sitting rooms are too small for 5.1, let alone immersive audio. Not to mention that a room will look like a low-budget sci-fi set, unless you are fortunate enough to own a dedicated room, again unlikely in Europe...

 

[Blumlein 88]

They give us a sense of the listening room, but we are trying to listen to the recording (or recorded space/spatial effect), so they are indeed distortion (inducing).

Stereo is an illusion. As such it requires from the listener a level of abstraction, but also some degree of enhancement seem to be favoured by many or we would all be listening to ultra-accurate equipment over headphones to remove any trace of euphonic distortion and room interference.

Room interference in the mids and treble is just as much distortion as below Schroeder. But the perceived effects (widening of images and soundstage, spaciousness, envelopment) sound nice(r), at least to some people.

The perceived effects of widening the images and soundstage spaciousness work best when the off axis response doesn't have peaks and oddities to draw attention to themselves in the room instead of letting you hear thru the room to the recordings. Sorry, you just keep backing up one step at at time to not agree with the data.

 

[voodooless]

Is multi-channel really a goal for many in the US? In the UK, and Europe in general I suppose, sitting rooms are too small for 5.1, let alone immersive audio. Not to mention that a room will look like a low-budget sci-fi set, unless you are fortunate enough to own a dedicated room, again unlikely in Europe...

I think the general idea is not to put speakers everywhere . Look at those multichannel sound bars. Those are all the rage! That is where the development frontier is.

 

[tuga]

Well I frankly don't understand your point. I found this graph, which I believe you posted, and is basis of your argument.
This is a study done by Floyd Toole in 1983 where he shows the sound quality and spatial ratings of three loudspeakers in mono and stereo.

The graphic tells us the listeners were less discriminating in stereo compared to mono, for both sound quality of spatial quality. Most of the difference in spatial ratings between tests can be attributed to the Quad, a dipole loudspeaker.

In mono, its colorations and higher directivity produced lower fidelity and spatial ratings. In stereo, the colorations were apparently less audible and it had higher sound quality and spatial ratings. As Floyd shows in the paper, the Quad's spatial ratings varied significantly with program material (unlike the others) where independent ratings were given. For choral and pop recordings its spatial ratings were last, and approximate its ratings given in mono. I wonder if its sound quality ratings also fluctuated with program. Unfortunately, it doesn't show this data, but it timbre tracked spatial that would indicate they are not independent.

First, it seems a bit selective to discount mono tests based on 1 test and 1 loudspeaker. I don't think any conclusions or generalizations can be made from one sample. I don't know if separate tests were done for sound quality and spatial ratings so the ratings were independent judgements. This would have been more work but minimize potential halo bias effects ( ratings for different attributes tend to be highly correlated with preference). In other words, did the change in timbre across tests influence the spatial ratings?

Also, the results of this test, did not convince Floyd to abandon mono testing and do tests in stereo. In fact, if anything it convinced him to stick to mono and abandon stereo because mono are more sensitive tests. The spatial differences in the stereo tests were largely isolated to one minimal and largely variable and attributed to the recordings.

His conclusions:

" Conducting the listening tests in monophonic and stereophonic modes revealed some important similarities and some interesting contrasts in the results. In general, assessments of sound quality were very similar in both modes, except for loudspeakers with significant imperfections, in which case the monophonic evaluations result in lower ratings than the stereophonic tests. In other words, in respect of Judging sound quality - the transduction accuracy of the loudspeaker - monophonic tests appear to be more demanding, or stereophonic tests less sensitive. "

Sorry, English is not my first language and I am not particularly eloquent.

What I am trying to say is that although mono testing is adequate for assessing speaker performance (indentifying issues in the transduction of the signal - distortions), it is unable to create the streophonic illusion.
And because most people will buy speakers to listen to music in stereo, for preference assessment stereo listening is in my view mandatory.
Also, mono testing doesn't take into account the positioning of the loudspeaker in relation to the room and its boundaries because it is placed in the centre of the testing room where early reflections from side-walls are minimal, far from what one will experience in a domestic setup.
Then there is the issue of room-interference below Schroeder: different speakers will require a different position to achieve the best balance in the bass and sub-bass, and mono testing speakers in the same position will not take that into account. Likewise, there is an optimal listener position so using a large audience of seats will not be optimal. And some speakers have different requirements regarding toe-in or distance to boundaries for correct performance.

 

[tuga]

The perceived effects of widening the images and soundstage spaciousness work best when the off axis response doesn't have peaks and oddities to draw attention to themselves in the room instead of letting you hear thru the room to the recordings. Sorry, you just keep backing up one step at at time to not agree with the data.

How do you know that?
Yes, peaks and dips off-axis do affect the tonal balance negatively but many people enjoy the perceived effect of a dip in the presence region, which is what a large woofer into a tweeter with no wave-guide will do (I am referring to sound power).

 

[tuga]

And an accurate tonal balance doesn't not seem to be a priority for the majority, since research seems to indicate that most untrained listeners enjoy much exaggerated low- and sub-bass, and up-lifted treble.

 

[tuga]

@Sean Olive
If you would be so kind…

How would rate the importance of:
- Olive score
- Compression
- Distortion
- etc

In the overall selection process?
(Personally I use more of an even mix of OS, Distortion, and maybe a touch deweighted on compression.)

I am leaving out looks, WAF-index, etc.

I am curious as what you'll do if the speakers you come up with using measurements and the preference score don't match your preference, or, in other words, don't "sound good" to you?