Why evaluating the sound of a single speaker is essential

[Floyd Toole]

Was there any formal research on stereo vs. mono listening tests or transferring the results from one to another, conducted by independent researchers such as universities, independent or publish institutions in recent 10 or 15 years? So far, I am aware only of what Harman and their affiliated researchers were concluding

I have been on vacation and am just catching up with this thread. I want to clarify what is meant when it is asserted that the research I and "affiliated researchers" was not "Independent". The referenced research began in 1966 at the National Research Council of Canada (NRCC) - a taxpayer funded research establishment with no manufacturing or marketing departments. This was an independent research activity. Others in universities and companies may have wanted to conduct such research, but they lacked the physical facilities and specialized apparatus to do so, and the paid professional staff that could devote many months of non-profit-making time to the task. I had it all - an anechoic chamber, an especially constructed listening room, and educated full-time staff and colleagues to assist in the effort. Our product, upon which we were judged, was knowledge, published in refereed scientific journals. The data being discussed in this thread was published in 1985 -40 years ago! - all done at the NRCC, years before I was invited to join Harman in 1991. There I set up another research group, that was permitted to continue publishing for the benefit of Harman's competitors. Harman spent over $1M in facilities alone for this research, which in retrospect I find utterly remarkable. This may explain why there is a near complete absence of comparable "Independent" research on stereo vs mono imaging in the past 40 years. The price tag is high. Opinons are abundant, though, and much cheaper.

The good news is that as a result of this ancient research loudspeakers that are basically timbrally neutral are now increasingly available - Amir's measurements identify them. Imaging in mono, when listeners responded with answers to the question, was a surprise at the time. The best loudspeakers drew less attention to themselves, somewhat "disappearing" behind the blind screen. This has to be a good starting point for any form of imaging, stereo or multichannel.

In stereo the only soundstage images that can in any direct sense be correlated with loudspeaker directivity are the hard left and hard right images that occur from either amplitude panning in consoles and coincident microphone arrays at a recording site. These are essentially mono sources and adjacent boundary reflections tend to 'soften' them. Spaced stereo mics add inter channel time difference and the result is that there are no purely mono sound sources - a very high percentage of "classical", or "purist" recordings are done in this manner. In all recordings all phantom soundstage images are comb filtered by the acoustical crosstalk appearing at the ears, so both sound quality and spatial perceptions are abnormal - corrupted in fact. Consequently, recordings have an enormous influence on one's perceptions of soundstage and imaging. The 4th edition of my book discusses this topic, but the situation should be obvious once you think about it. What we perceive includes the recordings, and there are no standards for these - it is art. The problems arise when listeners are disappointed with the art, and seek solutions in the playback apparatus. Loudspeakers in small rooms are problems when the listener expects large room perceptions.

Try multichannel or binaural . . .

 

[Thomas_A]

The 4th edition of my book discusses this topic, but the situation should be obvious once you think about it. What we perceive includes the recordings, and there are no standards for these - it is art. The problems arise when listeners are disappointed with the art, and seek solutions in the playback apparatus. Loudspeakers in small rooms are problems when the listener expects large room perceptions.

Try multichannel or binaural . . .

I agree - that there is no standard for recordings - just art. Multichannel or at least three-channel should be the standard, but stereo is still dominating.

(And it is a difficult task to reproduce the signal at the ear-drum as the mixer/mastering engineer once had when the production was finished. For two-channel there is are compromises, and what is the really the "correct" ratio of direct to indirect sound at the listening position? If I dial in my quite wide-dispersing speakers for 45° toe-in, the is a dry center focus almost as listening to mono speakers; having 58° gets a bit more ambience and envelope. If on-axis response is linear (within ±1.5 dB, with no broad-band dips/peaks) it is good, IMO.)

 

[MAB]

Interestingly, on three different occasions, some recording engineers completely independent from each other, found massively flawed sound quality within these speakers, leading to the most negative feedback I have ever witnessed in such circles.

What speakers are you talking about?

 

[MattHooper]

Thank you.

However, I was aware of everything you wrote about, and I was separating out the notion of “ personal preference” from the possible facts of “ what a loudspeaker actually sounds like” and asking if there is research showing whether people tend to converge in agreement on what a speaker sounds like, as expressed by their descriptions.

“I prefer speaker B” doesn’t tell us what
“ speaker B sounds like.”

As I pointed out, I can differ with somebody in terms of preference for a speaker, while agreeing with them on sonic characteristics of that speaker (eg boosted upper mids, dip in upper bass, spacious, open sounding etc). Just as a friend and I can both try a recipe that we agree is quite “spicy” but he may prefer that and I may not.

And again, I’m aware that the type of blind testing you were engaged in showed a large degree of convergence in preferences, but that’s not my question. It’s more along the lines of “ if asked to describe the characteristics of the sound they are hearing, what level of convergence (or divergence) is found among those reports?”

Floyd discusses in his book the difference in a reviewer talking about good frequency response or very low distortion vs the more superfluous statements such as : the liquid midrange was transparent while sense of timing was euphoric. One describes what is technically good while the other tells more about the persons emotional response.

Well, even terms like “ liquid midrange” and
“ transparent” aren’t necessarily merely
“ emotional reactions” - they could be attempts to describe actual sonic differences between loudspeakers.

For instance if there’s distortion or ragged frequency response in the upper mid range of loudspeaker A but very low distortion in the same range in speaker B, there’s nothing in principle wrong with describing these differences with terms like “A’s midrange sounded ‘rough’ or ‘coarse’ ‘harsh’ ‘grainy’ etc, whereas speaker B’s upper midrange and highs could be described as ‘smoother’ more ‘liquid’ etc.

So we’re not necessarily talking about emotional reactions as much as attempts to put the real subjective consequences of different loudspeakers into words. After all we want to be able to correlate loudspeaker measurements to “ how it sounds.”

Conflating "feelings" or 'impressions" without examining the science (physics of sound) leads you down a path into myths and faith based beliefs which may or may not be true. It is not reliable and difficult to repeat.

But IF you apply scientific controls to the question: are sonic impressions (not just preferences !) still difficult to repeat?

Cheers.

 

[Kal Rubinson]

Multichannel or at least three-channel should be the standard, but stereo is still dominating.

3-channel should have been the stereo standard from the beginning but it is irrelevant today.

 

[MattHooper]

3-channel should have been the stereo standard from the beginning but it is irrelevant today.

Perhaps sonically, but three channel would’ve been a significantly more awkward and difficult proposition for most people - both in terms of finding a place for the third speaker and arriving at a good level of coherence across the three speakers.

 

[Kal Rubinson]

Perhaps sonically, but three channel would’ve been a significantly more awkward and difficult proposition for most people - both in terms of finding a place for the third speaker and arriving at a good level of coherence across the three speakers.

Sure but the real reason it was not offered is that, back then, there was no delivery system for it.

 

[MattHooper]

Sure but the real reason it was not offered is that, back then, there was no delivery system for it.

Which doesn’t seem to me to change the issue that I pointed out.

I mean, there have been delivery systems available for ever more elaborate surround sound, but there’s practical reasons why most people haven’t gone to the length of setting up surround system systems to take advantage of it.

It’s interesting to contemplate the counterfactual of what consumer sound systems would look like through the years if 3 channel had been promoted instead of stereo.

 

[kyuu]

Which doesn’t seem to me to change the issue that I pointed out.

I mean, there have been delivery systems available for ever more elaborate surround sound, but there’s practical reasons why most people haven’t gone to the length of setting up surround system systems to take advantage of it.

It’s interesting to contemplate the counterfactual of what consumer sound systems would look like through the years if 3 channel had been promoted instead of stereo.

I don't think setting up 3 front channels is equivalent to even 5 channel surround. Hooking up an additional front channel is relatively simple, it's getting things around to the back that typically cause the most issues. And if we're going by what "most people" are willing to spend the time and effort on, we might conclude that mono is the winning system as that's probably the most common way people listen nowadays (phone/tablet speakers, portable bluetooth speakers), with the exception of car audio.

 

[xanalog]

3-channel should have been the stereo standard from the beginning but it is irrelevant today.

idem for RedBook.

 

[Floyd Toole]

However, I was aware of everything you wrote about, and I was separating out the notion of “ personal preference” from the possible facts of “ what a loudspeaker actually sounds like” and asking if there is research showing whether people tend to converge in agreement on what a speaker sounds like, as expressed by their descriptions.

The multiple-loudspeaker double-blind evaluations focus the listener's attention on audible differences that distinguish the loudspeakers in terms of sound quality. A very large number of such evaluations over 50+ years shows that the dominant factor is resonances. Resonances exist in different Q factors, and the low-Q, broadband, ones are the most easily detectable, at deviations in frequency response of less than a dB. So, the result is that one can say that there is agreement in that context about what constitutes a good loudspeaker, and it is recognizable in the right set of anechoic measurements - not room curves (sorry, but I had to add that). The most favoured loudspeakers all approach smooth and flat on-axis frequency responses. The direct sound is a dominant factor in evaluating sound quality. Clearly it is timbral neutrality that is being looked for by listeners, and the program itself turns out not to be a critical factor so long as it has the necessary spectral density and bandwidth to excite the resonances in the loudspeakers. In such well controlled evaluations the listening room is constant, bass quality is normalized, and the interactions with boundaries are also constant.

If you look at the questionnaire used in the stereo vs mono tests published in 1985 (in my books as well) you will see that listeners had lots of opportunity to describe what they heard in subjectively descriptive and technical terms. The analytical comments on spatial perceptions related to soundstage and imaging were quite detailed as displayed in a separate figure. It is evident that stereo is not an unambiguous differentiating factor among the loudspeakers.

The descriptions offered by listeners in these tests almost always refer to audible problems. The highest rated loudspeakers tend to be those with the fewest comments. Clearly these are not conventional "speaker reviews". When efforts were made to get listeners to focus separately on sound quality and spatial quality, the differences were surprisingly small. There may be a form of perceptual merging.

However, if one takes even the most neutral loudspeaker, as identified in such subjective tests or by measurements, and listens in isolation, no comparisons, the results can be different, for different reasons. First, the room cannot be normalized, so room resonance problems become consequential and bass performance accounts for about 30% of overall sound quality ratings. All our rooms and room setups are different, so opinions will predictably vary. Any collection of musical selections from various sources will reveal some that need a bit of bass boost or cut to be gratifying - at least that has been my experience over the years. Some are deliberately hyped for popular consumption, but others are inexplicably bass shy - too much bass in the control room? Treble balance also can vary. This is art, but business interests and egos get into the act. Neutrality is not always the goal in control and mastering rooms. Over the years recording control rooms have been generally relatively dead spaces. International standards specify quite low values: ITU-R BS.1116-3 targets 0.25s. Add to this the current fashion of near-field, close listening, and those in the positions of decision making are listening to a dominant direct sound. Having seen, and published measurements on, several popular studio monitor loudspeakers over the years the explanation of varying spectral content in recordings is evident. Add to this the inevitable variations when "custom room EQ" is invoked and it is surprising that things sound as good as they do. We humans are remarkably adaptable in our determinations of what, as you ask, loudspeakers actually sound like. Olive and Welti did such a method-of-adjustment of bass and treble tone controls, finding that inexperienced listeners could prefer considerable bass and treble boost, while experienced listeners remembered what neutral sound was like and adjusted accordingly. So, one must conclude that "it depends . . ."

 

[Floyd Toole]

with the exception of car audio

And in cars nobody sits in the stereo seat, which is why upmixing to multichannel is common.

 

[Sokel]

What's my question about evaluating speakers in mono is about the music itself.
It must be a mono recording all the way, isn't it?

We have seen lately that even with music which bass appears as mono under certain freqs, its phase varies wildly between channels even down to 10Hz.
So, depending the channel used if music is stereo, evaluation can differ by phase interaction with room alone.

I'm I wrong at this?

 

[Cbdb2]

But would it? Dr Toole's posts, above, seem to suggest that:-

1. If we prefer Speaker A over B in a single-speaker test then we will prefer it in a two-speaker test.
2. Spatial preference (which I presume is where your term 'stereo imaging' falls) doesn't flip from A to B when we add a second speaker.

He has also suggested elsewhere that doing it with sighted listening is almost self-defeating, if the goal is to know what the sound waves themselves sound like. And suggested that he would rather evaluate a speaker from its Spinorama than by sighted listening.

cheers

There is one reason to test/listen to a pair. If your 2 speakers have frequency response differences, (they always do) that are large it will blur your sound stage. Genelec guaranties .5 db max difference 20 to 20k for this reason.

 

[Jaxx1138]

Thank you.

However, I was aware of everything you wrote about, and I was separating out the notion of “ personal preference” from the possible facts of “ what a loudspeaker actually sounds like” and asking if there is research showing whether people tend to converge in agreement on what a speaker sounds like, as expressed by their descriptions.

“I prefer speaker B” doesn’t tell us what
“ speaker B sounds like.”

As I pointed out, I can differ with somebody in terms of preference for a speaker, while agreeing with them on sonic characteristics of that speaker (eg boosted upper mids, dip in upper bass, spacious, open sounding etc). Just as a friend and I can both try a recipe that we agree is quite “spicy” but he may prefer that and I may not.

And again, I’m aware that the type of blind testing you were engaged in showed a large degree of convergence in preferences, but that’s not my question. It’s more along the lines of “ if asked to describe the characteristics of the sound they are hearing, what level of convergence (or divergence) is found among those reports?”



Well, even terms like “ liquid midrange” and
“ transparent” aren’t necessarily merely
“ emotional reactions” - they could be attempts to describe actual sonic differences between loudspeakers.

For instance if there’s distortion or ragged frequency response in the upper mid range of loudspeaker A but very low distortion in the same range in speaker B, there’s nothing in principle wrong with describing these differences with terms like “A’s midrange sounded ‘rough’ or ‘coarse’ ‘harsh’ ‘grainy’ etc, whereas speaker B’s upper midrange and highs could be described as ‘smoother’ more ‘liquid’ etc.

So we’re not necessarily talking about emotional reactions as much as attempts to put the real subjective consequences of different loudspeakers into words. After all we want to be able to correlate loudspeaker measurements to “ how it sounds.”



But IF you apply scientific controls to the question: are sonic impressions (not just preferences !) still difficult to repeat?

Cheers.

But of course in engineering and scientific research we quantify things for a more definitive description. What may be “liquid smooth” to the laymen is “low distortion , linear directivity index “ to the engineer or scientist. Objective vs subjective.

 

[Thomas_A]

3-channel should have been the stereo standard from the beginning but it is irrelevant today.

Yes, agree fully.

 

[Kal Rubinson]

Which doesn’t seem to me to change the issue that I pointed out.

As I said, sure. I was not questioning your point but adding on.

I mean, there have been delivery systems available for ever more elaborate surround sound, but there’s practical reasons why most people haven’t gone to the length of setting up surround system systems to take advantage of it.

Yup. All of audiophilia is a niche and multichannel is a niche of that for many practical reasons.

 

[Duke]

... if one takes even the most neutral loudspeaker, as identified in such subjective tests or by measurements, and listens in isolation, no comparisons, the results can be different, for different reasons. First, the room cannot be normalized, so room resonance problems become consequential and bass performance accounts for about 30% of overall sound quality ratings. All our rooms and room setups are different, so opinions will predictably vary.

This is taken from the notes I made after reading Section 7.6.2 in the 3rd edition (my copy of The Book is out on loan so I can't double-check right now):

A study was done in which binaural recordings were made of three different cone-n-dome direct-radiator loudspeakers in four fairly dissimilar (but not atypical) rooms. This way the listeners didn't need to actually be in the rooms. When the scoring was compared for the three speakers recorded in a given room, their preference rankings were consistent from room to room. But when the evaluation scoring was processed to factor in the rooms themselves, the room (or, in other words, the speaker/room interaction) is what dominated the preference scores!

Please correct me if I have misunderstood.

And if not, then might there be possible utility in exploring approaches to speaker design which deliberately seek to improve, or enable useful optimization of, the speaker/room interaction? For example, suppose the largest room had the best preference scores. If it were possible to get speakers to interact with their room as if the room was larger than it actually is, might that be something worth exploring?

 

[sceptical1]

My 5.1 mid-level system (box store quality) is far superior to my 2 ch. low distortion amplified speakers designed and performing according to Toole's research. Also, this is 2 ch. hi resolution sources vs. a compressed dts music upmix.

The surround system is simply producing a sound that is so completely effortless that I can not be distracted by every stupid characteristic associated with accurate stereo reproduction. That is, what distinguishes a product as superior based on objective measurement and the crippling effect of the room restricting, what everyone considers a reasonable way to be tricked into thinking that stereo can become 3 dimensional.

I am not saying that this is impossible, only that you can get the same performance in surround at 10% of what a defective means of reproduction can offer and then only when you manage to come across that perfect recording 1% of the time.

If Toole (Rubinson) is making a case for the superiority of surround vs stereo they do it based on the same experience of validating what is preferred, and what most of you agree, for accuracy in reproduction of loudspeakers made over those 40 years of testing. (Although, if Toole uses Revel towers and Rubinson uses Kef blades for surround I don't get it, this is an obnoxious oversight unless this is what is needed to properly evaluate stereo recordings, according to them). ??

The thing is, surround methods and reproduction are inherently a means of accuracy by way of appealing colorations and non-specific localization, sounds are large and diffuse because they are 3 dimensional by reconstruction of the playback making direct sound primary and reducing the effects of the room for a consistent quality not capable with stereo.



Just a thought to distract me from this endless talk of the correct dispersion profile that matters not compared to Toole's work toward correct bass response in-room at the level needed to really bring a live performance to the listener, even in stereo.

 

[sceptical1]

And in cars nobody sits in the stereo seat, which is why upmixing to multichannel is common.

Everyone is listening near field so everyone is happy, no matter the mix.