Dr. Toole, in case I have your ear for one last question...your mention of blind testing here brought up something I've been batting around with other folks on the forum:
I'd love to hear you chime in on the following issue:
As it relates to the research using Blind Testing for evaluating Loudspeakers:
What is the relevance of the results of Blinded Listening tests to to the sighted listening conditions under which most audiophiles will actually be listening to their system?
Do the results of the blind tests help predict listener satisfaction in the sighted conditions after purchasing a loudspeaker?
There seem to be interesting implications depending on the answers.
To flesh out the issues I'm getting at:
Scientific controls are the best way to get at reliable knowledge. We know from the research you cite (and tons elsewhere) how our perception can be influenced by various biases, so when you really want to understand something like "what type of sound do people prefer in loudspeakers?" using blind testing to control for bias effects makes all the sense in the world. The differences you've shown between results of sighted preference ratings and blinded ratings give reason to hold some caution, or skepticism, regarding sighted listening to speakers.
The problem I see on the horizon, though, is if we let our skepticism of sighted listening conditions (for audibly different devices like speakers) go too far we can verge on a sort of self-defeating hyper-skepticism. So for instance, we can show through all sorts of optical illusions that our sighted perception can be mislead. But it would be folly to infer from those conditions to "therefore our sight is totally unreliable." Such an inference would not explain all the ways it is clearly reliable-enough (e.g. how else do we manage to drive a car anywhere?).
Similarly, we can be put in conditions that show our auditory perception under "sighted/uncontrolled" conditions is fallible. But it would be a similar folly to leap from that to "therefore, unless under blinded conditions, our auditory perception is totally unreliable." That too couldn't make sense, since we successfully use our hearing all day long under "sighted" conditions. And any number of tests could show our hearing can be reliable 'enough' (e.g. if asked to discern between my wife's and Donald Trump's voice, I think we can expect I'd score %100 in both sighted and blinded conditions).
So my position is: while we are justified in being wary about confounding factors in our perception, and when we really want to get to the bottom of a phenomenon we will want scientific controls, we also have to keep in sight where our perception is "good enough/reliable enough" to accept under the more pragmatic, less rigorous conditions of every day use.
And this to me is where the question I asked earlier arises: If we are TOO skeptical about what we can accurately perceive from a speaker under sighted conditions, then usefulness of the blind test research will come in to question, when it comes to choosing speakers.
We won't be listening to our speakers under blinded conditions. If our perception of a speaker's sound characteristics is so distorted and swamped by sighted bias, then how could we "hear" or appreciate the sonic characteristics identified in the blind listening tests?
If there will be no meaningful relationship between sighted and blinded perception of the sound, then one may as well buy whatever sounds good under the sighted conditions in which you'll actually be listening.
As far as I can see it, the only way out of that apparent conundrum is to say that...actually...once we get that speaker home, even under sighted conditions we WILL be able to identify those sonic qualities that were apparent under blinded conditions. That is, at least, with *enough* accuracy and reliability to make the blind listening results relevant to our sighted listening conditions.
But, if we stick with a hyper-skeptical "you can NEVER rely on ascertaining the sonic character of a speaker under sighted conditions" then it seems the conundrum enters the picture, and the blind testing perception would seem irrelevant for regular audiophile consumers.
What do you think?
Thanks!
*(BTW, all the above does not address the issue of accuracy/Circle Of Confusion because that's a different issue than the one I'm getting at. I'm talking about the research regarding sound preferences when choosing speakers).
MattHooper said: “But, if we stick with a hyper-skeptical
"you can NEVER rely on ascertaining the sonic character of a speaker under sighted conditions" then it seems the conundrum enters the picture, and the blind testing perception would seem irrelevant for regular audiophile consumers.
What do you think?”
The situation you are focusing on is very much a real world one. As I interpret it, what you are asking is, in terms of the satisfaction of personal ownership, how much weight is placed on non-auditory factors? Obviously, the appearance, size, brand prestige, and price are factors that matter to most people.
I have a peculiar perspective on the topic, because it was blind tests that revealed the audible flaws with sufficient reliability and repeatability that it was possible trace them to measurable characteristics. It is not that sighted tests are useless, it is simply that blind tests yield substantially more repeatable (i.e. statistically useful), judgements from a wide population of listeners (most people with “normal” hearing). Nuisance variables had been attenuated. When I began in 1966, loudspeaker sound quality was all over the map, nothing I encountered was neutral. A popular demonstration of stereo “soundstage” was a train running from left channel to right channel. The “hole-in-the-middle” was a popular discussion topic. Primitive stuff.
My work was focused on sound quality, although in the early stages I included “preference” as a second rating category. It was immediately clear that the two ratings were the same; listeners “preferred” loudspeakers with high “sound quality”. Measurements indicated that highest ratings in both categories were given to loudspeakers with the fewest measurable defects, notably resonances, but occasionally non-linear distortion. It was all logical, and over the years repeatable with hundreds of listeners and loudspeakers, evaluated in different rooms.
BUT, how much does this matter to customer satisfaction and pride of ownership when appearance, size, brand prestige and price are included? Obviously, one would think that selecting a loudspeaker with minimal audible colorations is an essential starting point. These days one can find impressively neutral sounding loudspeakers from many brands, in many different sizes, exhibiting many different industrial design styles. There is a large choice, but how does one identify such loudspeakers? Sadly, customers are rarely if ever given the opportunity to judge sound quality by comparative listening in an unbiased (blind) situation, so non-auditory factors are involved. Even the methodology matters: If one adds in the desirability of having multiple (3 or 4) loudspeakers in the subjective comparison test, the opportunities for truly critical listening vanish. Too many uncontrolled variables are at play.
For this reason, I have devoted a substantial part of my life teaching the value of measurements. Personally, from the perspective of sound quality, I would trust my interpretation of a “spinorama” presentation of anechoic measurements more than my subjective judgment in a typical consumer listening circumstance. But this is sound quality, not “soundstage and imaging” when listening in stereo. I believe that most listeners would not choose to live with a “honky” or “nasal” coloration whatever the imaging characteristics, but each to his own. I have already elaborated on the idiosyncrasies and defects inherent in stereo listening in Post #504, indicating that there are trade-offs that we all have learned to live with – human tolerance and adaptation are wonderful. Once “neutral” timbre is identified the remaining factor affecting soundstage and imaging is loudspeaker directivity. Here there is a strong interaction with the recording itself – there are no standards for microphone technique and mixing for stereo imaging – and the listening room setup – there are no standards here either. This is truly a matter of personal preference, and both the recording personnel and the playback listeners are free to exercise their preferences.
Directivity is displayed in a spinorama, so it is possible to infer something about soundstage and imaging from the measured data. I mention this because omnidirectional, bipole and dipole loudspeakers have significantly distinctive positioning constraints compared to “generic” forward-firing loudspeakers. [An aside: an acoustic “point source” is a descriptor of a totally omnidirectional sound source, it is NOT a way to describe a forward-firing loudspeaker]. So, one’s preferences in imaging and music will influence the physical appearance of the loudspeaker itself and how it looks in a room. Neutral timbre would still be a necessary starting point.
Back to your question. It remains a problem for consumers that reliable unbiased listening tests are generally not available, so opinions about sound quality are to some extent contaminated by non-auditory factors. Some people may be more resistant than others to such influences, but questions remain. Trustworthy measurements are occasionally seen from manufacturers, and now spinoramas are appearing in audio forums and elsewhere. See
https://pierreaubert.github.io/spinorama/ for an impressive compilation of spinorama data from many sources. For those who trust measurements and who have taken the time to learn how to interpret them it is possible to gain significant insight into the inherent performance of the products. Learn to “read” the curves; don’t rely on calculated scores. When the product is in the home, though, what is heard is still greatly influenced by differences in recordings, the listening room setup and bass.
I keep coming back to bass, which is ALWAYS a major (approximately 30%) factor in our perception of sound quality and it is ALWAYS dominated by listening room resonances. Unless low-frequency performance is under some degree of control, a shadow is cast over the entire discussion above. This is an area that “regular audiophile consumers” routinely fail to address. Many, in their ignorance, avoid judicious equalization, or subwoofers or, especially, multiple subwoofers. Pity.