A Broad Discussion of Speakers with Major Audio Luminaries

[krabapple]

The word is inherently (not 'completely and totally'). Sighted eval is inherently unreliable. As in, some degree of unreliability is unavoidable in the method. This is why scientific audio research abjures faith in sighted methods.

It doesn't mean sighted calls can't ever be right. The level of uncertainty can vary depending on conditions. For example , when 'the difference in sound is great enough to overcome residual bias (quoting Dr. Toole), a sighted call of 'difference' is likely to right. Sighted calls are (more ) reliable under THAT condition, than one where the difference is small.

That great difference in sound will of course be independently MEASURABLE as a great difference in sound. That's where audiophiles stumble. They just claim ' it sounded obviously different to me' when the other evidence they've given provides no reason to consider that likely.

 

[MattHooper]

I’m presuming you meant, like MarkS, “sighted listening” (which can be subject to sighted bias).

More than one poster here observes that sighted bias is completely and totally unreliable. Please include me as well.

I’m fascinated by what seems to read as a pretty extreme position.

I’m wondering what you mean exactly.

There’s a position along the lines of: sighted listening perception can be subject to error due to bias effects. Therefore, I prefer to rely on measurements and/or data drawn from listening under conditions controlling for those bias effects.

Basically this position amounts to: sighted listening is unreliable in the sense that it is not reliable enough for the level of certainty and reliability I’m seeking.

Is that essentially what you mean?

Which seems an eminently justifiable position to take.

But your statement seems more extreme than that, more absolute.

When you say that sighted listening is “completely and totally unreliable” do you mean something else something stronger?

For instance, do you mean that our perception in sighted listening is so reliably skewed by bias effects that it simply cannot be trusted at all and is completely unreliable on those grounds?

And that no reasonable inference can be drawn under such conditions?

If you mean, something more like that… it seems like a rather extreme form of scepticism that can’t hold up very well to inquiry. It seems to run to all the issues and questions that has been raised for such a claim.

The first is that it implies a pretty extreme scepticism about our perception that would make lots of every day life inexplicable.

Secondly, if that’s the case, why care about how a speaker measures at all (or blind test results)? If the sound depicted by those measurements will be reliably distorted under the sighted conditions, what use or relevance are the measurements for selecting loudspeakers to use under normal conditions?

Third: how does your position account for examples like Erin (I often bring up here because the evidence is publicly available)?
Erin, in his sighted listening impressions before he measures speakers, tends to detect various sonic characteristics that show up in the measurements - frequency response characteristics, dispersion characteristics etc. How would that be possible on your highly sceptical position regarding sighted listening? Is it your position that no inferences from sighted listening could be justified? (Isn’t some nuance available in terms of confidence levels from lower to higher, rather than extreme “ confidence” versus “ zero confidence”?)

Cheers

 

[MattHooper]

The word is inherently (not 'completely and totally'). Sighted eval is inherently unreliable. As in, some degree of unreliability is unavoidable in the method. This is why scientific audio research abjures faith in sighted methods.

It doesn't mean sighted calls can't ever be right. The level of uncertainty can vary depending on other conditions. For example , when 'the difference in sound is great enough to overcome residual bias", a sighted call of 'difference' is likely to right. Sighted calls are (more ) reliable under THAT condition, than one where the difference is small.

That great difference in sound will of course be independently MEASURABLE as a great difference in sound. That's where audiophiles stumble. They just claim ' it sounded obviously different to me' when the other evidence they've given provides no support to consider that likely.

Broadly speaking, that seems right to me.

 

[Sal1950]

More than one poster here observes that sighted bias listening is completely and totally unreliable. Please include me as well.

Maybe a bit extreme on the surface but still correct in the absolute.
Sighted listening impressions that aren't backed up with measurable results are just a guess and completely under the control of said listeners preferences.

Third: how does your position account for examples like Erin (I often bring up here because the evidence is publicly available)?
Erin, in his sighted listening impressions before he measures speakers, tends to detect various sonic characteristics that show up in the measurements - frequency response characteristics, dispersion characteristics etc.

There are a few "golden ears" out there that you might be able to trust, but without the measurements how would you know if he's having a good or bad listening day, we all have them. Erin, like Amir, Floyd and others have spent decades both listening and correlating what they hear with what the meters say. But going back to the "trustworthiness" of sighted reports made without any scientific evidence, these reports remain "totally unreliable". Many THINK they know what they're hearing, but do they?

 

[j_j]

The word is inherently (not 'completely and totally'). Sighted eval is inherently unreliable. As in, some degree of unreliability is unavoidable in the method. This is why scientific audio research abjures faith in sighted methods.

It doesn't mean sighted calls can't ever be right. The level of uncertainty can vary depending on conditions. For example , when 'the difference in sound is great enough to overcome residual bias (quoting Dr. Toole), a sighted call of 'difference' is likely to right. Sighted calls are (more ) reliable under THAT condition, than one where the difference is small.

That great difference in sound will of course be independently MEASURABLE as a great difference in sound. That's where audiophiles stumble. They just claim ' it sounded obviously different to me' when the other evidence they've given provides no reason to consider that likely.

This is a pretty good way to put it. If the subject knows the test conditions before listening, the subject will integrate that information. It's part of being a human being. We survived and are good at handling situations BECAUSE we are so good at integrating everything we sense now, sensed before, know, and expect. It's a part of reducing all the sensory information by the necessary factor of a million or such into actual intellectual expression, i.e. a decision "i heard", for example.

 

[krabapple]

And there are a half dozen caveats and conditions I could have added...like, while reporting simple difference, sighted, is pretty reliable when the difference is great (though the question becomes...what constitutes great?), relying on sighted eval to report sound preference is hugely more fraught even when the sonic difference is great. Two different speakers can plainly sound different, but still need to be preference-tested blind, such are the effects of appearance, brand, knowledge of price...

 

[j_j]

And there are a half dozen caveats and conditions I could have added...like, while reporting simple difference, sighted, is pretty reliable when the difference is great (though the question becomes...what constitutes great?), relying on sighted eval to report sound preference is hugely more fraught even when the sonic difference is great. Two different speakers can plainly sound different, but still need to be preference-tested blind, such are the effects of appearance, brand, knowledge of price...

Well, yes, this goes back to my long ago experience with somebody's borrowed '9 ohm matched speaker cables' that had enormous capacitance, and the amplifier that could not handle that. The sudden popping noise from the amplifier and the very unhappy power transformer sounds were definitely auditory stimulii, and were very different than the intended Tanya Tucker. The silence that followed, likewise, accompanied soon after by the sight and smell of the "magic smoke" that was leaking out of the poor, unfortunate power amplifier. Absence of sound, or great semantic difference can be detected without a blind test.

But "how great" is a very sticky wicket. It's based on shifting attention when you know what you're listening to. If you shift attention to some other part of what you're experiencing, you WILL remember a difference. You have no idea if that is the result of the SOUND or the listening experience, or purely random factors.

 

[MattHooper]

And let’s not forget, we can take “Unbiased data” (whether it’s raw data, or from research studies), and still interpret that in biased ways. Nobody being immune to this.

(Sometimes it can feel like appealing directly to data or to certain studies automatically puts the Truth Stamp on our arguments. We might be unaware of the role our own interpretation of the implications are playing - biased slips in reasoning etc).

 

[Tim Link]

To make the results work perceptually in a real world listening situation, the use of blind testing suggests we should not be looking at the speakers while listening, instead keeping them hidden behind some kind of blind. Otherwise, our enjoyment may be effected by what our eyes are telling us we should be hearing. If we are going to be seeing the speakers under normal listening conditions, then we should take in to account how seeing them affects our perception of the sound. So blind testing for blind listening. Sighted testing for sighted listening. Does this not make sense?

 

[j_j]

To make the results work perceptually in a real world listening situation, the use of blind testing suggests we should not be looking at the speakers while listening, instead keeping them hidden behind some kind of blind. Otherwise, our enjoyment may be effected by what our eyes are telling us we should be hearing. If we are going to be seeing the speakers under normal listening conditions, then we should take in to account how seeing them affects our perception of the sound. So blind testing for blind listening. Sighted testing for sighted listening. Does this not make sense?

The problem is that this can lead to expectation bias that may lead the sighted listener far, far astray.

For instance, somebody may like the nice walnut cabinets, and that's fine, BUT then they may notice the "midrange problem" or whatever. Meanwhile the MDF cabinets coated in truck bed liner vinyl might actually sound a lot better, when "better" is unanimous in blind testing. Of course, it's also your living room, and your spouse's.

Just speaking from observations, here.

 

[Salt]

The problem is that this can lead to expectation bias that may lead the sighted listener far, far astray.

For instance, somebody may like the nice walnut cabinets, and that's fine, BUT then they may notice the "midrange problem" or whatever. Meanwhile the MDF cabinets coated in truck bed liner vinyl might actually sound a lot better, when "better" is unanimous in blind testing. Of course, it's also your living room, and your spouse's.

Just speaking from observations, here.

You are very polite, noticable.
But Your're right to give a chance for those who might need some hint to correct some 'subjective impression'.
If this is the way ASR will go along into the future I really appreciate this, instead of 'understand or leave', what is, on the other hand, understandable by those who know ... but not for those masses who enter into this 'sphere' (or bubble).

 

[MarkS]

Quoting Floyd Toole from earlier in this thread:

If I have a final word on the topic let it be: The factors influencing opinions formed in all listening tests involve acoustical and non-acoustical factors. Controlling the auditory factors is difficult. Comparisons require equal loudness, rapid changes, and positional substitutions if room effects are to be normalized. When this is done in double-blind fashion there are decades of results showing usefully reliable opinions from hundreds of listeners of all backgrounds. The principal factor causing statistical variations has been hearing loss.

Sighted evaluations introduce the possibility that opinions will be biased by non-auditory factors. The extent of the bias depends on the nature and importance of the non-auditory factors to the individual listeners. [emphasis added]

In science one does all that is possible to eliminate biasing factors. End of story. Cheers!

 

[MattHooper]

To make the results work perceptually in a real world listening situation, the use of blind testing suggests we should not be looking at the speakers while listening, instead keeping them hidden behind some kind of blind. Otherwise, our enjoyment may be effected by what our eyes are telling us we should be hearing. If we are going to be seeing the speakers under normal listening conditions, then we should take in to account how seeing them affects our perception of the sound. So blind testing for blind listening. Sighted testing for sighted listening. Does this not make sense?

I think it makes sense to a certain degree, but the liabilities are obvious (as JJ just pointed out). That path also leads to all the pseudoscience and woo-woo and snake oil that has infected “ high end audio” for so long now.

As I remember Floyd Toole remarking: in terms of his own approach evaluating speakers, he would be much more comfortable basing evaluation on a good suit of measurements, rather than on his uncontrollled subjective impressions, in unfamiliar conditions (eg like an audio store).

Which makes perfect sense. If you understand measurements and you know what you’re looking for that can be good enough.

I personally navigate this issue with the basic heuristic of “extraordinary claims require extra extraordinary evidence.”

Comes down to personal comfort levels with uncertainty. If a proposition is technically dubious or implausible, my demand for measurements or blind testing goes up.

Long ago when I was given a number of expensive and well reviewed AC cables to evaluate, in sighted listening it seemed that at least one of the cables made a Sonic difference in my system.

If I had gone strictly with the “ if I seem to hear it in sighted listening that that’s good enough for me” approach, I suppose I could’ve purchased that cable.

But since I knew this was technically dubious I had an engineer friend helped me do a blind test between the expensive cable and the cheap off the shelf AC cable I was using with the gear in question. The result was that once I didn’t know which cable I was listening to I couldn’t detect any difference at all.

I was happy to have that learning moment and to abandon the idea of spending lots of money on AC cables.

I’ve mentioned before that my perception of the sonic character of my tube amps has been so persistent and reliable for 25 years, despite many dalliances with solid state, that if it’s purely a bias effect it doesn’t seem I can overcome the impression, and I find it enhances my listening pleasure. In which case I’m fine just rolling with it, understanding that there’s a level of uncertainty in my evaluation in terms of what is actually going on. (Though as I’ve mentioned I did successfully identify at least my tube preamplifier against the state in blind testing).

When it comes to loudspeakers, the prospect of hearing differences become even more plausible. For my own purposes, since I’m not approaching my hobby strictly as a science project, I don’t mind drawing conclusions under sighted listening. With lower confidence levels built-in to my conclusions.

Plus, speakers are something that I’m going to be staring at when I’m listening. I have noticed over the years that my sighted impressions of loudspeakers stay very consistent. Therefore if there is a sighted bias component mixed into my perception of the sound, then I’m fine with that. It’s one reason why I prefer to audition speakers in person - listen to them essentially the way I will be using them - rather than rely purely on measurements. (Though I’m always happier if measurements are available as well).

 

[gnarly]

My problem with my own bias is not so much with sighted.... Heck, I honestly don't think I have any real visual bias.....(other than maybe having a negative bias towards speakers that are clearly about eye candy)

My problem is having an expectation bias that good measurements have to sound good.
All my DIY speakers are fully tuned via measurements before any listening and I definitely have bias in how well they measured vs how well they will sound (at first listen).
And then after an extending listening period, as I experiment with different tuning techniques such as IIR vs lin-phase FIR, there's no question my "sight", my bias, is about measurement expectations.

No escaping these mellons of ours....

 

[RexrothPigeon]

More than one poster here observes that sighted bias listening is completely and totally unreliable. Please include me as well.

I too am Spartacus

 

[RexrothPigeon]

More seriously, the pro-sighted listening posters seem to think that when we say "sighted listening is inherently unreliable" we mean "sighted listening is always inaccurate." That's a bad misreading, and a telling one. For instance, if I were to say that fortune tellers are inherently unreliable, I think most would agree while also understanding that occasionally a fortune teller makes an accurate prediction, just as some people who guess lottery numbers (accurately) guess the right ones.
(Obviously there's a spectrum here. Sighted listening, even when performed by septuagenarians, isn't as random as guessing lottery numbers.)

 

[MattHooper]

More seriously, the pro-sighted listening posters seem to think that when we say "sighted listening is inherently unreliable" we mean "sighted listening is always inaccurate." That's a bad misreading, and a telling one.

But often enough people here really are making statements that read as fairly extreme sceptical positions on sighted listening - where implicit or explicit in the statements is the proposition that sighted listening perception is inevitably polluted by biases.
And to a degree that any sighted listening report cannot have any information value for anyone else, and sometimes not even information value the individual.

I mean, I was just involved in a conversation where somebody on ASR was explicitly claiming that subjective descriptions derived from sighted listening cannot be meaningful for anybody else but the individual’s own perception. Which is a pretty extreme sceptical position to take. And others have taken pretty extreme sceptical positions on the purported uselessness of subjective descriptions.

Now, if people’s views are actually much more nuanced, that would be fine. Then I wish they would just be more clear about what they’re actually saying instead of being more sloppy about it.

I found on ASR that there really is a sort of POE’s LAW problem in this respect - some people will say something that implies unwarranted levels of scepticism, and may draw back from that claim in discussion, where others may fight tooth and nail for that ultra sceptical position. So you never know and it’s hard for you to speak for everyone.

I’m never trying to put words in people’s mouth myself; I’m typically trying to draw out somebody’s actual view so that I understand what they actually believe.

For instance, if I were to say that fortune tellers are inherently unreliable, I think most would agree while also understanding that occasionally a fortune teller makes an accurate prediction, just as some people who guess lottery numbers (accurately) guess the right ones.

If you want to talk about a “telling” example, that would be one! Yes, I saw the caveat you gave in parenthesis afterwards . But the impulse was to immediately compare sighted listening to randomness.

And that’s not very unusual here. For instance, when I pointed to subjective reviewers identifying characteristics that show up in the measurements (for instance in Stereophile) it’s sometimes waved away with responses like “Sure even a stopped clock is right twice a day.” There’s often enough no serious attempt to seriously account for this. It’s easy to just hand, wave it all away and put it all in the same bucket of ‘b.s. poetry and imagination.’
And since the subjective review industry is Public Enemy Number One around here, it seems many people do not want to give any quarter.

I mean, there’s plenty of well justified arrows to sling at that crowd. But sometimes it seems reasonable scepticism slides into mere cynicism.

(Obviously there's a spectrum here. Sighted listening, even when performed by septuagenarians, isn't as random as guessing lottery numbers.)

Agree.

And that’s the spectrum that I keep bringing up. Which I find interesting: given most audiophiles are not scientists able to do scientifically controlled vetting of gear, and given there’s plenty of speakers out there that audiophiles are interested in that do not have Klippel measurements… what is a reasonable way to traverse this terrain? It seems very often there’s just standard “ listen for yourself and draw your conclusions.” If no reasonable inferences can be drawn this way, then what’s left? And what would it say about the status of our listening impressions at home and the relevance of measurements?
But if reasonable inferences can be drawn this way, then what type of reasonable inferences and why?

I’ve given my answer to this question many times. I understand why some people aren’t interested - they just will go only to gear, which has the appropriate set of measurements. But then I still think it’s worth being careful about dismissing any worth or usefulness of informal listening impressions, or communicating about those.
I think it’s worth being careful about moving from “ those methods are useless for my purposes” to generalizations of “ those methods are useless.”

Cheers.

 

[RexrothPigeon]

And that’s the spectrum that I keep bringing up.

Yes, I always assumed this was obvious, but it has no effect on whether or not sighted listening is reliable, since there's no reliable way to determine its relationship to accuracy. This means that while of course sighted listening impressions aren't the same as random number guesses, they are of similar reliability. (Again: I'm not saying they are of similar accuracy, but that their accuracy is unknown and thus unreliable).

And if we also factor in (as we must) the undeniable fact that subjective impressions have been weaponized for decades in the service of foisting undiluted snake oil on consumers, well, I'm not sure there exists a level of skepticism (or even cynicism) that is truly unmerited.

 

[RexrothPigeon]

given most audiophiles are not scientists able to do scientifically controlled vetting of gear, and given there’s plenty of speakers out there that audiophiles are interested in that do not have Klippel measurements… what is a reasonable way to traverse this terrain?

We all have our own tolerances for uncertainty, for spending, for accuracy, and for legwork, of course, and I have no prescriptions for the general listening public. For me, though, the rapidly growing set of measured devices gave me plenty of options to choose from and I can't imagine needing to buy speakers that hadn't been measured.

 

[MAB]

People are misreading and/or misinterpreting the meaning of a reliable observation.
You can't call an observation that is subject to bias reliable in a general sense. Calling sighted listening reliable is actually quite extreme, especially when so many controls are needed to prevent incorrect observations. Look at the work and the sample sizes Toole et. al. had to do to get reliable statistically relevant results. Calling results of a medical trial reliable that didn't properly control for bias is another example of an extreme position.
It's that simple.