A Broad Discussion of Speakers with Major Audio Luminaries

[respice finem]

...Calling sighted listening reliable is actually quite extreme...

But one could call it reliably biased

 

[olieb]

But often enough people here really are making statements that read as fairly extreme sceptical positions on sighted listening - where implicit or explicit in the statements is the proposition that sighted listening perception is inevitably polluted by biases.

(Bolding is mine)

I do not see that as an extreme position at all, maybe mildly sceptical in the way science has to be.
It is a result of scientific research that listening perception is easily (and regularly) polluted by biases, and that without controls in place (being sighted is an obvious lack of control) the outcome is highly unreliable.

BTW "unreliable" does not mean the correlation has to be zero, it might be somewhat smaller or bigger, positive or negative(!). Most of the time you just don't know, and you can't expect it to approach 1 at all.

 

[MattHooper]

People are misreading and/or misinterpreting the meaning of a reliable observation.

I’m not sure about that.

For instance, earlier, if you had said “Sighted listening is unreliable” then I don’t think there would be any disagreement or misunderstanding. I would assume I know what you mean - as you say now “ in a general sense” - and I’d also agree.
(or as I have been putting it: sighted listening is not reliable enough to produce certainty or high confidence levels, of the type one is looking for in science, or that people are often seeking on ASR).

But instead, you wrote sighted listening is “completely and totally unreliable.”
It was those added words of emphasis that had me puzzled. Because in my years here, I really have encountered a fair amount of, in my view, overreach in the scepticism of sighted listening (and the value of subjective descriptions).

So it’s really not easy to tell where you were on this scale. I mean, if by “ completely and totally unreliable” you meant that you totally ruled out sighted listening’s usefulness in evaluating speakers - and that certainly is what that sentence sounds like - then I would disagree. Which is why I was asking for clarification.

And to be honest, I’m still not exactly clear what your position is. For instance, do you think that one can not gain any useful information about a loudspeaker in sighted conditions….auditioning at an audio store?….or at home?

Just intoning the words reliable or unreliable doesn’t in itself clarify these issues.

 

[MattHooper]

I do not see that as an extreme position at all, maybe mildly sceptical in the way science has to be.
It is a result of scientific research that listening perception is easily (and regularly) polluted by biases, and that without controls in place (being sighted is an obvious lack of control) the outcome is highly unreliable.

Yes, I can see your point, and even when I was writing that sentence I was thinking I should come up with a better way of saying that.

So I might change it to: the proposition that any sighted listening impressions are inevitably the result of biases.

Which would carry the suggestion that sighted listening impressions are always going to be the result of some bias effect rather than accurately apprehending any characteristics of the loudspeaker, or the “sound waves” as one member likes to put it.

As I said, I’d have no problem with anybody saying that if you’re listening to a loudspeaker under sighted conditions you can’t be sure your perception isn’t influenced by a bias effect. But it’s often implied or suggested here a scepticism that goes beyond this, in which sighted listening inevitably distorts perception to a degree that it is essentially fruitless for apprehending the character of a loudspeaker.

 

[RexrothPigeon]

So I might change it to: the proposition that any sighted listening impressions are inevitably the result of biases.

This "inevitably" is unnecessary. all sighted listening impressions are unreliable because bias can never be ruled out. so sometimes sighted listening impressions may be accurate, but without a reliable way to know when or if that's the case, they are inherently dubious. In other words, while sighted listening impressions are not as random as, say, picking lottery numbers, the outcome is similar: they can't be relied upon to be accurate in any given case.

 

[MarkS]

The limited data presented in Flyod Toole's book clearly shows a correlation between blind and sighted results. Here is one relevant figure illustrating this:

If there was zero correlation, it would mean that blind tests were useless for evaluating speakers that are to be listened to sighted in the home. It would mean that sighted bias is so overwhelming that the actual sound has no impact at all on sighted listeners' perception of the sound. It would mean that, for maximum enjoyment, speakers should be purchased on the basis of non-sonic factors such as looks and cost only. Measurements (of the sound) and blind testing would both be useless for determining how listeners reacted to the speakers when they make sound, only non-sonic factors would matter.

Of course this absurd situation is not the case. The data clearly shows that results of blind and sighted listening are correlated, as basic common sense demands. The expert opinion of Dr. Floyd Toole, in his "final word" on the subject (quoted at greater length above), is that "Sighted evaluations introduce the possibility that opinions will be biased by non-auditory factors. The extent of the bias depends on the nature and importance of the non-auditory factors to the individual listeners."

 

[sigbergaudio]

Well said @MarkS. It's a pretty stark contrast between the quote from Toole here and implying that sighted listening is essentially like playing the lottery (which obviously is highly exaggerated).

I find it hard to believe that those who suggest this actually believes this to be so when thinking about themselves.

If you sit down in front of any speaker system, are you essentially completely clueless about the sound quality? Of course you are not. There may be any number of bias that impact your impression in a number of ways, and it is impossible to quantify to which extent you are biased. This means any report you share with others about the experience is inherently subjective and anecdotal.

That is still a far cry from implying that your chance of giving a reasonably accurate report of the sound is as small as winning the lottery.

 

[olieb]

The data clearly shows that results of blind and sighted listening are correlated, as basic common sense demands.

There certainly is correlation, as otherwise - as you pointed out - listening would be random.
But the graphs you present are not representing the situation of a single listener in a sighted (versus blind) test.
Toole's results are already from controlled tests:
In the same room, level matched and above all for a number of (how high?) of listeners, so that individual biases are evened out.
It would look different for a single listener, and one can only speculate how it would look for a single listener in some unknown room listening to some recording commenting on a single speaker characteristic ("spaciousness", "transients", "speed of bass") instead of integrating all into "preference".
THAT is the situation that was brought up.
But yes, I would expect some (unknown degree) of correlation still.

 

[Newman]

If there was zero correlation, it would mean that blind tests were useless for evaluating speakers that are to be listened to sighted in the home. It would mean that sighted bias is so overwhelming that the actual sound has no impact at all on sighted listeners' perception of the sound. It would mean that, for maximum enjoyment, speakers should be purchased on the basis of non-sonic factors such as looks and cost only. Measurements (of the sound) and blind testing would both be useless for determining how listeners reacted to the speakers when they make sound, only non-sonic factors would matter.

Of course this absurd situation is not the case.

Well said @MarkS. It's a pretty stark contrast between the quote from Toole here and implying that sighted listening is essentially like playing the lottery (which obviously is highly exaggerated).

Why are you patting him on the back for using a straw man argument? Cheap debating tactics. "Well said" is almost the opposite of what he deserves for his game of malicious misrepresentation.

Do you openly endorse straw man debating tactics? Discussions in bad faith?

 

[sigbergaudio]

Why are you patting him on the back for using a straw man argument? Cheap debating tactics. "Well said" is almost the opposite of what he deserves for his game of malicious misrepresentation.

Do you openly endorse straw man debating tactics? Discussions in bad faith?

Some go far to imply sighted listening is completely meaningless in determining sound quality, which would indicate zero correlation. So slight strawman perhaps, but far from malicious misrepresentration.

 

[Geert]

Why are you patting him on the back for using a straw man argument?

Cause and effect. It all starts with the absolute and non-nuanced claim "sighted listening is completely and totally unreliable" (or likewise: "can't be relied upon to be accurate in any given case"), while we have following quote from Dr. Toole: "The extent of the bias depends on the nature and importance of the non-auditory factors to the individual listeners".

 

[MayI]

The limited data presented in Flyod Toole's book clearly shows a correlation between blind and sighted results. Here is one relevant figure illustrating this:

View attachment 523357

Hi Mark, I see it differently. The "visually identical" speakers seemed to correlate well with the blind listening tests. We don't know whether they measure similarly or not, but seemingly correlated sighted vs blind. The "US" vs Sub/sat do not appear correlated in my view, the sighted test showed significant preference for the US, the listening test a slight preference for the Sub/sat. How is that correlated? I think your example actually shows the fallibility of sighted listening if I'm reading the limited data correctly. I wouldn't consider sighted listening as "useless", but rather "fallible". Based on this sample at least.

 

[gnarly]

The limited data presented in Flyod Toole's book.........<big snip>

Appreciated your common sense analysis that I snipped...

Personally, I view the data as far too sparse to be valid for the depths of discussions and conclusions about it, that folks seem to want to make.
I think many are making mountains out of molehills.

 

[Sokel]

There is one instance that things get messy and you only learn to interpret them with both listening* and measuring.

A glorious example, impact.

My factory default x-over for the semi-actives I use is at about 240Hz.
At this setting I can have all the impact I want, etc.

Now, I like playing with analog el. x-overs (and also building some) so I can test other x-over points as well.
And here comes the experience, as keeping the same FR, EQ for lows, etc, the works, lowering the x-over point also reduces impact, to the point there's none at 80Hz or so (F3 is at 31Hz or so)

REW shows the same at the usual sweeps but measuring the same song at the exact same conditions with the SPL chart, differences at MAX and peaks are obvious. And it makes perfect sense, one can't use a 7" mid to push air.

Anything can be useful if you put it to work for you and not the other way around.

*should I say "feeling" instead? Impact is physical, you can't escape it or fake it

 

[levimax]

There is one instance that things get messy and you only learn to interpret them with both listening* and measuring.

A glorious example, impact.

My factory default x-over for the semi-actives I use is at about 240Hz.
At this setting I can have all the impact I want, etc.

Now, I like playing with analog el. x-overs (and also building some) so I can test other x-over points as well.
And here comes the experience, as keeping the same FR, EQ for lows, etc, the works, lowering the x-over point also reduces impact, to the point there's none at 80Hz or so (F3 is at 31Hz or so)

REW shows the same at the usual sweeps but measuring the same song at the exact same conditions with the SPL chart, differences at MAX and peaks are obvious. And it makes perfect sense, one can't use a 7" mid to push air.

Anything can be useful if you put it to work for you and not the other way around.

*should I say "feeling" instead? Impact is physical, you can't escape it or fake it

I notice the same thing. Since "impact" uses "time" in the denominator are you sure the "different feeling" is due to higher SPL or could it be due to better time domain performance (shorter time in the denominator). My guess is it is both.

 

[Sokel]

I notice the same thing. Since "impact" uses "time" in the denominator are you sure the "different feeling" is due to higher SPL or could it be due to better time domain performance (shorter time in the denominator). My guess is it is both.

I have some thoughts but are conflicting, to be entirely honest I don't know what to think.

I didn't notice that searching about that, but during one of the usual x-over ASR debates about stereo bass or about detecting direction at lows, it was pure coincidence just trying to replicate a usual 80Hz subs (although I don't use any) with mains. There, I noticed that impact has left the room and tried to see what happening and where, through REW and my chest

Excluding anything else (including sweeps, they are identical) I arrived at the SPL chart.
I was also very careful to stay well at speaker's comfort zone about SPL, including peaks at steady state (somewhere at ASR I have a comparative 80dB to 100dB+ distortion test that we did, mostly for fun, through REW's distortion chart, with zero compression)*

*Edit: here:

"System" SINAD

Would the concept of "system" SINAD (from source to speakers) be useful in component selection for a good stereo system? Let's assume a listening SPL of 85dB @ 1m, and a room noise floor of around 40dB SPL. Assuming +20dB headroom for signal peaks, and another -20dB noise headroom. A system...

www.audiosciencereview.com

 

[bmc0]

Hi Mark, I see it differently. [...]

I agree and find it puzzling that MarkS and others would claim that the data are favorable evidence for the reliability of sighted tests. If I remember correctly, Floyd Toole said that loudspeakers 1 and 2 differed only in their crossover circuit. Depending on the details of how the tests were conducted, the "sighted" tests may have been effectively blind anyway between those two speakers.
The significant increase in preference for speakers 1 and 2, and moderate increase for 4 (the expensive, likely impressive-looking ones) along with the significant decrease for 3 (the cheap, small one) in sighted tests clearly demonstrates bias.

 

[sigbergaudio]

I agree and find it puzzling that MarkS and others would claim that the data are favorable evidence for the reliability of sighted tests. If I remember correctly, Floyd Toole said that loudspeakers 1 and 2 differed only in their crossover circuit. Depending on the details of how the tests were conducted, the "sighted" tests may have been effectively blind anyway between those two speakers.
The significant increase in preference for speakers 1 and 2, and moderate increase for 4 (the expensive, likely impressive-looking ones) along with the significant decrease for 3 (the cheap, small one) in sighted tests clearly demonstrates bias.

It is interesting how difficult this, and I am sorry that both sides (perhaps me included) keep exaggerating so that it's impossible to find common ground.

But this is again strawman. I don't think anyone has said sighted tests are reliable, and also no one has said that sighted tests aren't prone to bias.

 

[bmc0]

It is interesting how difficult this

Imprecision of language, perhaps. I don't think my comment is a strawman. The claim was that the data presented demonstrate meaningful positive correlation between sighted and blind tests. I don't agree with that interpretation of the data and stated why.

 

[sigbergaudio]

Imprecision of language, perhaps. I don't think my comment is a strawman. The claim was that the data presented demonstrate meaningful positive correlation between sighted and blind tests. I don't agree with that interpretation of the data and stated why.

Somewhat imprecise yes. What the data says aside, the suggestion of presence of meaningful positive correlation is not the same as saying sighted listening is reliable, and also not the same as implying that bias doesn't exist.

The whole thing is problematic in the context that all of us will in practice be doing sighted listening when we use our systems. If our preferences when we actually use the systems we purchase don't have any correlation with their measured performance, the whole thing is moot. So obviously no one is suggesting that, right? And no one is suggesting that we are free of bias during sighted listening either.