Master Thread: Are measurements Everything or Nothing?

[pkane]

The correct representation of a hand isn't five fingers, though.

It's four, and the thumb.

I would not contradict our new AI overlords.

 

[Jimbob54]

So let's just briefly consider your hypothesis without the philosophical bent as a working scientific one. It clearly fails.
Why? Because blind testing (and much of the actual scientific work is not ABX) tells a coherent story. The vast majority of people give the same responses. If there was a "bias" it would be held to different strengths, or not at all, then the story told by blind testing would be incoherent. The blind testing tells us something about human hearing - not bias but properties. It's actually pretty pointless to ask why blind tests are wrong, and certainly a problem to do so in isolation from studying sighted response.

A reasonable question for study in the area is the opposite one: why does the coherent story of blind testing break down when testing is instead sighted?

I'll be neutral and request we all leave the purely philosophical pondering and at least try and stick to the usual arguing about measurements telling (or not ) the whole story and all that jazz .

 

[krabapple]

I didn't think the fact that it "ressembles" some subjectivists claims was enough to make it "suspicious"

I agree with everything you said, so I think I have failed to make myself understood.

I had in mind the "sighted listening vs blinded listening" debate that happened here. What I got from it was something like that:

That debate has 'happened' since at least the 1980s, across multiple media. You seem entirely new to it.

But you expect us to respect your superior expertise in...what? Epistemology?

Bob says "I like speakers A a lot more than speakers B" in sighted conditions.
People (here) would apparently reply: "no but you can't trust your experience, it was biased by the fact that you could see the speakers".
to that I question: "wouldn't you be differently biased in blind listening?"

Bob didn't claim he liked A more than B based just on their sound. So no, that would not be the reply.


Argue better.

 

[krabapple]

Maybe it helps when I say that with blind testing it is not just not seeing speakers but rather having no clues about what is playing. Of course you can't get rid of all clues. You are just testing for preference. It is not for gathering scientific info.

Please stop writing this. Testing for preference absolutely can be an instance of 'gathering scientific info'.

 

[krabapple]

In a perfect world - but as I pointed out, doing back to back blind comparisons in one's own home isn't likely to happen due to the practicalities.

What's simple is tempering or qualifying one's claims about the device's sound, based on knowing the flaws of sighted evaluation.

So simple, really. But apparently so hard for many.

 

[Mart68]

Perhaps it's more pertinent to consider how many of us actually used the results of properly conducted, statistically valid blind tests of loudspeakers when choosing our own.

More pertinent, yes, but I suspect the answer again would be 'not many'.

Current speakers I bought because they were cheap (used), looked like they might be good, and were within a reasonable distance to collect. Happily they are great and have been in use for about 7 years now.

There was nothing scientific involved since no measurements or controlled listening test data was available - even if there was, it would still, to some extent, have been a roll of the dice.

 

[Mart68]

What's simple is tempering or qualifying one's claims about the device's sound, based on knowing the flaws of sighted evaluation.

So simple, really. But apparently so hard for many.

Agreed.

 

[krabapple]

Maybe if we had a show of hands here to see how many did actually select their speakers by doing a blind comparison? Did anyone?

Show of hands to see how many would, if they could.

I regret that I can't. Therefore I don't make dubious claims about the sound of the ones I have. (Though Amir has measured one of them, so I can make some objectively supported claims about that one)

 

[krabapple]

And many of these exist supporting most of the basic beliefs you see on this site. Unfortunately, most are behind the AES paywall (or, for audiology, other paywalls), but @amirm has summarized quite a few on the site.

...a site pabloloco claims to have read for a year before posting.

 

[solderdude]

Please stop writing this. Testing for preference absolutely can be an instance of 'gathering scientific info'.

It can be when doing scientific research for preference.
It hardly is when comparing 2 speakers for personal preference to make a purchase decision.
IME about 99% of people buying speakers is simply comparing them and not do any blind testing. They end up buying the one they prefer based on how it sounds and looks.
They usually don't do any blind listening tests just for preference for the sake of knowing which they prefer and not buying them.

 

[krabapple]

It can be when doing scientific research for preference.
It hardly is when comparing 2 speakers for personal preference to make a purchase decision.
IME about 99% of people buying speakers is simply comparing them and not do any blind testing. They end up buying the one they prefer based on how it sounds and looks.
They usually don't do any blind listening tests just for preference for the sake of knowing which they prefer and not buying them.

Of course; and almost no one does controlled tests of audible difference at home, either.

But I want to always emphasize on ASR that preference CAN be (and HAS BEEN) subjected to scientific measurement, because some posters here seem to believe that because preference is 'inviolable' (in the sense that there is no 'right' or 'wrong') it is also outside the purview of science.

 

[dshreter]

A thought - if someone explained they had a professional design and install a home theater, nobody would bat an eye.

If someone stated they selected speakers by measurements instead of trial, they would get a lot of criticism.

Why is it that stereo speakers are treated like a spouse? Can you think of anything else people make such a fuss over?

 

[MattHooper]

But I want to always emphasize on ASR that preference CAN be (and HAS BEEN) subjected to scientific measurement, because some posters here seem to believe that because preference is 'inviolable' (in the sense that there is no 'right' or 'wrong') it is also outside the purview of science.

I strongly agree.

Preference certainly can be and has been studied scientifically. It’s studied all the time.

But the sharpen up the point even more, I think I would put it:

1. It is true to say there is no right and wrong with regard to preference, when we are talking about someone’s actual preference.

However

2. It’s possible to be wrong when making statements of preference. You can be wrong in stating what most people prefer. You can even, to a degree, be wrong about what you yourself prefer!

You may think you prefer the sound of one cable or another, but it can be discovered through testing that you can’t actually tell them apart.

You may think you prefer A loudspeaker over B, but it can be discovered through testing that you are wrong and that you actually prefer B. (At least sonically. You would not be in fact wrong about what you prefer under sighted conditions, but could be wrong in what you believe about why you prefer the speaker, believing you prefer simply the sound itself, when other factors actually are influencing your preference).

 

[ahofer]

Why is it that stereo speakers are treated like a spouse? Can you think of anything else people make such a fuss over?

A lot of luxury goods, guitars, clothes, cigars, wine, etc. Consider the intrinsic value of a Goyard Bag.

 

[Galliardist]

I strongly agree.

Preference certainly can be and has been studied scientifically. It’s studied all the time.

But the sharpen up the point even more, I think I would put it:

1. It is true to say there is no right and wrong with regard to preference, when we are talking about someone’s actual preference.

However

2. It’s possible to be wrong when making statements of preference. You can be wrong in stating what most people prefer. You can even, to a degree, be wrong about what you yourself prefer!

You may think you prefer the sound of one cable or another, but it can be discovered through testing that you can’t actually tell them apart.

You may think you prefer A loudspeaker over B, but it can be discovered through testing that you are wrong and that you actually prefer B. (At least sonically. You would not be in fact wrong about what you prefer under sighted conditions, but could be wrong in what you believe about why you prefer the speaker, believing you prefer simply the sound itself, when other factors actually are influencing your preference).

Point 2 should really make the point that you can be wrong about the reasons for your preference.

And that leads to a lot of bad science and pseudoscience. When you can find patents where the main proof of an audio effect consists of "someone in the office listened to it" you know there are problems.

 

[Newman]

Perhaps it's more pertinent to consider how many of us actually used the results of properly conducted, statistically valid blind tests of loudspeakers when choosing our own.

I did.

But I'll also say this: I think it is fair to answer that making intelligent readings of high-grade Spinorama tests of speakers is a rational and reasonable proxy for conducting blind tests of the speakers themselves.

That was one of the major outcomes of all that research: if we understand the research properly, we understand its value to our own decision making. It's actually a better indicator of our preferences for the sound waves themselves than if we had done sighted listening to the speakers in person.

cheers

 

[Galliardist]

I did.

But I'll also say this: I think it is fair to answer that making intelligent readings of high-grade Spinorama tests of speakers is a rational and reasonable proxy for conducting blind tests of the speakers themselves.

That was one of the major outcomes of all that research: if we understand the research properly, we understand its value to our own decision making. It's actually a better indicator of our preferences for the sound waves themselves than if we had done sighted listening to the speakers in person.

cheers

Good answer: but I'd add that without the preference testing, we wouldn't know which speakers to choose from a Spinorama or any other frequency response measurement. Choosing an earlier paradigm such as flat in-room wouldn't have sounded any better because of a better testing method.

We still get (by necessity) that final exercise of sighted listening, of course, when we actually use them if not before purchase. So I'd still say that the research we have is the best starting point, rather than necessarily the final act. If that sighted response is overall negative, we still have to move on...

 

[MattHooper]

But I'll also say this: I think it is fair to answer that making intelligent readings of high-grade Spinorama tests of speakers is a rational and reasonable proxy for conducting blind tests of the speakers themselves.

Yes. And in a similar vein, Proper measurements of other audio gear, such as amplifiers, audio cables, AC cables etc. can be understood in the context i’ve known audible thresholds for distortion established by proper listening tests. And so those too can be proxy for making decisions.

It’s reasonable to just use the least expensive, properly constructed AC cable for a piece of gear rather than spend thousands of dollars expecting audible differences.

 

[MattHooper]

We still get (by necessity) that final exercise of sighted listening, of course, when we actually use them if not before purchase. So I'd still say that the research we have is the best starting point, rather than necessarily the final act. If that sighted response is overall negative, we still have to move on...

I’m not sure everyone feels that way.

For one thing, there are people who buy just on measurements, and who are satisfied mostly on that basis.

Your last sentence “ if that sighted response is overall negative” is an interesting issue.
And I think even here people may diverge.

There have been people on this forum
who’ve said they are seeking high Fidelity equipment and they don’t actually trust their own ears to make that decision. So even listening, in sighted conditions they don’t trust their sighted impressions (presumably even if negative) and would go on measurements.

That’s a different mindset than my own admittedly. But I understand it.

My own approach is to pay attention to measurements but ultimately I go on my sighted listening impressions of loudspeakers for my purchases.

This is because in my own experience, my sighted listening impressions tend to be extremely robust and long lasting. So that even if my impressions are due to some combination of the actual sound/bias effects, I seem to be mostly stuck with them. Once I’ve had a good listen to a loudspeaker my impression of its characteristics tend not to change much at all over time. So if I happen to be listening to loudspeaker A that is somewhat measurably more accurate than loudspeaker B, but A is not grabbing me whereas I want to just keep listening to B, and I’m going to purchase speaker B.
Given how much money we are often talking about with high-end loudspeakers, I’m not willing to roll the dice on the idea that, if I don’t care for the sound of a loudspeaker after an extensive audition, well maybe I’ll change my mind about it if I actually commit all the money to buying it. None of my experience gives that proposition any encouragement. And I have virtually always done well buying on my sighted impressions in terms of my satisfaction.

But we all bring our own set of criteria and biases and goals as to what approach is going to satisfy us.

 

[dshreter]

There have been people on this forum
who’ve said they are seeking high Fidelity equipment and they don’t actually trust their own ears to make that decision. So even listening, in sighted conditions they don’t trust their sighted impressions (presumably even if negative) and would go on measurements.

I would put myself in that camp that I don't particularly trust my ears. I've done enough blind testing of different types and I'm usually disappointed by my individual performance and ability to identify or distinguish between things.

If something doesn't sound good, then I wouldn't listen to it because that would be absurd. But I also think it is more effective to rely on measurements to find something suitable for purpose and get good sound, and I don't consider there to be something special about my ears to require a unique solution. I don't see loudspeakers as being personal like a musical instrument.

I'm comfortable with the idea of picking out a speaker that meets my functional criteria and then worrying about something else. For example, if I was a baker and I bought a stand mixer with sufficient capacity and power to knead my dough, with a scientifically proven ideally shaped dough hook for gluten formation, and then my bread came out poorly.... I wouldn't go shopping for new stand mixers until my bread was good. I would verify that the stand mixer was doing its job properly and consider the other factors that influenced my recipe.

That's how I feel about speakers. If they measure well and are performing to spec, I have little reason to worry about it. I can focus on something else like improving the room acoustics and layout. There's magic in listening to a well performing system, but what speakers do is not mystical, and they only need to do their job, nothing more nothing less.