• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Catalogue of blind tests

Spkrdctr

Major Contributor
Forum Donor
Joined
Apr 22, 2021
Messages
1,349
Likes
1,709
The much-taken snake oil at ASR is that specs on electronics matter.

Heretic! Jezibel! Witch! You will be burned at the stake for that comment.

I "try" not to be too serious on these forums as thousand of other posters are extremely serious and often times have to break out the ruler to start measuring. I try not to do that, but I'm sure I have mistakenly went down that path at least once. Live and learn. Now I just live and let live, mostly. Ok, till I get riled up. But I do try!
 

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,787
Likes
3,908
Location
Sin City, NV
The much-taken snake oil at ASR is that specs on electronics matter.

I would agree with that... up to a point. If you'd phrased it as "specs on electronics matter necessarily" - then we'd be in complete agreement. It's human nature (particularly in organisms with high testosterone levels) to equate specificity with expertise. However, in an age where there are so many literal con-artists operating in the CE space - it's sometimes comforting to see a justification (or lack thereof) in product marketing and pricing.

Except in the most egregious cases however, the audible effect of these differences are negligible at best. "Enthusiasts" is really just another way of saying "someone with way too much time on their hands and a bad case of either OCD or insecurity" - likely both (and yes, I admit to being one of them). ;)
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
2,176
Likes
3,700
Location
New York City
A sentence towards the end of this got me thinking:

The problem is, every alternative medicine fad has evidence approximately this good. Some are better; there are dozens of positive RCTs for homeopathy. This is why the replication crisis sucks so much - just because a few converging lines of evidence support a theory and it has lots of positive studies including an RCT or two, doesn't mean it's any good. Where do you draw the line?

Isn't it interesting that cables, amps, tweaks, don't seem to have even have the requisite few crackpot studies to lean on?
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
2,176
Likes
3,700
Location
New York City
So has there been any blind tests on DACs?
Google helps



 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
2,176
Likes
3,700
Location
New York City
I can’t remember whether I linked to this excellent list of blind tests, along with valuable interpration advice:

 

Samudra1825

Member
Joined
Jan 10, 2022
Messages
21
Likes
1
I agree with the sentiments above. But I believe it remains true that nearly all the blind tests of amps/cables/digital sources fail to reject the null hypothesis that there is no audible difference.
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
 

SIY

Master Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
8,398
Likes
18,897
Location
Alfred, NY
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
No, it means that the hypothesis of the experiment, “there is an audible difference under these conditions,” was not supported by the results. Or “no audible difference was detected.”
 

artburda

Active Member
Joined
Feb 12, 2019
Messages
212
Likes
256
Location
Switzerland
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
I think he’s saying that the results of most blind tests fail to show that there is a audible difference. Not because they are flawed but because the participants simply are unable to hear a difference on a statistically significant level.
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
2,176
Likes
3,700
Location
New York City
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
Statistics reasoning and clear English sometimes don’t go together. I’ve found at ASR one has to be careful with one’s claims to avoid being jumped on. So…

Statistics from trials aren’t considered formal proof of anything. We can only *reject* a hypothesis, typically a hypothesis of no relationship (hence “null”) between two variables. In this case, the null hypothesis is “There is no audible distinction between cables (electronics, etc.)”. If someone gets a statistically significant result distinguishing (blind) between cables, for instance, that is a result that rejects the null hypothesis, not *proof* that they can hear a difference. But as we do many experiments and fail to reject the null hypothesis, we become more comfortable with the null hypothesis. Or, if we do many (blind) trials and repeatedly reject the null hypothesis, we become comfortable that there may, in fact, be a relationship between the cable and the audible result.

With medicine, for instance, the null hypothesis would be that the medication under testing makes no significant difference from a placebo in double blind trials. We decide a medicine may work if trials reject that null hypothesis repeatedly, to a statistically significant level (typically only a 5% chance that the result was random).

If expensive cables and electronics were medication for an audibly better listening experience, they would not be approved under accepted methodologies. They would be in the health food aisle with the homeopathic remedies. And yet, there are a few people here who say our belief in the null hypothesis is “flat earth” thinking. quite something.

That is the laborious, sometimes double negative, nature of repeated testing.

I’m sorry if this is pedantic. I can’t imagine reading statistical trials in either of my foreign languages.
 
Last edited:

Samudra1825

Member
Joined
Jan 10, 2022
Messages
21
Likes
1
thank you @SIY and @artbuda, I think I
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
I think he’s saying that the results of most blind tests fail to show that there is a audible difference. Not because they are flawed but because the participants simply are unable to hear a difference on a statistically significant level.
thank you. i think my brain skips the word "reject" and thought SIY said that nearly all tests failed to prove that there is no audible difference or something around the line
 

Samudra1825

Member
Joined
Jan 10, 2022
Messages
21
Likes
1
If someone gets a statistically significant result distinguishing (blind) between cables, for instance, that is a result that rejects the null hypothesis, not *proof* that they can hear a difference
is there any difference between rejecting the null hypothesis and proof that they can hear a difference?
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
2,176
Likes
3,700
Location
New York City
is there any difference between rejecting the null hypothesis and proof that they can hear a difference?
Yes, the former does not reach the level of the latter - actual "proof" of a causal relationship is not really something you can show statistically. But we accept a correlation as truth it if it is repeatable, as with drug clinical trials.

We're stuck with "unlikely" or "likely".

Maybe next I should cover how we might use Bayesian inference to save time in audio decisions....
 

SIY

Master Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
8,398
Likes
18,897
Location
Alfred, NY
is there any difference between rejecting the null hypothesis and proof that they can hear a difference?
I wouldn't say "proof." But certainly if the results are consistent with a hypothesis of audibility, it's actual real evidence.

edit: ahofer got there first.
 
Top Bottom