• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Catalogue of blind tests

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
The much-taken snake oil at ASR is that specs on electronics matter.

I would agree with that... up to a point. If you'd phrased it as "specs on electronics matter necessarily" - then we'd be in complete agreement. It's human nature (particularly in organisms with high testosterone levels) to equate specificity with expertise. However, in an age where there are so many literal con-artists operating in the CE space - it's sometimes comforting to see a justification (or lack thereof) in product marketing and pricing.

Except in the most egregious cases however, the audible effect of these differences are negligible at best. "Enthusiasts" is really just another way of saying "someone with way too much time on their hands and a bad case of either OCD or insecurity" - likely both (and yes, I admit to being one of them). ;)
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
A sentence towards the end of this got me thinking:

The problem is, every alternative medicine fad has evidence approximately this good. Some are better; there are dozens of positive RCTs for homeopathy. This is why the replication crisis sucks so much - just because a few converging lines of evidence support a theory and it has lots of positive studies including an RCT or two, doesn't mean it's any good. Where do you draw the line?

Isn't it interesting that cables, amps, tweaks, don't seem to have even have the requisite few crackpot studies to lean on?
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
So has there been any blind tests on DACs?
Google helps



 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
I can’t remember whether I linked to this excellent list of blind tests, along with valuable interpration advice:

 

Samudra1825

Member
Joined
Jan 10, 2022
Messages
21
Likes
1
I agree with the sentiments above. But I believe it remains true that nearly all the blind tests of amps/cables/digital sources fail to reject the null hypothesis that there is no audible difference.
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
No, it means that the hypothesis of the experiment, “there is an audible difference under these conditions,” was not supported by the results. Or “no audible difference was detected.”
 

artburda

Active Member
Joined
Feb 12, 2019
Messages
287
Likes
378
Location
Switzerland
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
I think he’s saying that the results of most blind tests fail to show that there is a audible difference. Not because they are flawed but because the participants simply are unable to hear a difference on a statistically significant level.
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
Statistics reasoning and clear English sometimes don’t go together. I’ve found at ASR one has to be careful with one’s claims to avoid being jumped on. So…

Statistics from trials aren’t considered formal proof of anything. We can only *reject* a hypothesis, typically a hypothesis of no relationship (hence “null”) between two variables. In this case, the null hypothesis is “There is no audible distinction between cables (electronics, etc.)”. If someone gets a statistically significant result distinguishing (blind) between cables, for instance, that is a result that rejects the null hypothesis, not *proof* that they can hear a difference. But as we do many experiments and fail to reject the null hypothesis, we become more comfortable with the null hypothesis. Or, if we do many (blind) trials and repeatedly reject the null hypothesis, we become comfortable that there may, in fact, be a relationship between the cable and the audible result.

With medicine, for instance, the null hypothesis would be that the medication under testing makes no significant difference from a placebo in double blind trials. We decide a medicine may work if trials reject that null hypothesis repeatedly, to a statistically significant level (typically only a 5% chance that the result was random).

If expensive cables and electronics were medication for an audibly better listening experience, they would not be approved under accepted methodologies. They would be in the health food aisle with the homeopathic remedies. And yet, there are a few people here who say our belief in the null hypothesis is “flat earth” thinking. quite something.

That is the laborious, sometimes double negative, nature of repeated testing.

I’m sorry if this is pedantic. I can’t imagine reading statistical trials in either of my foreign languages.
 
Last edited:

Samudra1825

Member
Joined
Jan 10, 2022
Messages
21
Likes
1
thank you @SIY and @artbuda, I think I
by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused
I think he’s saying that the results of most blind tests fail to show that there is a audible difference. Not because they are flawed but because the participants simply are unable to hear a difference on a statistically significant level.
thank you. i think my brain skips the word "reject" and thought SIY said that nearly all tests failed to prove that there is no audible difference or something around the line
 

Samudra1825

Member
Joined
Jan 10, 2022
Messages
21
Likes
1
If someone gets a statistically significant result distinguishing (blind) between cables, for instance, that is a result that rejects the null hypothesis, not *proof* that they can hear a difference
is there any difference between rejecting the null hypothesis and proof that they can hear a difference?
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
is there any difference between rejecting the null hypothesis and proof that they can hear a difference?
Yes, the former does not reach the level of the latter - actual "proof" of a causal relationship is not really something you can show statistically. But we accept a correlation as truth it if it is repeatable, as with drug clinical trials.

We're stuck with "unlikely" or "likely".

Maybe next I should cover how we might use Bayesian inference to save time in audio decisions....
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
is there any difference between rejecting the null hypothesis and proof that they can hear a difference?
I wouldn't say "proof." But certainly if the results are consistent with a hypothesis of audibility, it's actual real evidence.

edit: ahofer got there first.
 

Mal

Active Member
Joined
Sep 30, 2022
Messages
157
Likes
61
I was hoping for a catalogue, but this thread is another general chat about blind testing. Some interesting points being made, but is there an actual catalogue of speaker blind tests anywhere on the web? As the OP said, it's probably best to make that a wiki... he must have been thinking that forums are really about chatting, not providing blunt information (nothing wrong with that, but sometimes you just want a catalogue...)
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
I was hoping for a catalogue, but this thread is another general chat about blind testing. Some interesting points being made, but is there an actual catalogue of speaker blind tests anywhere on the web? As the OP said, it's probably best to make that a wiki... he must have been thinking that forums are really about chatting, not providing blunt information (nothing wrong with that, but sometimes you just want a catalogue...)
There are links to some lists throughout, but Amir did not want to add a wiki to the site.
 

Mal

Active Member
Joined
Sep 30, 2022
Messages
157
Likes
61
There are links to some lists throughout, but Amir did not want to add a wiki to the site.
But that means reading all the chat! What's needed is a link to all lists - a proper catalogue. No reason why someone can't create a wiki on another site and link to that from here. Then again, it doesn't need to be a wiki, maybe a web page would do, as long as the creator is dedicated and has a completist tendency!
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
But that means reading all the chat! What's needed is a link to all lists - a proper catalogue. No reason why someone can't create a wiki on another site and link to that from here. Then again, it doesn't need to be a wiki, maybe a web page would do, as long as the creator is dedicated and has a completist tendency!
Are you volunteering? ;)
 

Mal

Active Member
Joined
Sep 30, 2022
Messages
157
Likes
61
Do partially-sighted speaker faceoffs tally well with blind test results? If they can be shown to tally well then we we can trust many comparisons - anyone can do a face off (and many do...), while blind tests are too tricky for most to attempt. Example:

"We did sighted tests, but made rough attempts to pay little attention to which speakers were in positions 1 or 2. Gene set up each listening session so the rest of us were oblivious, at least until we got further along, which speaker we were initially listening to. We did not attempt to do a controlled blind test, nor did we desire to set up any sort of mechanism to remove all forms of bias during the process. This was mostly due to time and the equipment on hand to facilitate such a process. We used no speaker toe-in having all of the speakers firing directly ahead in the listening area."

 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
Do partially-sighted speaker faceoffs tally well with blind test results? If they can be shown to tally well then we we can trust many comparisons - anyone can do a face off (and many do...), while blind tests are too tricky for most to attempt. Example:

"We did sighted tests, but made rough attempts to pay little attention to which speakers were in positions 1 or 2. Gene set up each listening session so the rest of us were oblivious, at least until we got further along, which speaker we were initially listening to. We did not attempt to do a controlled blind test, nor did we desire to set up any sort of mechanism to remove all forms of bias during the process. This was mostly due to time and the equipment on hand to facilitate such a process. We used no speaker toe-in having all of the speakers firing directly ahead in the listening area."

For cataloguing purposes, if there is an attempt at volume-leveling and unsighted listening, I’d include it. In the list linked in the first post there were some pretty dodgy tests (and then even more dodgy stats conclusions from Stereophile-an interpretation that would have received a zero on an entry-level college stats quiz)
 
OP
ahofer

ahofer

Major Contributor
Forum Donor
Joined
Jun 3, 2019
Messages
4,952
Likes
8,698
Location
New York City
I hadn’t seen this one. Interesting that he documented the subject’s satisfaction with the setup prior to the test, but after some practice switching.

 
Top Bottom