Catalogue of blind tests

[digicidal]

The much-taken snake oil at ASR is that specs on electronics matter.

I would agree with that... up to a point. If you'd phrased it as "specs on electronics matter necessarily" - then we'd be in complete agreement. It's human nature (particularly in organisms with high testosterone levels) to equate specificity with expertise. However, in an age where there are so many literal con-artists operating in the CE space - it's sometimes comforting to see a justification (or lack thereof) in product marketing and pricing.

Except in the most egregious cases however, the audible effect of these differences are negligible at best. "Enthusiasts" is really just another way of saying "someone with way too much time on their hands and a bad case of either OCD or insecurity" - likely both (and yes, I admit to being one of them).

 

[ahofer]

A sentence towards the end of this got me thinking:

The problem is, every alternative medicine fad has evidence approximately this good. Some are better; there are dozens of positive RCTs for homeopathy. This is why the replication crisis sucks so much - just because a few converging lines of evidence support a theory and it has lots of positive studies including an RCT or two, doesn't mean it's any good. Where do you draw the line?

Isn't it interesting that cables, amps, tweaks, don't seem to have even have the requisite few crackpot studies to lean on?

 

[Rat God]

So has there been any blind tests on DACs?

 

[ahofer]

So has there been any blind tests on DACs?

Google helps

DAC blind test: NO audible difference whatsoever

(UPDATES BELOW) That blind test was long due, it's now done. Started with a 30$ unit (Fiio) against a 3000$ one (Forssell) and once SPL-matched (massive gain difference), no one could tell the difference in a ABX test. Then, we switched to a different set-up, using a pair of B&W CM9 speakers...

www.diyaudio.com

https://www.avsforum.com/threads/perfect-dac-test.3139906/

BLIND TEST Results Part 3: "Do digital audio players sound different playing 16/44.1 music?" - Listener Results.

A blog for audiophiles about more objective topics. Measurements of audio gear. Reasonable, realistic, no snakeoil assessment of sound, and equipment.

archimago.blogspot.com

 

[ahofer]

I can’t remember whether I linked to this excellent list of blind tests, along with valuable interpration advice:

Links to blind listening tests

Links to blind listening tests

hydrogenaud.io

 

[Samudra1825]

I agree with the sentiments above. But I believe it remains true that nearly all the blind tests of amps/cables/digital sources fail to reject the null hypothesis that there is no audible difference.

by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused

 

[SIY]

by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused

No, it means that the hypothesis of the experiment, “there is an audible difference under these conditions,” was not supported by the results. Or “no audible difference was detected.”

 

[artburda]

by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused

I think he’s saying that the results of most blind tests fail to show that there is a audible difference. Not because they are flawed but because the participants simply are unable to hear a difference on a statistically significant level.

 

[ahofer]

by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused

Statistics reasoning and clear English sometimes don’t go together. I’ve found at ASR one has to be careful with one’s claims to avoid being jumped on. So…

Statistics from trials aren’t considered formal proof of anything. We can only *reject* a hypothesis, typically a hypothesis of no relationship (hence “null”) between two variables. In this case, the null hypothesis is “There is no audible distinction between cables (electronics, etc.)”. If someone gets a statistically significant result distinguishing (blind) between cables, for instance, that is a result that rejects the null hypothesis, not *proof* that they can hear a difference. But as we do many experiments and fail to reject the null hypothesis, we become more comfortable with the null hypothesis. Or, if we do many (blind) trials and repeatedly reject the null hypothesis, we become comfortable that there may, in fact, be a relationship between the cable and the audible result.

With medicine, for instance, the null hypothesis would be that the medication under testing makes no significant difference from a placebo in double blind trials. We decide a medicine may work if trials reject that null hypothesis repeatedly, to a statistically significant level (typically only a 5% chance that the result was random).

If expensive cables and electronics were medication for an audibly better listening experience, they would not be approved under accepted methodologies. They would be in the health food aisle with the homeopathic remedies. And yet, there are a few people here who say our belief in the null hypothesis is “flat earth” thinking. quite something.

That is the laborious, sometimes double negative, nature of repeated testing.

I’m sorry if this is pedantic. I can’t imagine reading statistical trials in either of my foreign languages.

 

[Samudra1825]

thank you @SIY and @artbuda, I think I

by "fail to reject the null hypothesis that there is no audible difference" do you mean that most blind tests are flawed? sorry I'm not english native and got kinda confused

I think he’s saying that the results of most blind tests fail to show that there is a audible difference. Not because they are flawed but because the participants simply are unable to hear a difference on a statistically significant level.

thank you. i think my brain skips the word "reject" and thought SIY said that nearly all tests failed to prove that there is no audible difference or something around the line

 

[Samudra1825]

If someone gets a statistically significant result distinguishing (blind) between cables, for instance, that is a result that rejects the null hypothesis, not *proof* that they can hear a difference

is there any difference between rejecting the null hypothesis and proof that they can hear a difference?

 

[ahofer]

is there any difference between rejecting the null hypothesis and proof that they can hear a difference?

Yes, the former does not reach the level of the latter - actual "proof" of a causal relationship is not really something you can show statistically. But we accept a correlation as truth it if it is repeatable, as with drug clinical trials.

We're stuck with "unlikely" or "likely".

Maybe next I should cover how we might use Bayesian inference to save time in audio decisions....

 

[SIY]

is there any difference between rejecting the null hypothesis and proof that they can hear a difference?

I wouldn't say "proof." But certainly if the results are consistent with a hypothesis of audibility, it's actual real evidence.

edit: ahofer got there first.

 

[Mal]

I was hoping for a catalogue, but this thread is another general chat about blind testing. Some interesting points being made, but is there an actual catalogue of speaker blind tests anywhere on the web? As the OP said, it's probably best to make that a wiki... he must have been thinking that forums are really about chatting, not providing blunt information (nothing wrong with that, but sometimes you just want a catalogue...)

 

[ahofer]

I was hoping for a catalogue, but this thread is another general chat about blind testing. Some interesting points being made, but is there an actual catalogue of speaker blind tests anywhere on the web? As the OP said, it's probably best to make that a wiki... he must have been thinking that forums are really about chatting, not providing blunt information (nothing wrong with that, but sometimes you just want a catalogue...)

There are links to some lists throughout, but Amir did not want to add a wiki to the site.

 

[Mal]

There are links to some lists throughout, but Amir did not want to add a wiki to the site.

But that means reading all the chat! What's needed is a link to all lists - a proper catalogue. No reason why someone can't create a wiki on another site and link to that from here. Then again, it doesn't need to be a wiki, maybe a web page would do, as long as the creator is dedicated and has a completist tendency!

 

[ahofer]

But that means reading all the chat! What's needed is a link to all lists - a proper catalogue. No reason why someone can't create a wiki on another site and link to that from here. Then again, it doesn't need to be a wiki, maybe a web page would do, as long as the creator is dedicated and has a completist tendency!

Are you volunteering?

 

[Mal]

Do partially-sighted speaker faceoffs tally well with blind test results? If they can be shown to tally well then we we can trust many comparisons - anyone can do a face off (and many do...), while blind tests are too tricky for most to attempt. Example:

"We did sighted tests, but made rough attempts to pay little attention to which speakers were in positions 1 or 2. Gene set up each listening session so the rest of us were oblivious, at least until we got further along, which speaker we were initially listening to. We did not attempt to do a controlled blind test, nor did we desire to set up any sort of mechanism to remove all forms of bias during the process. This was mostly due to time and the equipment on hand to facilitate such a process. We used no speaker toe-in having all of the speakers firing directly ahead in the listening area."

Bookshelf Speaker Faceoff 2007

The gloves come off as Audioholics takes on eight pairs of bookshelf speakers priced from $200 - $1800 a pair and tells you who leads the pack and who you should send packing. While there may not be

www.audioholics.com

 

[ahofer]

Do partially-sighted speaker faceoffs tally well with blind test results? If they can be shown to tally well then we we can trust many comparisons - anyone can do a face off (and many do...), while blind tests are too tricky for most to attempt. Example:

"We did sighted tests, but made rough attempts to pay little attention to which speakers were in positions 1 or 2. Gene set up each listening session so the rest of us were oblivious, at least until we got further along, which speaker we were initially listening to. We did not attempt to do a controlled blind test, nor did we desire to set up any sort of mechanism to remove all forms of bias during the process. This was mostly due to time and the equipment on hand to facilitate such a process. We used no speaker toe-in having all of the speakers firing directly ahead in the listening area."

Bookshelf Speaker Faceoff 2007

The gloves come off as Audioholics takes on eight pairs of bookshelf speakers priced from $200 - $1800 a pair and tells you who leads the pack and who you should send packing. While there may not be

www.audioholics.com

For cataloguing purposes, if there is an attempt at volume-leveling and unsighted listening, I’d include it. In the list linked in the first post there were some pretty dodgy tests (and then even more dodgy stats conclusions from Stereophile-an interpretation that would have received a zero on an entry-level college stats quiz)

 

[ahofer]

I hadn’t seen this one. Interesting that he documented the subject’s satisfaction with the setup prior to the test, but after some practice switching.