Actual Double Blind Studies

[hawkjeff]

I just read yet another person trying to defend a position related to audio by referencing "double blind studies" generally. The idea always sounds great as a listening test. Truly see if people are able to notice differences between things. If we use a fairly controlled method, this information should grow in precision over time. The more blind tests that are done, the closer the results of those tests should point towards some actual underlying truth. This would be fantastic for all sorts of reasons. The problem is, this sort of testing does require enough repetition to make it valuable. And if that repetition was there, it should be tracked somewhere, right? So why is it so hard to find results of such tests? Is there anyone who does these on a large scale? I think it would be great if it was done.

Does anyone have any links to high quality double blind listening tests and their results? Does such a thing exist?

 

[JSmith]

There are heaps of thread on such here;

Search results for query: blind test

www.audiosciencereview.com

So why is it so hard to find results of such tests?

Maybe because most of the time the results don't show what was "expected"... especially when it comes to cables and DAC's. Either that, or the old argument comes out that they're "too stressful"... test setup not good enough etc.

Oh, btw welcome to ASR.


JSmith

 

[Sancus]

Read Dr. Toole's book(citations at the end) and search the AES database. You'll find many double blind studies. They're hard to run properly but hardly uncommon let alone hard to find.

 

[DVDdoug]

It's rare, and yes it's difficult and time consuming. It wouldn't be practical for Amir do blind tests of the equipment he reviews. When magazines had staffs they could have arranged it. I don't know is Stereophile has the staff, but I suspect they are opposed to it (I don't read it).

Single-blind tests are a little more common and there are "tricks" so the subject can't see the (non-blinded) person running the test.

There is software for running a blind ABX Test to compare two files (such as comparing an MP3 to the lossless original) It's single-blind but software is randomizing and running the test and it's non-biased and it doesn't give any hints.

There have been ABX boxes built so a computer can take care of randomizing and switching between hardware but they are rare and expensive.

Maybe because most of the time the results don't show what was "expected".

TRUTH! Most "audiophiles" don't believe in blind listening tests... They start making all kinds of excuses about why blind tests are invalid.

HydrogenAudio (ABX link above) is "dedicated to" blind listening tests and you can't post any claims about sound quality without one. But other than using the software to compare files not many people there are doing it.

P.S.
Blind wine tasting is common. I'm not sure if it's double-blind, but probably so. The famous Pepsi Challenge was also blind.

 

[hawkjeff]

There are heaps of thread on such here;

Search results for query: blind test

www.audiosciencereview.com

Maybe because most of the time the results don't show what was "expected"... especially when it comes to cables and DAC's. Either that, or the old argument comes out that they're "too stressful"... test setup not good enough etc.

Oh, btw welcome to ASR.


JSmith

Thanks. I'm a longtime lurker.

I checked (again just now) on the first five threads for that search. One of them has an actual double blind test and it is done in his space with a dozen or maybe 15 persons. He runs through a list of problems with his tests. As you go down the list, more and more of them are individuals running their own tests or maybe someone running it with friends. I still have yet to be able to find anything that would be remotely acceptable and/or meaningful in any scientific way. That's not to say it isn't there or to question what the results would be if it was, but just to say that I haven't been able to find them.

 

[hawkjeff]

It's rare, and yes it's difficult and time consuming. It wouldn't be practical for Amir do blind tests of the equipment he reviews. When magazines had staffs they could have arranged it. I don't know is Stereophile has the staff, but I suspect they are opposed to it (I don't read it).

Single-blind tests are a little more common and there are "tricks" so the subject can't see the (non-blinded) person running the test.

There is software for running a blind ABX Test to compare two files (such as comparing an MP3 to the lossless original) It's single-blind but software is randomizing and running the test and it's non-biased and it doesn't give any hints.

There have been ABX boxes built so a computer can take care of randomizing and switching between hardware but they are rare and expensive.


TRUTH! Most "audiophiles" don't believe in blind listening tests... They start making all kinds of excuses about why blind tests are invalid.

HydrogenAudio (ABX link above) is "dedicated to" blind listening tests and you can't post any claims about sound quality without one. But other than using the software to compare files not many people there are doing it.

P.S.
Blind wine tasting is common. I'm not sure if it's double-blind, but probably so. The famous Pepsi Challenge was also blind.

It does seem that double blind tests would be hard to implement and therefore rare or hard to have as useful. The problem is, so many people point to them as if there is some master database of double blind studies that show what they want it to show. I just want to see those studies or else referencing them means nothing. And it has been really hard to find any, let alone enough to consider them scientifically valuable.

Maybe it is a pipe dream but I think it would be fascinating to see it done on some kind of larger scale. I've thought it would make a good touring exhibit for things like museums of science. The exhibit could be all about sound and part of it could be some kind of listening room/testing center so you could get large numbers through it.

 

[MAB]

Thanks. I'm a longtime lurker.

I checked (again just now) on the first five threads for that search. One of them has an actual double blind test and it is done in his space with a dozen or maybe 15 persons. He runs through a list of problems with his tests. As you go down the list, more and more of them are individuals running their own tests or maybe someone running it with friends. I still have yet to be able to find anything that would be remotely acceptable and/or meaningful in any scientific way. That's not to say it isn't there or to question what the results would be if it was, but just to say that I haven't been able to find them.

On page 78 is a quite famous test between some very well known high-end amplifiers published in Stereo Review:

https://americanradiohistory.com/Archive-HiFI-Stereo/80s/HiFi-Stereo-Review-1987-01.pdf

I worked in a store that sold Levinson amps at the time, to be honest I wasn't surprised.

 

[ahofer]

Catalogue of blind tests

I thought this thread on Head-Fi was a valiant effort to put together blind tests that have been performed over the years. Given that there have been many more (I'm thinking Archimago's tests, among others), it would be fun to open source a complete list...

www.audiosciencereview.com

I halfheartedly keep it up, but there are many there, and links to lists of many more.

 

[hawkjeff]

Catalogue of blind tests

I thought this thread on Head-Fi was a valiant effort to put together blind tests that have been performed over the years. Given that there have been many more (I'm thinking Archimago's tests, among others), it would be fun to open source a complete list...

www.audiosciencereview.com

I halfheartedly keep it up, but there are many there, and links to lists of many more.

Thanks. I'll dig in. I see the first comment is that it would be fun to build a master list. Did anyone ever do so? I guess I'll read and find out.

 

[ahofer]

Thanks. I'll dig in. I see the first comment is that it would be fun to build a master list. Did anyone ever do so? I guess I'll read and find out.

No. Amir doesn't like wikis, and I was more interested in crowdsourcing than compiling on my own. I think also, since it's been a while, there's some link rot.

 

[flipflop]

I still have yet to be able to find anything that would be remotely acceptable and/or meaningful in any scientific way. That's not to say it isn't there or to question what the results would be if it was, but just to say that I haven't been able to find them.

The problem is, so many people point to them as if there is some master database of double blind studies that show what they want it to show. I just want to see those studies or else referencing them means nothing. And it has been really hard to find any, let alone enough to consider them scientifically valuable.

You might want to scroll back up and read post #3.

 

[ahofer]

No. Amir doesn't like wikis, and I was more interested in crowdsourcing than compiling on my own. I think also, since it's been a while, there's some link rot.

I guess I'd add that I didn't try to catalog the stuff in AES, as I don't have access to it all. Those, of course, would be the most scientific studies, but going after the audibility of much narrower phenemona. Such as this:

AES Journal Forum » Audibility of a CD-Standard A/DA/A Loop Inserted into High-Resolution Audio Playback

The AES Forum is where our members get to interact with each other in between conferences and section meetings.

secure.aes.org

The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

One of my favorite less scientifically rigorous ones is the matrix hifi club, since they put so much cheap stuff in the signal chain.

Matrix HiFi --> Blind testing high end full equipments

www.matrixhifi.com

But the one that planted the objectivist seed in my mind was Stereo Review, 1988. Notice how the test bends over backwards to deal with audiophile objections.

(Stereo Review, Jan 1987, start around page 78 of the magazine, 80 of the pdf)

Stereophile ran another one comparing amps, and absolutely murdered stats to try to position it as a pass (one or two people in a roomful is P-hacking, and they completely ignore the bias to hear a difference, and even then, the results are far from convincing)

Blind Listening Page 2

System & Methodology As suggested in one of this month's "Letters," I had actually considered restaging the 1985 "Carver Challenge," where a Carver M1.0t could be compared with a pair of Conrad-Johnson Premier Fives. In the event, however, I decided that as Stereophile had just bought a recent...

www.stereophile.com

But I think the most important question you should ask yourself is this:

If audio companies had discovered an audible phenomenon that a) didn't show up in measurements and/or b) went below well-known thresholds of human hearing, wouldn't they be publishing, demonstrating, or making patents?

I think that one question puts the industry's claims in perspective.

@hawkjeff does that help?

 

[ctrl]

I just want to see those studies or else referencing them means nothing. And it has been really hard to find any, let alone enough to consider them scientifically valuable.

ABX tests are a common tool in scientific papers, such as on the audibility of group delay (as an effect of phase shift) from 2021 for example - Audibility of Group-Delay Equalization

But another major benefit of ABX testing affects the individual. ABX tests (for example using ABX function of audio software foobar) give an immediate answer to questions, because for you personally the result is immediately available and valid.
Can you distinguish a MP3 VBR320 from lossless audio, can you hear the phase shift (and resulting group delay) at a fourth order crossover @2kHz, ...
To decide this for yourself, a few ABX tests you perform are enough.

 

[hawkjeff]

You might want to scroll back up and read post #3.

If you mean the post with the Dr. Toole’s recommendation then I did read it and investigated quite a bit. I’ve wanted that book for a while, though it is $90 which is a lot for me. The AES database mentioned looks super interesting, but again is a paid service. $33 per report. Also, an AES search for “double blind” turns up nothing relevant. Maybe the info is there, but I can’t find it, nor am I paying for it. Especially when the articles that come up include ones from over 40 years ago. It isn’t that it is surprising, mind you. Legitimate scientific studies cost a lot of money and are challenging to implement. I still think it is very odd that people run around referencing the results of double blind studies when they can’t provide any actual citations.

 

[ahofer]

You can also try a search like the one below on google scholar. "ABX" is a common test description. You'll still get spurious results, but find more.

(audio OR auditory) AND (double-blind OR ABX) AND (distortion OR resolution OR phase)

Play around with the search string for your pet issue (digital encoding, distortion, phase, frequency response)


see also: http://djcarlst.provide.net/abx_peri.htm

Is there any way to archive these reference articles?

There seems to be alot of good reading in the following cable related reference materials. AES articles probably be easy to get [by paying a fee], but

forums.audioreview.com

 

[ahofer]

There is a sort of problem (non-problem?, lacuna?) with scientific papers as I understand it: Some of the things audiophiles aver don't even merit a paper, for much of the society. Differences between cables, classes of amplification, etc. probably just don't seem like worthy subjects of investigation, other than by wire or amp tech advocates. So an absence of papers might say to you "they haven't investigated it" whereas it might really mean "most engineers think the topic is settled enough not to merit further study".

I'm not an AES member, so correct me if I'm wrong, but that happens in a lot of fields, and mostly (but not entirely) for the right reasons.

 

[dominikz]

The AES database mentioned looks super interesting, but again is a paid service. $33 per report.

Some of the AES articles are free ("open access") - you can search these here.
By the way, AES members get full access to all AES articles while the membership is active, though I understand the yearly membership fee might be too much if one doesn't require access for work.
Web of Science, Mendeley, ResearchGate and Google Scholar can also be useful, but a lot of full-texts will again be behind paywalls.

It can be difficult to start reading scientific literature, but once you do you'll see that finding additional relevant research becomes easier as it is referred to in the articles you've already read. This is why starting with an overview book from a field you're interested in is IMHO very good advice - e.g. dr. Toole's book if you're interested in loudspeakers.
Perhaps you can find a library that has the book if you'd rather not buy it?

 

[ahofer]

Toole is worth purchasing. You should think of it as a meta-study of other science in the field.

 

[Anton D]

I have always been curious about 'training' for these tests.

I saw a visual test that used an aerial pic of a full Rose Bowl, and instantaneous ABX testing was used to compare two pics and seeing if people could notice the difference between them. The subjects couldn't do it.

Then, the tester pointed out the one difference: one person in a different color shirt(out of, what, 100,000 people?) and after that the subjects had a 100% success rate. They had been taught to spot the difference.

Obviously not the same for audio, but are there listening tests where people were taught in advance the differences and looking to see how smaller and smaller 'learned' differences could still be identified blind?

For electronics, with good headphones, perhaps there is a way for people to hear in more detail what we measure. Not claiming any knowledge of this, just chatting.

No idea if this has been done and to what degree. I have seen results looking at difference sonic thresholds, but I do not know if there is a further training effect that improves performance.

 

[ahofer]

I have always been curious about 'training' for these tests.

I saw a visual test that used an aerial pic of a full Rose Bowl, and instantaneous ABX testing was used to compare two pics and seeing if people could notice the difference between them. The subjects couldn't do it.

Then, the tester pointed out the one difference: one person in a different color shirt(out of, what, 100,000 people?) and after that the subjects had a 100% success rate. They had been taught to spot the difference.

Obviously not the same for audio, but are there listening tests where people were taught in advance the differences and looking to see how smaller and smaller 'learned' differences could still be identified blind?

For electronics, with good headphones, perhaps there is a way for people to hear in more detail what we measure. Not claiming any knowledge of this, just chatting.

No idea if this has been done and to what degree. I have seen results looking at difference sonic thresholds, but I do not know if there is a further training effect that improves performance.

There is definitely training (@Amir has had it). In particular, you can be trained to locate the artifacts of lossy encoding and certain types of distortion. My understanding is that it doesn't go beyond normal audible thresholds, but helps you do better than average in a distortion test or MP3 vs lossless test. Also, it teaches you to correlate certain timbral presentations to the proper frequency imbalance.

Much like the sommelier training, where oenophiles learn to correlate color, viscosity and several flavor notes to vintages and varietals through lots of practice with samples.

PS - yes, headphones are much better instruments for detecting tiny differences.