• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Audio Blind Testing - You Are Doing It Wrong! (Video)

respice finem

Major Contributor
Joined
Feb 1, 2021
Messages
1,867
Likes
3,774
Excellent video. Not only for audio!
Much needed in times where an increasing number of people think that their subjective personal experiences are more trustworthy than scientific methods.
Psychology. The experience is mine, the other stuff is somebody else's - everybody understands the former, but not the latter.
Add bias to it, in all possible forms, and you know why it is, as it is.
 

johnnyx

Member
Joined
Feb 17, 2019
Messages
15
Likes
22
Audiophiles are obsessed with differences, especially regarding which is the best. I have two different systems in two different rooms, and I am struck by how similar they sound
 

audio2design

Major Contributor
Joined
Nov 29, 2020
Messages
1,769
Likes
1,830
Well the tests are worth "their" time. It really helps to line their pockets.
 

stevenswall

Major Contributor
Forum Donor
Joined
Dec 10, 2019
Messages
1,366
Likes
1,075
Location
Orem, UT
Audiophiles are obsessed with differences, especially regarding which is the best. I have two different systems in two different rooms, and I am struck by how similar they sound

What two systems? I wonder if Amir could describe the differences. I've run into a wall with being able to identify differences when I'm in the sweet spot with excellent speakers performing within their limits: I can't identify an audible issue and describe it off hand without pushing the speaker (and then it's just an SPL game) or moving around (to determine if something is coaxial.)

There's an interesting phenomenon in some native people who inhabit rainforests where they can literally see differences in green that most can't because they have had the words to describe it since birth and their brain it better at differentiating the shades they are seeing.
 

sajgre

Active Member
Joined
Mar 6, 2021
Messages
133
Likes
232
I'm so proud of my past self. I was buying equipment based on internet reviews but I was not ashamed to admit that I don't hear a difference between dedicated 1k$ dac and Yamaha xc50. I just thought that my ears suck until I discovered this place. Please keep sharing knowledge
 

AdamG

Proving your point makes it “Science”.
Moderator
Forum Donor
Joined
Jan 3, 2021
Messages
4,636
Likes
14,918
Location
Reality
And here I am, all this time thinking that “Blind” audio listening is when you hit shuffle on a random playlist? :oops:
 

Azathoth

Active Member
Joined
Oct 18, 2020
Messages
112
Likes
255
Psychology. The experience is mine, the other stuff is somebody else's - everybody understands the former, but not the latter.
Add bias to it, in all possible forms, and you know why it is, as it is.

I stumbled upon this video recently and it is undeniably relevant here. She makes a good point about using our senses alone. Great video, and channel as well for those who like physics.
 

ExUnoPlura

Member
Forum Donor
Joined
Jul 23, 2020
Messages
51
Likes
128
Location
Oregon Coast
Good stuff for deconstructing some of the nonsense out there, but there are so many experimental design issues and biases (individual differences, learning effects, frequency effects, recency effects, etc.) that only large population evaluations start to asymptotically approach something like a valid result. The Harman preference curve is a good example of an aggregate metric that is clearly useful and its development allows for some comparisons without the need for huge testing populations. The bottom line, since the enlightenment, is that public knowledge is valuable and private knowledge (or unverifiable personal gnosis or hope or whatnot) is relegated to a lesser status, expert opinions included.
 

audio2design

Major Contributor
Joined
Nov 29, 2020
Messages
1,769
Likes
1,830
Good stuff for deconstructing some of the nonsense out there, but there are so many experimental design issues and biases (individual differences, learning effects, frequency effects, recency effects, etc.) that only large population evaluations start to asymptotically approach something like a valid result. The Harman preference curve is a good example of an aggregate metric that is clearly useful and its development allows for some comparisons without the need for huge testing populations. The bottom line, since the enlightenment, is that public knowledge is valuable and private knowledge (or unverifiable personal gnosis or hope or whatnot) is relegated to a lesser status, expert opinions included.

You don't need a large population size for different / not-different. You just need one person to be able to reliably be able to tell the difference. You only need a population to make assumptions about preference.
 

ExUnoPlura

Member
Forum Donor
Joined
Jul 23, 2020
Messages
51
Likes
128
Location
Oregon Coast
You don't need a large population size for different / not-different. You just need one person to be able to reliably be able to tell the difference. You only need a population to make assumptions about preference.
Fair enough. Learning effects, frequency effects, etc. can run interference on that in repetitive test protocols, but randomized trials run long enough can at least give a basic signal that a person sees a difference.
 

pseudoid

Master Contributor
Forum Donor
Joined
Mar 23, 2021
Messages
5,116
Likes
3,416
Location
33.58 -117.88
I recently bought a Polk Reserve R350 Center/LCR speaker, which *HAD to be <5.7"Height and *HAD to be wood and *HAD to be non-ported.
R350's design, components, features, specifications, build quality and the words "Hi-Res Certified" sealed the deal/
I finally received the 20 pound R350 after waiting 2 months and never having auditioned it or never even having read a review of it.
Aesthetically (W.A.F.) the R350 matches perfectly with all the wood in the room.
...but it speaks a different language than my floor-standing speakers, even w/Atmos, balancing levels, and room equalization etc.
:rolleyes:Argh!
Double-Blind Tests?
:oops: In the 21st Century, us mere mortals are lucky if we can just audition a speaker...
 

Spkrdctr

Major Contributor
Joined
Apr 22, 2021
Messages
2,212
Likes
2,934
You don't need a large population size for different / not-different. You just need one person to be able to reliably be able to tell the difference. You only need a population to make assumptions about preference.

But what about the one guy who says "Hey! The emperor has no clothes"!
 

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,448
Likes
4,209
Very good.

BTW @amirm around 25m35s you explain how a minimum of 9 tests is needed to get confidence above 95%, you roughly said it comes from “if you are guessing A vs B and guess right once there is 50% chance you are guessing, get it right twice and it’s 25%….keep going until you get below 5%”.

Let’s do it.
1 test 50%
2 tests 25%
3 tests 12%
4 tests 6%
5 tests 3%

Whoops that only took 5 tests not 9, to get above 95%.

Can you expand on the difference please?

cheers
 

Spkrdctr

Major Contributor
Joined
Apr 22, 2021
Messages
2,212
Likes
2,934
In all seriousness I did want to commend Amir for a very, very good video that needs to be pinned at the top for years. That video stated very well what I did poorly on various threads. Amir, you knocked it out of the park! You hit all the needed points and explained each one as to why you had it in your presentation. I'm impressed with how ASR is clearly the leader of online audio. I don't know of a site that comes close. You presented it in a very laid back non-aggressive way and stressed it is for education and for your own knowledge. I almost stood and gave standing ovations for certain parts. It very pleasantly destroys all these people who don't know what they don't know. Spouting blind tests that they never did etc. I would do this daily on this forum but I would then be considered "not playing well with others", so I dial it back a lot!

I am still blown away. What an awesome learning video for all the regular people on this site. I will be happy for a week!
 

PeteL

Major Contributor
Joined
Jun 1, 2020
Messages
3,303
Likes
3,838
You don't need a large population size for different / not-different. You just need one person to be able to reliably be able to tell the difference. You only need a population to make assumptions about preference.
But how is it useful to prove difference, if doesn’t say anything about the superiority of one over the other?
 

Spkrdctr

Major Contributor
Joined
Apr 22, 2021
Messages
2,212
Likes
2,934
I was involved in difference testing. It started out as "which is better, A or B"? Then when all we got was random chance results, it was changed to "Can you hear ANY difference at all between A or B"? That still came up bust. We never found anything other than speakers that anyone could identify as different. We did interconnects, speaker wire, Amps, pre-amps and receivers. No one EVER found they could tell any difference at all, much less which was better. Speakers are getting better and better so many of them sound the same when leveled. Bass though is often times a big give away to the human brain. So, you will pick the bass heavier speaker as sounding better. The brain wants what it wants!
 

Bruce Morgen

Addicted to Fun and Learning
Forum Donor
Joined
Aug 3, 2019
Messages
864
Likes
1,237
Many of the folks involved in the industry are self taught -- neither Paul McGowan nor Danny Richie are EEs or have any sort of formal engineering background -- and they are therefore both unacquainted with what it takes to produce a credible professional paper and uninterested in test results that don't serve their marketing interests. I know McGowan employs folks who know the score in that regard, but it's his company and he controls what gets published under their names -- I'm sure he'd rather his people do friendly videos that have a positive marketing impact than let them publish test results that are unlikely to be particularly beneficial to the company bottom line.
 

JRS

Major Contributor
Joined
Sep 22, 2021
Messages
1,158
Likes
1,003
Location
Albuquerque, NM USA
Very good.

BTW @amirm around 25m35s you explain how a minimum of 9 tests is needed to get confidence above 95%, you roughly said it comes from “if you are guessing A vs B and guess right once there is 50% chance you are guessing, get it right twice and it’s 25%….keep going until you get below 5%”.

Let’s do it.
1 test 50%
2 tests 25%
3 tests 12%
4 tests 6%
5 tests 3%

Whoops that only took 5 tests not 9, to get above 95%.

Can you expand on the difference please?

cheers
I agree. There is something amiss. Number of ways to hit 9/10 is ten, number of ways to hit 10/10 is 1 so the p should be 11/1024. (1024 the total number of possible combinations). The odds for 8/10 (or better) is not far from 95%. One thing that was something of a small oversight IMO is that there is nothing inherently correct about using 90%. For example, if we were to do 20 trials, 13 of 20 would be sufficient to say with 95% "certainty" discrimination was occurring. Generally, the greater the number of trials, the smaller the deviation from 50% required. So when doing 1000 trials, something like 55% might be significant. The question of course is whether such a difference is worth it?
 
Last edited:
Top Bottom