Audio Blind Testing - You Are Doing It Wrong! (Video)

[JRS]

My bad, I was looking at greater than 5. You're absolutely right when including 5, 6, 7. Cheers.

 

[musicforcities]

If there were an afterlife, perhaps the leaders of Audioquest will spend eternity forced to do double blind tests of their snake oil or worse products.

 

[MRC01]

At least Audioquest made some excellent sounding recordings back in the 1990s... recordings so good it's hard to believe they came from the same company that sells crazy expensive cables.

 

[pglee]

Since this is a measurement heavy site, I wonder what is the threshold where these measurement differences, make an audible difference? I admit I get geeked out on discovering that a new piece of gear's S/N is 5db greater than some gear I already own, but it's never really clear how significant that is in reality. Oddly finding blind listening tests to determine a realistic threshold of where measurement differences become perceptible let alone significant are basically non-existent online (maybe they exist on some IEEE papers behind a paywall).

 

[MRC01]

Since this is a measurement heavy site, I wonder what is the threshold where these measurement differences, make an audible difference? ...

Level of audibility depends on several factors: the source material, frequency spectrum, levels, the type of phenomena you're listening for, etc.
It's been discussed here before, you can find a whole set of "audible" and "safe" thresholds for various measurements that are based at least in part on DBT with trained listeners.

 

[Spkrdctr]

Since this is a measurement heavy site, I wonder what is the threshold where these measurement differences, make an audible difference? I admit I get geeked out on discovering that a new piece of gear's S/N is 5db greater than some gear I already own, but it's never really clear how significant that is in reality. Oddly finding blind listening tests to determine a realistic threshold of where measurement differences become perceptible let alone significant are basically non-existent online (maybe they exist on some IEEE papers behind a paywall).

You would be surprised at how little any of it is audible in a music listening situation. I would say it would be shocking!

 

[ahofer]

Since this is a measurement heavy site, I wonder what is the threshold where these measurement differences, make an audible difference? I admit I get geeked out on discovering that a new piece of gear's S/N is 5db greater than some gear I already own, but it's never really clear how significant that is in reality. Oddly finding blind listening tests to determine a realistic threshold of where measurement differences become perceptible let alone significant are basically non-existent online (maybe they exist on some IEEE papers behind a paywall).

Some here:

Audibility thresholds of amp and DAC measurements

Introduction A recurring theme on ASR is whether or not the various measured qualities of the devices are audible. In this post, I'll present some clear and visual thresholds for when those imperfections can be considered a potentially audible concern. I will not explain the basics of amp/DAC...

www.audiosciencereview.com

 

[pglee]

Some here:

Audibility thresholds of amp and DAC measurements

Introduction A recurring theme on ASR is whether or not the various measured qualities of the devices are audible. In this post, I'll present some clear and visual thresholds for when those imperfections can be considered a potentially audible concern. I will not explain the basics of amp/DAC...

www.audiosciencereview.com

Ah. Good find. Thanks. This comment from amir is of note:

"Just a note: an exhaustive analysis and proof simply does not exist for any of this. As such, take them as guidelines which are better than having none."

 

[richard12511]

Since this is a measurement heavy site, I wonder what is the threshold where these measurement differences, make an audible difference? I admit I get geeked out on discovering that a new piece of gear's S/N is 5db greater than some gear I already own, but it's never really clear how significant that is in reality. Oddly finding blind listening tests to determine a realistic threshold of where measurement differences become perceptible let alone significant are basically non-existent online (maybe they exist on some IEEE papers behind a paywall).

For me, I think it's likely somewhere in that -60dB to -80dB range. Stuff in that range might be ever so slightly audible with the right pieces of music, but so small and insignificant that I don't really care. Difference in that range is smaller than the difference between two speakers of the exact same model, and I tend to not worry about sample variation with speakers much.

I'm pretty certain that anything beyond is -80dB is equivalent to my(and most) ears with music. Going beyond that tends to not really be relevant to actually listening to music. It often requires specific training and access to instant volume matched switching and short clips. You're not gonna be able to hear it by just plugging in a new device and listening.

 

[paulraphael]

The gold standard for testing should be double-deaf.

 

[cinemakinoeye]

Bravo! Thank you! A refreshing perspective, and as @Andreas007 points out, valuable in fields outside of the audio field.

 

[PeteL]

For me, I think it's likely somewhere in that -60dB to -80dB range. Stuff in that range might be ever so slightly audible with the right pieces of music, but so small and insignificant that I don't really care. Difference in that range is smaller than the difference between two speakers of the exact same model, and I tend to not worry about sample variation with speakers much.

I'm pretty certain that anything beyond is -80dB is equivalent to my(and most) ears with music. Going beyond that tends to not really be relevant to actually listening to music. It often requires specific training and access to instant volume matched switching and short clips. You're not gonna be able to hear it by just plugging in a new device and listening.

If this is really true, that would make something like the recently reviewed Topping PA5 an outrageously expensive piece of gear, if it doesn't offer anything more audibly that all those integrated in the same price range with a much larger set of features and that can offer that range in SINAD.

 

[krabapple]

Gee, ya think?

 

[PeteL]

Gee, ya think?

I do not know if we can rigorously confirm that anything 60 to 80 dB below the desired signal is rigorously inaudible, but it's my belief that anythig rigorously inaudible is a waste of engineering resources and money. Manufacturers design stuff to be listened to, working on stuff that nobody will hear is useless. But again, this 60-80 dB is not me saying.

 

[krabapple]

No one is saying 'rigorously inaudible', including the poster you replied to: "stuff in that range might be ever so slightly audible with the right pieces of music, but so small and insignificant that I don't really care."

It's a matter of priorities. Do I care about the rare chance of such 'stuff' being audible? No. I'd rather focus on big, definite impactors on audio (like the room, the speakers, the quality of the mastering of the source, or the upmixing options available to me...)

So, I couldn't care less about the better measurements of a Topping PA5 vs an AVR in this regard, and yes, I'd consider such incremental improvement 'outrageously expensive' at its price.

 

[respice finem]

I do not know if we can rigorously confirm that anything 60 to 80 dB below the desired signal is rigorously inaudible, but it's my belief that anythig rigorously inaudible is a waste of engineering resources and money. Manufacturers design stuff to be listened to, working on stuff that nobody will hear is useless. But again, this 60-80 dB is not me saying.

Essentially, you're right, but there are a few "buts" about this IMHO:

• A hobby is not all about sense, it's always a bit of personal "bias", emotion, etc. In this regard, the better will always be the enemy of the good.
• No one can say exactly what the other one hears (or what his brain makes of it). This is almost completely independent from measurements / parameters etc. (the LP hype is a good example)
• While a piece of a chain may be good enough as it is, two, three or more of them might not, because their "sins" are additive.
• The industry has to live off something

To the last point, I'm glad that the industry finally starts to acknowledge the "elephant in the room", which is... the room itself.
The market for absorbers, diffusers, room correction is growing, slowly but steadily. This may IMHO improve the "final" SQ for the average listener far more than another shiny new piece of audio gear.

 

[PeteL]

Essentially, you're right, but there are a few "buts" about this IMHO:

• A hobby is not all about sense, it's always a bit of personal "bias", emotion, etc. In this regard, the better will always be the enemy of the good.
• No one can say exactly what the other one hears (or what his brain makes of it). This is almost completely independent from measurements / parameters etc. (the LP hype is a good example)
• While a piece of a chain may be good enough as it is, two, three or more of them might not, because their "sins" are additive.
• The industry has to live off something

To the last point, I'm glad that the industry finally starts to acknowledge the "elephant in the room", which is... the room itself.
The market for absorbers, diffusers, room correction is growing, slowly but steadily. This may IMHO improve the "final" SQ for the average listener far more than another shiny new piece of audio gear.

I agree with all you say.

 

[Spocko]

No one is saying 'rigorously inaudible', including the poster you replied to: "stuff in that range might be ever so slightly audible with the right pieces of music, but so small and insignificant that I don't really care."

It's a matter of priorities. Do I care about the rare chance of such 'stuff' being audible? No. I'd rather focus on big, definite impactors on audio (like the room, the speakers, the quality of the mastering of the source, or the upmixing options available to me...)

So, I couldn't care less about the better measurements of a Topping PA5 vs an AVR in this regard, and yes, I'd consider such incremental improvement 'outrageously expensive' at its price.

So this discussion now goes to why Harman et al. have not focused on what appears to be a most relevant piece of research for consumer electronics: "A what point can we say changes in sound quality are rigorously inaudible?" The insight from this research would allow consumer electronics to focus on more important areas: features, reliability, UI design, surround immersion, DSP room correction, etc.

The conclusions may also be frequency specific as well as volume specific. Above a certain volume, certain distortions may be inaudible depending on the frequency of the distortion and the masking that may arise from nearby frequencies at that volume. Hmmm, this does sound a bit more complicated than I first thought...

 

[krabapple]

Research into just audible thresholds are the province of academic psychoacoustics, and there is a literature on it. Even a wikipedia article.

 

[Spkrdctr]

So this discussion now goes to why Harman et al. have not focused on what appears to be a most relevant piece of research for consumer electronics: "A what point can we say changes in sound quality are rigorously inaudible?" The insight from this research would allow consumer electronics to focus on more important areas: features, reliability, UI design, surround immersion, DSP room correction, etc.

The conclusions may also be frequency specific as well as volume specific. Above a certain volume, certain distortions may be inaudible depending on the frequency of the distortion and the masking that may arise from nearby frequencies at that volume. Hmmm, this does sound a bit more complicated than I first thought...

Yes, you are seeing at the end of your post that it is a HUGE rabbit hole that will suck up an engineering department and keep them busy for years. I also wish we had some standards for audibility. It would help manufacturers to stop wasting money chasing a spec that is not even audible! Also, a study on what is really in the music/movie sound frequencies. For example, I will make up an example. Lets say that we find that there is almost or basically nothing out there over 15kz. This would then allow companies to focus on 20 to 15kz and not have to worry at all about 25kz in their designs. I believe that if we had some serious, well done studies on audibility and performance of audio gear, the data would be screaming at us that room treatment and EQ should be our #1 focus after a good decent set of speakers. Then the market place could come up with all kinds of room friendly, very inexpensive room treatments that anyone can easily use. Making room treatment and EQ extremely easy and simple to do/use would fix a lot of in home sound issues that destroy a great set of speakers. But, the market is in the Buy More Expensive to get better sound rut and can't break out of it. Well, that is my two cents. Oh, and I'm drinking coffee too!