“Human ears are better than measurements!” / “Humans can hear things we can’t even measure!”

[Matt_Holland]

Those aren't our problem. ASR has no control over them. Why are you even asking?

Because nearly all reviews are uncontrolled and highly biased, it promotes an unscientific approach to what should be a scientific one to the consumer.

Again, it's called preference testing, and it has a long history in multiple consumer product areas, and it has been crucial to the advances in loudspeaker design since the 1980s.

Are you saying you can 'prefer' something that you 'enjoy' less?

I was responding to a previous comment where they were suggesting that audio products could be designed with specific traits that made them more enjoyable than their competition.

I think a studio mastering engineer can absolutely prefer a product/system’s sound quality that they enjoy less than another’s. It depends what the role of the system is. But I take your point.

The 'audio testing world' is not the 'customer' of manufacturers. 'Reviews' of audio gear in mainstream publication rarely do any credible testing, and where they do (Stereophile), they separate it from the 'review'.

I was being flippant and cynical, but we actually agree (apologies if my wording was confusing, I meant that the manufacturers are the customers). There are mostly no bad reviews as publishers don’t want to upset the manufacturer who might withhold products for future reviews, or (highly cynical) stop buying advertising.

 

[krabapple]

Because nearly all reviews are uncontrolled and highly biased, it promotes an unscientific approach to what should be a scientific one to the consumer.

And this is news, here?

 

[Matt_Holland]

And this is news, here?

Ok, grumpy. Point taken

 

[Cbdb2]

Thanks but I’m asking where are the controlled listening tests in the mainstream equipment reviews published.

Are you kidding. How much over expensive gear would they sell if they coudnt tell the difference between there uber $$$ Amp or DAC and a $200 one. Which is exactly what would happen if they ever did a proper ABX. There rags would be gone in a month.

 

[RandomEar]

Out of interest and to test the boundaries I have done an ABX comparison using pink noise generated at -6 dB and then attenuated by 0.5 dB down to 0.05 dB. The results were:

Attenuation vs. reference	foobar2000 ABX score	p-value
-0.5 dB	10/10	0.1%
-0.3 dB	8/10	5.5%
-0.2 dB	8/10	5.5%
-0.15 dB	8/10	5.5%
-0.10 dB	7/10	17.2%
-0.05 dB	7/10, 6/10 (repeated)	17.2%, 37.7%

I'm pretty confident I could score higher on the -0.3 dB run, but my patience today is limited For -0.2 dB and -0.15 dB, I'm sceptical I could repeatedly score 8/10. For -0.1 dB and below, I was essentially guessing but I'm surprised that my guesses did not lead to a single score below 6/10. I think we would need closer to 40 runs to get reliable results for those.

In conclusion, for critical listening tests I would not be happy with level matching of worse than 0.1 dB. Even considering that Scott et al. (2003) showed that noise is more revealing than music for SPL differences, you want to be on the safe side here.

Quoting myselfs here. Since I wasn't happy with the statistical power of the previous ABX results, I increased the number of runs to a total of 30 (so 10 or 20 additional runs per attenuation). This gave the following results:

Attenuation vs. reference	foobar2000 ABX score	p-value
-0.5 dB	30/30	0.0%
-0.3 dB	28/30	<0.01%
-0.2 dB	28/30	<0.01%
-0.15 dB	23/30	0.26%
-0.10 dB	20/30	4.94%
-0.05 dB	19/30	10.0%

This is much better than expected. Everything from -0.5 to -0.2 dB was 20/20 in the additional tests, with -0.15 dB giving 15/20 (significant as is) and -0.1 dB being down to 13/20 (not significant itself, but barely significant when combined with the previous run of 7/10). I think it's particularily nice to see the monotonically increasing nature in the p-value with regards to the attenuation level. From my perspective, this is an indicator that the number of repetitions per test is high enough to give reliable results.

Concerning my subjective impression of the audibility, I will say that 0.5 dB is pretty clearly audible as a difference in loudness while lower differences (<= 0.2 dB) sounded more or less equally loud but the louder track was "brighter". I would assume that this is linked to the equal loudness contour of our hearing. So having the constant amount of high frequency content in the form of pink noise definitely made this test much easier than it would have been with music.

From a perspective of difficulty, I expected more errors at 0.2 dB while running the test and was surprised by the results. Just to get an idea: The -0.5 dB test took me around 3 minutes for 20 runs, while the -0.2 dB test took 10 minutes and the -0.1 dB test was around 15 minutes. For -0.1 and -0.05 dB I would still say I was mostly guessing, but clearly, there was something there.

Taking these additional results into account and recognizing that my ears may not be the best on the planet plus that I'm not a trained listener, I would revise my previous assumption that level matching to 0.1 dB should be sufficient for critical listening tests (with near-instant switching). While loudness differences should be harder to detect in music than in noise, you can't test all types of music using ABX methods. So any safe threshold should be based on the results determined using noise, as long as no better method has been established. Therefore, the safe threshold is very likely below 0.1 dB. Maybe it is even below 0.05 dB, which sounds kinda absurd.

 

[voodooless]

Congrats on being a very precise relative SPL meter

 

[RandomEar]

Congrats on being a very precise relative SPL meter

You never know when you're gonna need it! (probably never)

 

[gzost]

Are you kidding. How much over expensive gear would they sell if they coudnt tell the difference between there uber $$$ Amp or DAC and a $200 one. Which is exactly what would happen if they ever did a proper ABX. There rags would be gone in a month.

If HiFi magazines were reality-based the main topics would be speaker/room interactions, room treatment and correction. That would not support the purchase of shiny gadgets, and have an even lower WAF than present hifi systems have. There really is no economic sense for the audiophile press to be sensible.

(stereophile has a "room treatment" review category, which is ... a bit of a wasteland: https://stereophile.com/category/room-treatment-reviews)

 

[solderdude]

about 35 years ago I built an attenuation box (which doubled as A-B box)

I could instantly attenuated the exact set attenuation level and used it (in those days) to determine some hearing limits and came to a less scientific but similar finding.
0.5dB level difference was easily detectable, 0.2dB was pretty hard to detect and 0.1dB I could not discern.
In the other direction of the scale, while playing pretty loud at no attenuation I could not hear anything once I passed 80dB attenuation.
Steps of 10dB indeed was as if the level was cut in half (or doubled).
The tests were 'sighted' as I operated the switches myself.

 

[ahofer]

Imagine if wine tasting was done this way though…

It often is done with labels obscured, which is the important part.