DAC audible differences are finally proven with ABX?

[RandomEar]

Josh is way ahead of you.
1. He consulted with Amir and others to agree best methods for level setting LAST YEAR:

DAC audible differences are finally proven with ABX?

Sampling theorem, sampled signal must be limited to F Fs/2 and as such it cannot be reconstructed properly. Anti-aliasing input filter. Or oversampling etc., There is enough literature.

www.audiosciencereview.com

2. He measured a 1 kHz sine wave and broadband CTA-2034 pink noise with a DVM.
3. He measured the levels at the headphone amp output using a Cosmos ADC and REW.
4. He repeated the final test after balancing the channels to check they were matched.
5. He just published frequency response measurements showing the comparator outputs were very closely matched between channel A and channel B, between left and right, and at different comparator volume settings. At each volume setting, all the traces are on top of each-other.

https://audiophilestyle.com/forums/topic/72134-article-updated-can-i-tell-audibly-transparent-dacs-apart-blind-testing-with-the-van-alstine-abx-comparator/#findComment-1347665

View attachment 548760

That's weird. According to the manual Josh quoted here, each step between 10 and 99 on the comparator should be 0.1 dB. So these two measurements should be about 1 dB apart. But they are 4 dB apart in the plot.

Am I missing something? Is it because of the REW scale being in dB_SPL instead of maybe dB_V?

 

[RandomEar]

You don't use test tones for such tests, you use real music signal, preferably one with good stereo reverb tails. 0.1B (1%) level difference manifests itself as the louder one having more "resolution" and "depth" because the tails take longer until their components fall below thresholds (which are highly individual).

OK, but could you please link to a paper showing this? I would like to add it to my archive.

Further, one random single person's level difference detection threshold suggests that there are likely other people with lower thresholds. Not like 20dB lower of course, but several dB is certainly possible. And listening level is a major variable, of course.

I don't claim to be the benchmark. I recently investigated this and listed multiple sources here, which cluster around 0.2 dB for different triggers.

 

[KSTR]

OK, but could you please link to a paper showing this? I would like to add it to my archive.

I don't rely on, or search for, papers on such topics as the chance of someone already have done work is often minimal or incomplete. For decades, I've done my own research, also within the context of a professional audio company. And testing things like level differences is so trivially easy in these digital days that any hour spent on fruitless web search is much better invested of doing your own experiments because that's a) how you actually learn something and b) how you become a trained listener.
On that specific topic, there might be papers that you can find on your own if that makes your boat float.

 

[Geert]

OK, but could you please link to a paper showing this? I would like to add it to my archive.

EQ-ing sound with a precision of 0,1dB to 0,2dB is standard practice in music production.

 

[ElJaimito]

Why not? What potential error do you expect?

I've been quite specific already...

 

[Deleted member 93733]

+1 but for audiability small 0.1 dB deviations are clearly audible if they are over very wide range , say a filter setting that makes a gentle slope in fr from 10kHz

Where is your reference for this statement?

Subjective response tests conducted by Stevens, 1957, 1972; Zwicker, 1958; Zwicker & Scharf, 1965, found that using a test sound in the mid audio frequency range at pressures greater than 40 dB re. 20uPa observed that a 3dB change was just perceptible in a group of “normal subjects”. In my industry (Acoustic Engineer) it is a commonly accepted convention that a 1 dB change is just perceptible under lab conditions.

 

[donjoe0]

Read this.

https://www.researchgate.net/publication/279594319_The_Modification_of_Timbre_by_ResonancesPerception_and_Measurement

Sure, as soon as I get the authors to approve my request for the full text. Thanks for not even bothering to say what that study proves in the context of the discussion, by the way, that's very helpful.

What about current knowledge makes you doubt that 20Hz to 21kHz -0.5dB is not good enough ?

Severe lack of studies. There isn't even a single study cited for this in the audibility thresholds thread for example.

a 3dB change was just perceptible in a group of “normal subjects”. In my industry (Acoustic Engineer) it is a commonly accepted convention that a 1 dB change is just perceptible under lab conditions.

You might be confusing different issues, I wasn't talking about loudness differences between sequential presentations of a simplistic test sound but between different frequencies in the same complex signal (music most of all). Even for pure loudness discrimination, 1 dB should be easy for anyone with good attention/training, when I last tried this on audiocheck.net I could get something like 70+% correct answers out of 20+ attempts for the 0.5 dB test, with pure chance being 33% since it was a 3-option test (louder, quieter, same). Just at home, no lab, and using the venerable Superlux HD668b.

 

[solderdude]

0.1dB = around 1% deviation so at 1Vrms one needs to be able to read 0.990 to 1.010 to reach that.
My true RMS Fluke 117 at 6VAC is 0.1% accurate (+/- 1.5 count) between 45Hz and 500Hz.
So any frequency between 45Hz and 500Hz would do (when absolute accuracy is needed) but for comparisons absolute is not a requirement.
100Hz to 400Hz would be just fine and one can adjust 1.001 to 0.999 on the screen which would be perfectly fine = around 0.01dB.
At 1kHz this meter would still be fine for comparative measurements and just a bit off for absolute measurements.
So any half decent DMM with ACV setting and a sine between 100Hz and 400Hz would likely be enough to level match up to at least within 0.02dB.

Not all DMMs are created equal and what's missing in the data for instance is which DMM was used (lets not whine about calibration reports)

My personal detection limit for level differences is 0.2dB and at 0.1dB I was not able to detect a level difference.
When using roll-off direct switching with music could be different as not only the FR changes but the phase at higher frequencies also changes (assuming 6dB/octave passive low pass filter).
Never tested for this myself.
DAC reconstruction filters can be linear phase or minimum phase where the phase rotates within the audible range

INTERNET BLIND TEST: Linear vs. Minimum Phase Upsampling Filters

A blog for audiophiles about more objective topics. Measurements of audio gear. Reasonable, realistic, no snakeoil assessment of sound, and equipment.

archimago.blogspot.com

so for younger ears and the right stimulus (a specific instrument/passage for instance) there can be detectable differences depending on the used filter.
Josh tested at 96kHz which shifts the filter frequency upwards way above the considered audible range which was a good choice.

T.b.h. I am not surprised that that even well regarded and good performing/measuring DACs could possibly be told apart reliably in a well performed test with the right music/passage and training. This does not invalidate the fact that in general these similar performing DACs would not sound the same for the vast majority of users but could very well depend on the used filter type. And as Amir's measurements show even various linear phase filters are not the same.
Does this lead to obviously different 'house sounds' and tighter bass and whatnot ... no it won't but in a careful and well executed blind test where the goal is not to prove everything sounds the same one might be able to pull it off and it is possible Josh's test could be evidence.

For that test to be considered evidence more test data. measurements and 'recordings' would be required and unfortunately is missing and it seems like will not become available either (as Josh would be needed). We can learn more about the box fortunately (thanks to Erin and/or Amir, if he can get his hands on it).

Other than a higher than reported THD I do not expect much else but the distortion would be applied to both input signals in a closely similar way (depending on attenuation level I suspect)

 

[solderdude]

Severe lack of studies. There isn't even a single study cited for this in the audibility thresholds thread.

I don't think there is a severe lack of studies.
The fact that they aren't public (behind pay walls) or easy to find does not mean this hasn't been studied properly.
I have to admit that I am DAC-deaf (when comparing linear-phase DACs) and my hearing limits are way less than they were when I was young.
That doesn't mean that someone else can not detect differences I can't.

 

[donjoe0]

I don't think there is a severe lack of studies.
The fact that they aren't public (behind pay walls) or easy to find does not mean this hasn't been studied properly.

I can only speak based on what I've seen: nobody bothers to cite studies on this, so I see high chances there isn't much to cite. Therefore I have to talk about overconfidence based on slim to no foundations, and thereby shoot the ball back in their court. Extraordinary bullying requires extraordinary evidence. ^_^

 

[welwynnick]

How did Josh Do It?

5. He just published frequency response measurements showing the comparator outputs were very closely matched between channel A and channel B, between left and right, and at different comparator volume settings. At each volume setting, all the traces are on top of each-other.

This frequency response chart made me think. The HF drops about half a dB, and that matches Amir's test for the SMSL SU-1.
Maybe that's audible to a good listener, and he was hearing the difference in FR between each DAC?
Goldensound was able to hear the difference in FR between reconstruction filters in his ABX test, after all.
Josh said: "I selected several delta-sigma DACs praised for their measurements: the Matrix X-SABRE Pro MQA, RME ADI-2 DAC FS, Gustard X30, Topping D900, and the budget-friendly SMSL SU-1." so I looked at their Audio Precision FR charts, in the above order:

Some of them have flat responses, and some of them are slightly rolled-off at the top end. Not every DAC was ABX-compared with every other one. Many of the comparisons were against the Matrix, which was a sort of benchmark, and it has flat FR when the linear phase fast roll-off filter is used.

Some of the comparisons were against the Gustard R30 and Schiit Yggdrasil LIM. They're both R2R DACs, and measure very well, but quite different animals to the delta-sigma DACs.

This is starting to look like the differences between the DACs might just be greater than the differences between the comparator channels.

Maybe the different frequency responses were what Josh was hearing?

 

[donjoe0]

Maybe that's audible to a good listener, and he was hearing the difference in FR between each DAC?

Keep in mind, also, that the highest frequency proven to be audible to at least a few humans is 28 kHz (if presented at something like 110 dB SPL). We can't just go around assuming we know how high this one random individual's hearing can go, even at normal listening volumes. IIRC in the GoldenSound case he explored his limit and found it to be at least 22k.

 

[solderdude]

I can only speak based on what I've seen: nobody bothers to cite studies on this, so I see high chances there isn't much to cite. Therefore I have to talk about overconfidence based on slim to no foundations, and thereby shoot the ball back in their court. Extraordinary bullying requires extraordinary evidence. ^_^

There is a bunch of references on the wiki page

Hearing range - Wikipedia

en.wikipedia.org

 

[solderdude]

How did Josh Do It?

This frequency response chart made me think. The HF drops about half a dB, and that matches Amir's test for the SMSL SU-1.
Maybe that's audible to a good listener, and he was hearing the difference in FR between each DAC?
Goldensound was able to hear the difference in FR between reconstruction filters in his ABX test, after all.
Josh said: "I selected several delta-sigma DACs praised for their measurements: the Matrix X-SABRE Pro MQA, RME ADI-2 DAC FS, Gustard X30, Topping D900, and the budget-friendly SMSL SU-1." so I looked at their Audio Precision FR charts, in the above order:

View attachment 548769 View attachment 548776 View attachment 548772 View attachment 548773 View attachment 548774

Some of them have flat responses, and some of them are slightly rolled-off at the top end. Not every DAC was ABX-compared with every other one. Many of the comparisons were against the Matrix, which was a sort of benchmark, and it has flat FR when the linear phase fast roll-off filter is used.

Some of the comparisons were against the Gustard R30 and Schiit Yggdrasil LIM. They're both R2R DACs, and measure very well, but quite different animals to the delta-sigma DACs.

This is starting to look like the differences between the DACs might just be greater than the differences between the comparator channels.

Maybe the different frequency responses were what Josh was hearing?

One has to keep in mind that Josh tested with 96kHz files so filter response at 44,1kHz is not so relevant and any filter response differences would be well above the audible range even for a 6-year old.

 

[welwynnick]

One has to keep in mind that Josh tested with 96kHz files so filter response at 44,1kHz is not so relevant and any filter response differences would be well above the audible range even for a 6-year old.

Sure, but I don't think it's necessarily down to the filters (though it might be, for other reasons, like phase linearity).
In each case where there's an HF roll-off, it begins around 10kHz, rather than close to 20kHz.

 

[solderdude]

Phase response depends on the type of filter. Josh claims to have selected 'comparable' filters when possible.
Given the 96kHz files roll-off (other than the up-sampled RB file) would not roll-off below 20kHz (well it could slightly be for filterless R2R at 96kHz).

 

[welwynnick]

In this case I don't think it's necessarily down to the filters so much as the rest of the electronics and the tuning choices that each manufacturer has made.

It doesn't matter whether it's down to the filters, or whether it's down to something else. There are lots of possible causes, but they're all distractions.

What matters is that the frequency responses are different.

 

[Sokel]

I hate to be the party breaker, but the usual DMM suggestion assumes that all DMM's are precise and consistent, Keysight 6.5 digit-grade.

And we keep suggesting that any would do. Would it do though? For the precision we ask for?

Here's an example of one of mine, not dirt cheap, I would call it half-decent:

Extech EX520

See the accuracy?

I do use it just fine of course, but would it do for ASR-level scrutiny? Not by my side.

It's not that simple as it seems, and I see people using Amazon 50 euro stuff without even the slightest documentation.

 

[voodooless]

And we keep suggesting that any would do. Would it do though? For the precision we ask for?

See the accuracy?

Not the same thing!

 

[Sokel]

Not the same thing!
View attachment 548780

I'll find you a diyaudio, old scrutiny about consistency and precision as well.
Hint: it goes with the price