DAC audible differences are finally proven with ABX?

[rjp]

Using a multimeter and a test tone within the specified bandwidth is absolutely the right approach. Also, Josh used multiple methods according to what he wrote, including pink noise evaluated by REW. It is, however, not clear to me which method he used to determine the final judgement of "good enough" matching.

He also asked Amir for advice on the best way to do this in the post excerpt a few pages back. I believe the answer was DMM. (post 635)

 

[solderdude]

Zero references on FR flatness (defect) detection.

What do you mean by that ?

 

[donjoe0]

What do you mean by that ?

Peaks, dips, slopes, filter ringing hills and valleys, anything that's not a straight line from 12 Hz to 28 kHz. And we don't even need to have a reference device with impossibly flat FR do the required experiments, just introduce the various possible defects at smaller and smaller delta vs. some quality DAC chosen as a reference, regardless if that one was perfect to begin with. We need the delta detection ability before anything else. Only when we have that can we get to the actual work of designing the perfectly flat reference DAC that hopefully no one (fingers crossed) will ever be able to hear as anything other than flat.

 

[NTK]

Sure, as soon as I get the authors to approve my request for the full text. Thanks for not even bothering to say what that study proves in the context of the discussion, by the way, that's very helpful.

Not even bothered to read the abstract? Have you ever done any scientific study or research? You have the title of the paper and the authors, and you are clueless as to what to do next? Is this place a daycare and I am here to nanny some spoil brats?

The "gaping hole" is your (lack of) knowledge of the subject.

 

[Jazigo]

I just want to dedicate a post to @Josh83 and thank him for his thorough work into the testing. I can only imagine how frightening it must be to hold that position, while undergoing so much scrutiny from people with vast knowledge in audio.

 

[RandomEar]

Sure, as soon as I get the authors to approve my request for the full text. Thanks for not even bothering to say what that study proves in the context of the discussion, by the way, that's very helpful.

It appears that the paper is available on ASR as well as at AES. So no need to wait for the authors

 

[donjoe0]

It appears that the paper is available on ASR as well as at AES. So no need to wait for the authors

Yeah, I found it with Google a minute ago, I just don't normally use it to search anymore since they're such a toxic company.

But what I'm seeing in there is that depending on the program material, transducer type and how the resonance is presented (delayed or not, repeated or not), just-noticeable levels can go as low as -40(!) dB below the signal. If my math is still math'ing that should be much much lower of a delta in the combined spectrum than even the most optimistic 0.1 dB you're quoting from the various sources you put together in the other thread. -40 added to the main signal is, what, +0.000434 dB if the resonance source is uncorrelated to the main program? Or do we have to assume every DAC defect necessarily produces correlated components on top of the main signal? Then it would be around +0.086 dB, which seems just about where you found the best-performing listener using pink noise in ScottNorcross et al. (2003).

The thing about the ScottNorcross study is I think they simply turned the whole program up or down instead of introducing a coloration like the Toole & Olive study. That would mean for the latter result it's more likely the delta value is lower since a coloration or resonance artifact also introduces phase shift, so correlated-sources math shouldn't apply, even though we wouldn't necessarily get to the extreme of a complete uncorrelated-sources result, I'm guessing the real result is somewhere in between - lower than 0.086 but I don't know by how much.

In any case, if this is the up-to-date research, to my mind it already shows that DAC measurements not zooming in on the FR graph beyond 0.1 dB resolution are simply not good enough to prove "audible transparency" vs. all possible human listeners and listening conditions in the world. And that would be the majority of DAC measurements published to date.

 

[Geert]

@Mnyb @KSTR @Geert I have added some fresh info regarding level differences / just noticeable differences to my thread about the limits of human hearing vs measurements. Feel free to add more info there if you feel it's missing stuff.

I didn’t read all the details, but it seems to confirm my claim: "The study by Scott et al. thereby seems to suggest that humans might be able to distinguish level differences below 0.1 dB for white noise and possibly as good as 0.15 dB for music".

Note that I specifically metioned EQ-ing to 0,2dB, which means the rest of the track acts as a reference to the band being attenuated or amplified. I’ve always been convinced that this makes people more sensitive to the effect of or difference between filters versus sensitive for differences in full spectrum level (what most audiophiles focus on when discussing thresholds).

 

[RandomEar]

Yeah, I found it with Google a minute ago, I just don't normally use it to search anymore since they're such a toxic company.

But what I'm seeing in there is that depending on the program material, transducer type and how the resonance is presented (delayed or not, repeated or not), just-noticeable levels can go as low as -40(!) dB below the signal. If my math is still math'ing that should be much much lower of a delta in the combined spectrum than even the most optimistic 0.1 dB you're quoting from the various sources you put together in the other thread. -40 added to the main signal is, what, +0.000434 dB if the resonance source is uncorrelated to the main program? Or do we have to assume every DAC defect necessarily produces correlated components on top of the main signal? Then it would be around +0.086 dB, which seems just about where you found the best-performing listener using pink noise in Scott et al. (2003).

The thing about the Scott study is I think they simply turned the whole program up or down instead of introducing a coloration like the Toole & Olive study. That would mean for the latter result it's more likely the delta value is lower since a coloration or resonance artifact also introduces phase shift, so correlated-sources math shouldn't apply, even though we wouldn't necessarily get to the extreme of a complete uncorrelated-sources result, I'm guessing the real result is somewhere in between - lower than 0.086 but I don't know by how much.

In any case, if this is the up-to-date research, to my mind it already shows that DAC measurements not zooming in on the FR graph beyond 0.1 dB resolution are simply not good enough to prove "audible transparency" vs. all possible human listeners and listening conditions in the world. And that would be the majority of DAC measurements published to date.

The Toole paper isn't about global level differences, so it's not really related to the other stuff I recently posted. And I don't think you can calculate any audible level difference threshold from that in the way you did. I just wanted to link to it so you can gain access.

 

[solderdude]

Peaks, dips, slopes, filter ringing hills and valleys, anything that's not a straight line from 12 Hz to 28 kHz.

It does not happen with electronics in any meanigfull way within the audible band. This is much more of an issue with transducers and acoustics though.
Sure a DAC filter can have a small bandpass riple. Try to ABX that if you want it researched in depth.
A gradual roll-off below 20Hz and above 15kHz also is not much of an issue. If you think it is get some irrifutible proof first.

And we don't even need to have a reference device with impossibly flat FR do the required experiments, \

Exactly how flat do you think is required and which bandwidth and above all why and based on what information ?

just introduce the various possible defects at smaller and smaller delta vs. some quality DAC chosen as a reference, regardless if that one was perfect to begin with. We need the delta detection ability before anything else.

Delta Wave and nulling does exactly that. It can detect sub 0.1dB differences with ease.

Only when we have that can we get to the actual work of designing the perfectly flat reference DAC that hopefully no one (fingers crossed) will ever be able to hear as anything other than flat.

Again ... how flat is needed and based on what factual information ?

 

[Thomas_A]

A bit more digging, as already noted previously:

"The Matrix X-Sabre Pro family (same ESS9038PRO platform as the X-Sabre Pro MQA/HQA variants) was measured by Paul Miller in Hi-Fi News. He explicitly noted that the impulse response is negative-going, concluding that the DAC is phase inverting. In other words, a positive digital impulse emerges as a negative analog impulse.

That statement was made for the closely related Matrix Element X, which Matrix themselves described as an evolution of the X-Sabre Pro platform."

The Matrix Sabre Pro HQA was released in November 2018
The Matrix Element X was announced in September, available late 2018

Can anyone point to a direct measurement of the Matrix Sabre Pro HQA that shows its polarity? I have not been able to find one.

 

[donjoe0]

Exactly how flat do you think is required and which bandwidth and above all why and based on what information ?

Yes.
That is MY question, to the people claiming this is established science, meaning any claims of DAC artifact audibility should be treated as 'extraordinary'. I want to see that established science, with numbers. (And I have, courtesy of RandomEar, I'm just not sure about those numbers, which ones can be extrapolated and how far. I think there might be fewer than 100 total participants in all relevant studies put together. Sadly the Toole and Olive one doesn't even mention the number of participants they had.)

And 'why' is because FR is the most important metric for audio reproduction devices.

Delta Wave and nulling does exactly that. It can detect sub 0.1dB differences with ease.

Humans. I need humans to do that, in scientific studies. Lots and lots of humans, so we can find the limit of human audibility of subtle colorations. If that hasn't been done with at least... I don't know, thousands of humans across all available studies, I don't think there can be any talk of "established science" (on this specific ability!), and therefore no talk of "audibly transparent" DACs being ubiquitous or DACs being a "solved problem" holds water.

 

[amirm]

That is MY question, to the people claiming this is established science, meaning any claims of DAC artifact audibility should be treated as 'extraordinary'. I want to see that established science, with numbers.

You can start with these two peer reviewed references:

“Noise: Methods for Estimating Detectability and Threshold, ” Stuart, J. Robert, JAES Volume 42 Issue 3 pp. 124-140; March 1994
“Dynamic-Range Issues in the Modern Digital Audio Environment, ” Fielder, Louis D., JAES Volume 43 Issue 5 pp. 322-339; May 1995

I have a brief overview of them in my article here: https://www.audiosciencereview.com/forum/index.php?threads/dynamic-range-how-quiet-is-quiet.14/

The sequence of papers form Fielder establishes, based on human hearing threshold vs maximum SPL that we may desire, how low an impairment needs to be to be audible. Here is an example analysis:

My shorthand for that threshold is about -115 dB. State of the art DACs easily achieve this, assuming transparency of noise and by implication, distortion.

Above is based on single tones relative to noise. Detection thresholds increase when playing back music due to perceptual masking. So above is the most conservative rating to use. I have a youtube video on this topic as well.

We also have research that shows reviewers tend to be very poor at producing reliable data in controlled testing:

Combined, these research which also include and draw from listening tests, give us the foundation to highly doubt the results as published.

 

[amirm]

I don't know, thousands of humans across all available studies, I don't think there can be any talk of "established science" (on this specific ability!), and therefore no talk of "audibly transparent" DACs being ubiquitous or DACs being a "solved problem" holds water.

We haven't sent that many people to the moon as to establish, by your standards, that there are no aliens living there. Or that there is breathable air.

There are plenty of studies that have established threshold of hearing and so the data is massively considered established science. There is no value in repeating them to satisfy a layman no more than there is to keep sending rockets up to moon to see there is air or humans there.

Note that the answer I gave you is far more powerful than what you ask. Instead of relying one expensive and error prone blind tests, we use threshold of hearing. Once a device's impairments land below that, then we know, with extreme confidence that listeners won't be able to detect them. That is the case because the analysis is conservative and assumes such things as masking does not exist.

 

[amirm]

Adding on, no, we can't defend the often said "any DAC over X (80 dB or whatever) SINAD" is transparent. That is indeed a guess and can be challenged, and seemingly has in this testing, to show it is not true.

 

[solderdude]

Yes.
That is MY question, to the people claiming this is established science, meaning any claims of DAC artifact audibility should be treated as 'extraordinary'. I want to see that established science, with numbers. (And I have, courtesy of RandomEar, I'm just not sure about those numbers, which ones can be extrapolated and how far. I think there might be fewer than 100 total participants in all relevant studies put together. Sadly the Toole and Olive one doesn't even mention the number of participants they had.)

And 'why' is because FR is the most important metric for audio reproduction devices.


Humans. I need humans to do that, in scientific studies. Lots and lots of humans, so we can find the limit of human audibility of subtle colorations. If that hasn't been done with at least... I don't know, thousands of humans across all available studies, I don't think there can be any talk of "established science" (on this specific ability!), and therefore no talk of "audibly transparent" DACs being ubiquitous or DACs being a "solved problem" holds water.

What you want is pre-chewed data from for instance AES articles and scientific research on J.N.D. for audio applications. And not only that, you demand others to do the search for you and link to a huge amount of research so YOU can be convinced.

JND depends on SPL, frequency, Q-factor and amplitude of dip/peak as well as training of an individual and for perception also highly dependent on the transducer/acoustics.

 

[pma]

It should be added that finding the lowest level in a music test sample and amplifying it of say 40 dB to hear a “difference” makes the listening test invalid. The music is to be listened at such level that we can hear the whole sample without turning volume up and down. The other thing is audibility threshold plots and the real dynamic, usable dynamic in real music. We can easily create unnatural technical conditions. 60dB of usable dynamics is a rare exception in real music recording. This indicates to necessary parameters for “transparent” music reproduction. And then, 80 dB is about right. 40 dB room ambient noise + 80 dB makes 120 dB SPL.

 

[donjoe0]

You can start with these two peer reviewed references:

I don't see how either of those targeted specifically our ability to detect minute colorations within a device's frequency response. It's the only thing I'm out here contesting as 'established science'. Detection of colorations within a broad response, not moving the entire level up or down, faintest SPL detectable, none of that. Colorations only.

We haven't sent that many people to the moon

The moon has nothing to do with exploring the generally valid (or absolute) limits of the human organism. Testing lots of human organisms does. I don't think asking for results based on thousands of patients would be out of place in medical science. I say the same standard should be applied before recognizing anyone's right to talk in absolutist terms like "audibly transparent" and especially "you're an idiot for thinking you actually heard that defect".

What you want is pre-chewed data [...] you demand others to do the search for you and link to a huge amount of research

Yes, OR stop using absolutist terms like "audibly transparent" and stop bullying people who claim to hear DAC defects like they're lunatics denying proven scientific facts corroborated by millennia of experimentation. They should either show how it's 'established science' or stop behaving like it is. Or, you know, stop claiming their approach is scientific, there's always that third option.

 

[pablolie]

...
Humans. I need humans to do that, in scientific studies. Lots and lots of humans, so we can find the limit of human audibility of subtle colorations. If that hasn't been done with at least...

No suggestion of standardized listening protocols/equipment with even thousands of inputs makes the test useless.

There have been many tests with strict peer reviews made on the matter. I contend the limits of human hearing are extremely credibly established. And listening protocols take training... at a certain level, test results are meaningless without a lot of training and a type of dedication to the test most people will never bring to the table.

 

[pjug]

A bit more digging, as already noted previously:

"The Matrix X-Sabre Pro family (same ESS9038PRO platform as the X-Sabre Pro MQA/HQA variants) was measured by Paul Miller in Hi-Fi News. He explicitly noted that the impulse response is negative-going, concluding that the DAC is phase inverting. In other words, a positive digital impulse emerges as a negative analog impulse.

That statement was made for the closely related Matrix Element X, which Matrix themselves described as an evolution of the X-Sabre Pro platform."

The Matrix Sabre Pro HQA was released in November 2018
The Matrix Element X was announced in September, available late 2018

Can anyone point to a direct measurement of the Matrix Sabre Pro HQA that shows its polarity? I have not been able to find one.

Josh did address this in the update. I am not familiar with how polarity is determined from multitone but I assume there is something in the software that gives this.

The RME, Matrix, Gustard, SMSL, and Schiit DACs all maintain absolute polarity, according to both published measurements and my own Multitone measurements on my specific units. I did not adjust the polarity of any DACs with an inversion option in the settings. The ABX box’s A and B paths share the same polarity, so polarity can’t differ between the two sides of a comparison.

I can’t verify the D900’s filter or polarity, since I no longer own it. The D900 was measured by Audio Science Review, and inverted polarity wasn’t mentioned. However, polarity isn’t always addressed in reviews.