DAC audible differences are finally proven with ABX?

[JustAnandaDourEyedDude]

Based on these tests alone are we making the conclusion that, for a typical human listener, the output of this system is indistinguishable from its input for any audio signal?
This is the part I need help with.

As you agree, we can predict (from the threshold of audibility limits plot, the first figure that Amir posted in Post #773) that, signal-wise, the distortion artifacts in the D900's output voltage seen in the amplitude-versus-frequency plots (in Post #784) would not be audible to any human. The digital input of perfect sine wave/s is of course a stream of discrete rational numbers input at the sampling rate through the USB port of the DAC. The analog output of the DAC as shown in the plots is a well-nigh perfect sine wave or waves (pure tone or tones). Based on the threshold of audibility plot, for simplicity Amir picks -115 dB absolute as a flat threshold guaranteeing inaudibilty regardless of frequency. The distortion artifacts would lie below that threshold for any normal listening volume levels of music we would use.

To draw a conclusion for "any audio signal", we would need a multitone measurement (such as seen in Post #784), wherein the input tone frequencies span the audio bandwidth sufficiently well (and the instantaneous amplitude never exceeds 0dBFS). Rather, we would need to know the SINAD for the multitone test (which the AP analyzer can calculate), instead of just eyeballing the plot. If the SINAD turns out to be 115 or greater, then we know the deviation from the digital multitone signal should not be audible. This shows that the output voltage varies linearly with the input signal (very closely within the allowed range) and that the output frequencies track the input frequencies (very closely within the audio band). From this linearity, we infer that the mathematical equivalence of an allowable audio signal to a sum of sine waves (of differing frequencies and amplitudes and starting phases) is mimicked closely enough by the DAC, such that the deviations would be inaudible. Therefore, the output analog signal is predicted to be a well-nigh perfect representation corresponding to any digital input audio or music signal (volume-limited and within the bandwidth of the DAC). So the answer to your above question is yes, the conclusion is based on the electronic tests alone (with the -115 dB absolute criterion).

This is my layperson's understanding of how a good multitone test result may permit us to infer that for allowable audio signal inputs, the errors in the DAC's output would be inaudible to humans. Some expert ASR members would hopefully correct me if I have gotten it wrong, or if we need additional tests such as a jitter test. I assumed an output filter with a quick roll-off above 40 kHz sample rate.

I had previously thought that we made this conclusion based on the fact that blind listening tests correlate well with measurement thresholding, but it sounds like you may be saying the measurements alone are sufficient without the ABX backing. Am I following correctly?

There is no ABX backing going on in Amir's testing. He tests DACs by electronic measurements using the AP analyzer. The only blind listening tests involved are ones performed by other researchers, such as the original study by Fletcher and Munson, updated a couple of times later by other researchers. These tests are more accurate and involved versions of the sort of hearing test your ENT specialist might do to check your hearing, and I guess only pure tones are involved. These blind listening tests are correlated with measurement thresholding only insomuch as Amir got the -115 dB absolute inaudibility criterion off of the threshold of audibility curve in one such study.

Amir typically does a separate listening test of the DAC, with music samples he is familiar with (not pure sine waves), and using a known good amp and speakers; there is no quantitative measurement going on in this listening. This listening test is done just as a sanity check, to make sure he hears nothing odd that would warrant further investigation.

 

[krabapple]

That is not the borders of this test. Lack of controls is outside of vocabulary of many who say just about every DAC above SINAD of 80 sound the same. If someone says they do sound different, they are told to do level matched, double blind test. Well, the author did that and if that was our only ask, it is more than satisfied. We can't just now be whistling dixie and say all is well because he didn't use controls. We don't know if those controls show anything. We also have doubts about the comparator but again, we don't know if that has done anything. The net result is a strong, very strong counter to routine posts I see as I mentioned above.

The only counter we have, which is a strong one, is unwillingness of the author to let us examine his work. But he has now offered to sell me his comparator and asking how much I would pay. How much should we/I pay? Seeing how members are building a much better version, it has no long term value for me other than investigation of this issue.

For full rigor wouldn't you also have to obtain the DACs he used? (Not the particular units he owns -- just the same models, assuming unit-to-unit variance is not a factor).

We shouldn't whistle dixie and say all is well -- I don't think anyone is quite doing that? -- but a set of wildly outlying results begs for further investigation, rather than acceptance on their face. It's not unreasonable to hold that there was more likely than not an error somewhere. That still leaves room for being wrong -- for the results to be 'real'. If so, it's paradigm-shifting.

 

[HalSF]

Now reaching a level of cranky exhaustion and bouncing the rubble re arguments for second-guessing and invalidating the test. "Stop! He's already dead!" as the wise children of Springfield said.

For me, the whole thing doesn't matter even if the test were to be proved valid and repeatable in achieving Josh's suspiciously unprecedented level of successfully discerning audible differences between DACs. Who cares? The qualities of the allegedly audible differences themselves were not only poorly defined and inadequately described, but were admitted to be minor to the point of virtual irrelevance. For someone like me who's in this racket for only one reason, the love of beautiful high-fidelity music, that's game over right there in terms of mattering to me.

 

[krabapple]

I would not expect many scientific publications on this narrow issue (DAC vs. DAC) because in general, research is more focused on global parameters: Can a person hear distortion level X? Can they distinguish level difference Y? Can they hear noise level Z? If those thresholds have been established, you can simply measure any device and judge whether it surpasses them.

Exactly. 'Can people hear difference between DACs' falls under the broader category of the limits of human hearing in several dimensions (e.g. frequency bandwidth, level difference) -- and that is what tends to get explored academically for publication in professional journal, as those findings have implications beyond just DACs.

DAC differences are more likely to be touted by DAC manufacturers...yet they never (IME) provide anything like the DBT evidence Josh gathered.

Ditto amplifiers, cables, etc. If you have audibility data on their underlying functional mechanism -- how they work -- it's reasonable to infer whether they are likely to 'sound different'.

 

[krabapple]

A thousand negative outcomes will not counter a reliable positive one. Either the sound is the same for all people or it is not. That is the question that needs answering.

But that n=1 (where n= number of subjects) only answers what is humanly possible. If it's just one guy generating that one positive (or a positive set, like Josh) considered heretofore extremely unlikely, it says nothing about its applicability to a populace until you get a broader sampling.

By itself it's a pretty weak reason for Joe Golden Ear to assume/claim he hears it too, in his normal listnening.

Which he will do, in my vast experience of the online and audio press golden ear population. Thereby adding more smug noise to the discourse.

 

[krabapple]

That thread is so unfortunate, and sadly not about the test.

There are impossibilities around, mostly about the gear needed.
We suppose to make it easier, right? We ask people to perform tests like that all the time, conditions are almost like a mantra.

And we certainly ask them for more data if the results are on their face extraordinary.
Data such as, audio files for comparison.

We already do that. It's not some ad hoc add-on we concocted for this particular instance.

Now seriously, can anyone in the right mind believe that these conditions will be ever met? Did anyone take a look a the "ideal" comparator thread to see what it takes?
Any single objection there will very well stand in any test.

Proper ABX is already a marathon done right, adding a decathlon on top of it is not a good picture.

Couldn't care less! It is necessarily part of the picture when the results are extraordinary.
And yes, as even audible difference between just 2 DACs that ostensibly 'measure the same', would be.
DBT and level matching are a minimum requirement to rule out the most basic perceptual biases.
That's just tough noogies.

Is Josh's refusal to back up his fantastically extraordinary results a 'good picture'?

FWIW, I doubt the ABX box is the culprit, but it's sure worth checking

 

[krabapple]

Yes.
That is MY question, to the people claiming this is established science, meaning any claims of DAC artifact audibility should be treated as 'extraordinary'. I want to see that established science, with numbers. (And I have, courtesy of RandomEar, I'm just not sure about those numbers, which ones can be extrapolated and how far. I think there might be fewer than 100 total participants in all relevant studies put together. Sadly the Toole and Olive one doesn't even mention the number of participants they had.)

'Audio science' doesn't claim 'it's impossible that anyone could hear a 0.1 dB difference, ever, under any circumstances'.

That isn't how scientific claims about such matters work.
They are expressed as probabilities, either with numbers or simply using ...words...like 'likely' or 'improbable'. Or reporting the conditions under which the thing becomes more or less likely audible.

On that basis, if you make an extraordinary claim for your own hearing abilities for something considered improbable (but not impossible) you will certainly and legitimately be asked to provide extraordinarily solid evidence that random old you are indeed part of the minority of the population residing in the word 'improbable'.

 

[amirm]

For full rigor wouldn't you also have to obtain the DACs he used? (Not the particular units he owns -- just the same models, assuming unit-to-unit variance is not a factor).

Yes and I have some of them and acquire the rest if needed.

 

[amirm]

We shouldn't whistle dixie and say all is well -- I don't think anyone is quite doing that? -- but a set of wildly outlying results begs for further investigation, rather than acceptance on their face.

Well, folks say to ignore it instead of investigating. None of us have accepted the results but we need hard evidence as to why we are not accepting it.

 

[amirm]

By itself it's a pretty weak reason for Joe Golden Ear to assume/claim he hears it too, in his normal listnening.

But Joe can say that measurements didn't predict what author heard. Ergo, dismiss all measurements and objectivity that goes with it. That there is some magic in the sound that is not seen by any instrument but by ears alone.

There is some good news as it dismissed the claim that such blind tests are so hard as to hide real differences. So those people need to be careful where they take the argument because we can ask them to do the testing blind.

 

[krabapple]

I'm gonna guess that cites for parameters of human hearing may be found in this classic text of psychoacoustics?
In, say, oh, I dunno, maybe chapter 7?
donjoe0, get our your wallet!

It's 20 years old though, so who knows, it might be completely obsolete

Amazon.com: Psychoacoustics: Facts and Models (Springer Series in Information Sciences): 9783540231592: Fastl, Hugo, Zwicker, Eberhard: Books

Amazon.com: Psychoacoustics: Facts and Models (Springer Series in Information Sciences): 9783540231592: Fastl, Hugo, Zwicker, Eberhard: Books

www.amazon.com

 

[gzost]

I am interested in the investigation into Josh's claims purely to learn more about ABX testing and the many ways in which tests can be compromised.

There is no possible outcome that will change my attitude towards DACs, i.e. they should be considered a solved engineering problem for all practical purposes, there is no need to spend big and I don't worry about any of the ones I use. Measurements were useful to check before purchasing that I don't get something that is so badly broken as to actually interfere with what matters: enjoying the music.

 

[Sokel]

And we certainly ask them for more data if the results are on their face extraordinary.
Data such as, audio files for comparison.

We already do that. It's not some ad hoc add-on we concocted for this particular instance.




Couldn't care less! It is necessarily part of the picture when the results are extraordinary.
And yes, as even audible difference between just 2 DACs that ostensibly 'measure the same', would be.
DBT and level matching are a minimum requirement to rule out the most basic perceptual biases.
That's just tough noogies.

Is Josh's refusal to back up his fantastically extraordinary results a 'good picture'?

FWIW, I doubt the ABX box is the culprit, but it's sure worth checking

Extraordinary results best way to debunk, is to repeat the task when conditions are known.

And there are the impossibilities, unless someone is invited at the author's bench, where these conditions are met.and I mean avery single one of them except the last variable, him.

The main problem here is the lack of standard procedure and gear. And I mean standard (as you read down the thread, not any multimeter would do for me as in other cases people wouldn't accept results other than from an AP for other results, etc) .

Been at the far-right side of measurements camp like me (for the shake of measurements alone, not SQ, just for the shake of good and not good-enough, like any other hobby) , I can think of and invent any hard condition so to make my case. I can turn it into a Sisyphean task, easy.

It's certainly easier than trying to replicate, right?
Wouldn't it be better with some standardize gear and conditions carved in rock? Otherwise each "camp" is just chasing the other "camp's" tail, it's not even about audio, it's about a classic Internet "who's right" thing, no different than any other rant.

 

[Thomas_A]

Yes and I have some of them and acquire the rest if needed.

If you can get hold of the specific Matrix MQA model (again, as you've reviewed it already...) and check both the XLR and RCA outputs for any odd behaviour like inverting polarity, that should finally close that discussion. (It is a bit interesting that the later Sabre 3 models changed the output stage with additional op-amps, as to fix something, and since Hifi News reviews have shown both inverted and non-inverted polarity of their models.)

 

[rjp]

Yes we can predict (from the threshold of audibility limits plot, the first figure that Amir posted in Post # 773) that, signal-wise, the distortion artifacts seen in the the output voltage seen in the amplitude-versus-frequency plots would not be audible to any human. The digital input to the DAC is not shown; the input sine wave/s is a stream of numbers input through the USB port of the DAC. The analog output of the DAC as shown in the plots is a well-nigh perfect sine wave or waves. The DUT (device-under-test) is a single DAC, not an entire audio system. The output signal would need to be fed into an amplifier and thence to a speaker or headphone or IEM in order to be audible. And yes, based on these measurements, by the linearity of the DAC, the output analog signal is predicted to be a well-nigh perfect representation corresponding to any digital input audio or music signal (within the bandwidth of the DAC).


I think there is some misunderstanding here. The test in Amir's plot was of a single DAC (the D900), so there is nothing blind about a listening test. Amir typically does a separate listening test of the DAC, with music samples he is familiar with (not pure sine waves), and using a known good amp and speakers. This listening test is done just as a sanity check, to make sure he hears nothing odd that would warrant further investigation. There is no double-blind ABX going on when Amir normally tests a DAC, because he does not normally do listening comparisons between DACs. You are correct that we are saying that the measurements alone are sufficient to predict the output signal quality without needing any ABX listening comparison with a reference DAC.

If the electronic testing (using the AP55X tester) for any particular DAC were to show artifacts in the output signal that should easily be audible, then one can next listen for such artifacts just to confirm that one hears anomalies. However, no one has yet constructed a comprehensive dictionary correlating types of spurious artifacts in the electronic measurements with types of audible anomalies when listening to appropriately revealing music using the same DUT. ABX listening comparisons are horrendously time-consuming to do, and impractical when Amir is testing so much audio equipment of many varieties, and they would not be particularly informative.

Josh's study is different. He picked DACs that measured so well with such electronic measurements that we would expect to hear no audible differences between such DACs in listening to identical music samples using the same amp and IEMs. However, Josh reported that in his double-blind ABX tests using the comparator to switch between DACs, he was able to reliably distinguish between such well-measuring DACs.

I appreciate you taking the time to reply but for the most part I already know these points and agree with you. What I am asking about is something different. It’s the inference step. I tried to make it clear in my post. I don’t want to clutter up the thread, but feel free to pm if you want to discuss further.

 

[NTTY]

And it only confirms that incompetent approach and lack of knowledge of the author.

I concur.

 

[amirm]

If you can get hold of the specific Matrix MQA model (again, as you've reviewed it already...) and check both the XLR and RCA outputs for any odd behaviour like inverting polarity, that should finally close that discussion.

Is this the one? https://www.audiosciencereview.com/...bre-pro-mqa-best-audio-dac-in-the-world.8729/

Is so, it should be here somewhere. Confirm and I will dig it up to measure.

 

[Thomas_A]

Is this the one? https://www.audiosciencereview.com/...bre-pro-mqa-best-audio-dac-in-the-world.8729/

Is so, it should be here somewhere. Confirm and I will dig it up to measure.

Yes!

 

[JustAnandaDourEyedDude]

I appreciate you taking the time to reply but for the most part I already know these points and agree with you. What I am asking about is something different. It’s the inference step. I tried to make it clear in my post. I don’t want to clutter up the thread, but feel free to pm if you want to discuss further.

I apologize for my earlier misunderstanding of your questions; I was confused about the reference to blind ABX listening tests. I have revised my previous post to reflect my (hopefully) better revised understanding of your questions. Hope the revision better answers them.

 

[rjp]

I apologize for my earlier misunderstanding of your questions; I was confused about the reference to blind ABX listening tests. I have revised my previous post to reflect my (hopefully) better revised understanding of your questions. Hope the revision better answers them.

Yes! Your revised post answers my questions very well. In fact, this is by far the best technical answer I have ever heard on this topic Thank you!

Your answer seems to be a solid technical explanation for how threshold-based parameter testing can indeed imply audible transparency, even without the support of ABX testing.

I suspected the multi-tone case was key, and now I think you have convinced me it is. This makes so much more sense now. If the system (in this case just a dac of course) can pass a composite test signal of 32 tones with final THD and noise below the threshold of human hearing, then it seems reasonable that we could further increase the number of tones without causing the test to fail. This may actually be provable. Since we know from Fourier theory that all bandlimited input signals can be represented as the superposition of (complex) scaled tones, this suggests any such signal passed by this system should be indistinguishable from the input. i.e., the system is transparent.

I am very glad you answered my question tonight @JustAnandaDourEyedDude .