DAC audible differences are finally proven with ABX?

[Plcamp]

I like seeing the pictures. What strikes me most is the routing of four backpanel connectors directly between relay coils and contacts.

I would like to hear from the manufacturer about what they have done to limit induced noise here (and overall). I would guess that best practice wouldn’t route that way, and would use coax wire for shielding?

 

[pma]

I would like to hear from the manufacturer about what they have done to limit induced noise here (and overall). I would guess that best practice wouldn’t route that way, and would use coax wire for shielding?

I am sorry to be quite sincere, but the layout and wiring is poor. How much effect it has would be seen from the independent test of the unit.

 

[Blake Klondike]

this reminds me of discussions i was part of when i was younger: people saying that they knew that angels were real because of the way they *felt* when they thought about it. or another where the person said she knows that she has a soul because it *feels* like she has a soul. the next statement is always that "you can't tell me i don't have a soul because you don't know what is true for me internally." as though someone's feelings have any bearing on whether or not something is objectively true.

 

[AndyRoo]

For me this thread is done.

If Josh won’t allow verification of his results then he’s either shooting himself in the foot or protecting himself from having a flaw in his methodology exposed.

I’m not saying he’s aware of any such flaw, but having published his results it would be somewhat embarrassing if scrutiny revealed one or more errors in his tests.

If you publish publicly then you need to be ready and able to provide the evidence to support your findings.

 

[HalSF]

Assuming the test was properly conducted and the results are valid, how can we dismiss them as trivial and insignificant?
We thought DACs that measure well were audibly indistinguishable. If I turns out they aren't, this to me, seems to be pretty big news

"We thought DACs that measure well were audibly indistinguishable," like "all amps sound identical," is a common ASR rhetorical mantra, not an audio science absolute. Those arguably overgeneralizing statements are deployed in opposition to all the woo-woo, adjective-crazed, taste-testing descriptions subjectivists offer in reporting sighted testing impressions. Meanwhile, there are all kinds of reasons DACs and amps might sound *very slightly noticeably different* than other similar components in an ABX test intent on hearing exactly those tiny differences and nothing else.

To me it's *not* pretty big news if DACs can be identified as distinguishable from other DACs in a highly focused, narrowly targeted test about the mere fact of distinguishability. (I'm not addressing whether or not the test was robust and valid, just what it purported to do.) The author of this ABX report was at pains to minimize the differences he says he detected, and was pretty nebulous about what those differences sounded like. His report has almost nothing to do with overconfident and dubious subjective reporting about the sound quality of DACs (or amps). It's not a big deal if DACs display itty bitty non-fundamental differences in sound. So what? That's the unanswered question.

 

[amirm]

To me it's *not* pretty big news if DACs can be identified as distinguishable from other DACs in a highly focused, narrowly targeted test about the mere fact of distinguishability.

I would have been more or less cool about such an outcome. What causes doubt for me is extremely high success rate. As I have said, I have taken part in many next to impossible ABX tests. The level of concentration and focus required is extreme, causing occasional fails, to say nothing of the training required. To keep getting perfect or near perfect results is just not in the cards unless the difference is large and significant. Or there is a "tell" indicating which is which in ABX tests.

So possible results but not to this extent.

 

[KSTR]

Thanks for sharing.
Now we know how the attenuation is done, using AD5262 digital pots. And we see loud monster relays, not really ideal for line-level switching. No coupling caps with pre-charging in the inputs. No GND side switching (which would ideally require active diff inputs). Sloppy wiring. And then we have those weird signal-combining RCA jacks with 10k(!) isolation resistors which I assume were used by Josh to combine the A and B outputs to feed his HP amp.

==> Doesn't look like a professional design to me. OK, it is over 10 years old and likely was never intended to be a top-tier device for line level ABXing, but still...

So far my conclusions would be
a) The ABX box itself likely has issues.
b) The setup in general maybe had issues (like polarity, etc).
c) The procedure certainly had issues (no controls used, etc).
And because no recordings were made, we'll never know.

 

[MAB]

Hey, folks. Been a while. I promised myself years ago no more forum posts because I always wind up spending way too much time posting and not enough getting other things done. But I think nearly 4 years is enough self-timeout time. Hopefully I can remember how to use the bbcode correctly.

Amir emailed me about this thread and asked about borrowing my ABX comparator to review/measure. We had a bit of discussion via email about this. I did give him my permission to provide my responses here but think it's worthwhile to share them myself because - as a user of this unit - I can provide my own experience.

With that said, get your comfy pillow under your butt and get ready for a long post...

Email 1:




Email 2:




I think that covers it.

Thanks,
Erin


View attachment 547500View attachment 547501View attachment 547502View attachment 547503View attachment 547504View attachment 547505View attachment 547506View attachment 547507View attachment 547508View attachment 547509


View attachment 547510

Thanks Erin. This is interesting, and not exactly surprising.

Experimental setup is so important.
Results near 100%, based on tells like soundstage, not believable.
What is believable is the switching box generates the tell.

Perhaps we never get to the bottom of this. I never figured out the sound of my old my Vandersteens until you measured them explicitly - an example of one of life's solvable mysteries if someone has the right experimental setup.

Good to hear from you.

edit: for clarity

 

[amirm]

Now we know how the attenuation is done, using AD5262 digital pots.

Very good eye!

From the specs for that part: "Low resolution DAC replacement."

Yes, we are running the output of high performance DAC through a low performance one to adjust its level!

Author says that the effect is constant on both inputs. Not so unless the level of attenuation is the same. Which was not the case or else it would not be needed!

We are talking this level of non-linearity here:

This is just not the intended use, i.e. test very high high performance sources. It is made to compare amplifiers and such.

 

[amirm]

FYI, I had a similar experience while testing audio codecs decades ago at Microsoft. I realized that depending on where I set the volume control in the media player, the sound would get brighter or be normal. Well, turned out the implementation of volume control in XP was extremely poor, with no dither, etc. This was the motivation to re-write the whole audio subsystem as led by my chief architect, JJ.

The effects that the author is describing then to some extent can be explained by the level change distortions.

 

[pjug]

Very good eye!

From the specs for that part: "Low resolution DAC replacement."
View attachment 547520

Yes, we are running the output of high performance DAC through a low performance one to adjust its level!

Author says that the effect is constant on both inputs. Not so unless the level of attenuation is the same. Which was not the case or else it would not be needed!

We are talking this level of non-linearity here:

View attachment 547519

This is just not the intended use, i.e. test very high high performance sources. It is made to compare amplifiers and such.

I am not following how the linearity of the digital pot is a problem, as long as the setting used gets to 0.1dB matching and the resistance is stable. On the other hand, I can see noise getting into the measurement there if there isn't good power supply filtering. That sort of thing.

 

[krabapple]

For me this thread is done.

If Josh won’t allow verification of his results then he’s either shooting himself in the foot or protecting himself from having a flaw in his methodology exposed.

Or, he just doesn't care enough, as he now knows what time/work investment was involved to get what he already got.

People who've done something can rationally decide that they have other things to do with their lives/time, than do it again.

Someone else with the time and will could try to replicate, of course. Until then it's a dubious outlier.

"We thought DACs that measure well were audibly indistinguishable," like "all amps sound identical," is a common ASR rhetorical mantra,

Bullsh*t. This tells me you either are too lazy to read accurately, or you intend to misrepresent.

Meanwhile, there are all kinds of reasons DACs and amps might sound *very slightly noticeably different* than other similar components in an ABX test intent on hearing exactly those tiny differences and nothing else.

To me it's *not* pretty big news if DACs can be identified as distinguishable from other DACs in a highly focused, narrowly targeted test about the mere fact of distinguishability.

Results like Josh's are quite extraordinary, and would be considered so 'in science', whether that's news to you or not.

 

[voodooless]

Even I know this is not how you layout the decoupling caps.

 

[NTTY]

I try to stay away from such a thread, not to suffer from the Brandolini effect. However, I feel too uncomfortable, going through some messages. There are many obvious flaws from the testing, and seeing that the community, in search of and need for understanding, all in goodwill, is pushed back with near random reasons, or worse, being criticized for what they’ve not said (ie being blind to others’ opinions), is disappointing to say the least. So, here are my 2 cents.

Since this publication defeats tons of similarly well-documented research (including here on ASR), it inevitably calls for evidence (there’s none provided) so we can all learn something new, and I was very surprised that Josh took his own findings for granted, concluding with an obvious definitive “all DACs sound different”.

Anyways, the asks for (much) more details have been rejected by Josh, for the reason of “I need to turn to other work now”. In DACs alone, this test represents a $15k investment, with many DACs that a lot of members only dream of owning. So, when people try to understand, do they really deserve a “Sorry I have other stuff to attend”? I don’t think this is fair.

We are left with not much but many unanswered and valid questions/requests:

• Proposition to review the ABX device: Rejected for the reason the device is fragile and rare. Well, I received devices for review similarly rare, more fragile, and all went well, every time.
• Proposition to record the output via the high precisions Cosmos ADC that Josh owns: Rejected for the reason of “no time”.
• Proposition to perform more measurements of the ABX device via the same Cosmos ADC: no reply
• Proposition to perform ABX test from the output of one DAC with a cable splitter: no reply
• Etc…

The few more information we got, and calling for more questions, are:

• Josh measured a SNR (how? weighted?) of 105dB, meaning that the ABX device has 10 times more noise than some of the DACs reviewed. What’s the nature of that noise, random, correlated?
• Josh measured IMD (which one?) of -108dB and TD+N of -112dB. How can the TD+N be better than the SNR, or the noise said to be at -108dB?
• We’ve been told that the box “does add some distortion, but that it's uniform across the inputs”. What does that mean? Do we get also 10 times more noise from the box than from the DACs?

Maybe Josh has “no time”, but says he’s still using the box “regularly”. I’d be him, I’d be eager to ensure I’m not running my tests in a vacuum. Obviously I am not Josh.

Provided this test really happened, since we have 0 proof of it, not even a picture of all DACs together which would have made a nice family photo, and actually we’ve not even been granted a picture of the inside of the box, and asked to look it up for ourselves on the web…, there are growing and tempting reasons to doubt all of it, honestly.

And sorry, but Josh saying:
“My intention wasn’t to score points for either camp or to make sweeping claims. I hope that came through.”

Contradicts:
“To be clear, I’m sure there are some DACs that sound the same, and if I had enough DACs on hand, I’d run into that result.”

The unwritten but implicit conclusion is “All DACs sound different and it can be heard with 100% accuracy” even using a device that masks all their tiny differences.

That’s a bold statement, and what appears to be the initial intention is a planned conclusion of the test before going for it. Not only the lack of proper documentation is an issue, but the absence of support to the community to understand and learn (repeat) from this experience is the elephant in the room.

Trying to stay positive and constructive, besides what many have shared, when it comes to mandatory complementary tests, I would add the below, which should have been included from the beginning:

• SINAD, SNR and DR measurements from each pair of DACs at the output of the box, as attenuated for the necessary level matching, both DACs driven. That is because the box has so much noise and distortion that lowering the gain for one DAC inevitably provides the other DAC with an advantage (it has more DR and distortion free-range). But in what range?
• Crosstalk between inputs, especially above 5kHz. Since the two DACs are running in parallel, there’s probably disruption from one input to the other. Some analog devices are bad at this, and some ears are sensible to high frequency distortion.
• Intermodulation crosstalk, for the same reason as above.
• IMD, Multitone and THD(+N) vs level are, for the same reasons, absolutely necessary to highlight the differences and potentially rule out differences from the box when being driven by different voltage input.

That’s a basis for more. Next obvious step is to run a comparison test “a la” Archimago, and the community would have been willing to help. Start with two DACs. How long does it take to record the output of the audio from the comparison box via the Cosmos ADC? Maybe one hour, if the box is regularly used, is that really impossible?

To me, I’m sad to see this review is no more than anybody claiming they can easily make a difference between two OP amps.

 

[Salt]

I try to stay away from such a thread, not to suffer from the Brandolini effect. However, I feel too uncomfortable, going through some messages. There are many obvious flaws from the testing, and seeing that the community, in search of and need for understanding, all in goodwill, is pushed back with near random reasons, or worse, being criticized for what they’ve not said (ie being blind to others’ opinions), is disappointing to say the least. So, here are my 2 cents.

Since this publication defeats tons of similarly well-documented research (including here on ASR), it inevitably calls for evidence (there’s none provided) so we can all learn something new, and I was very surprised that Josh took his own findings for granted, concluding with an obvious definitive “all DACs sound different”.

Anyways, the asks for (much) more details have been rejected by Josh, for the reason of “I need to turn to other work now”. In DACs alone, this test represents a $15k investment, with many DACs that a lot of members only dream of owning. So, when people try to understand, do they really deserve a “Sorry I have other stuff to attend”? I don’t think this is fair.

We are left with not much but many unanswered and valid questions/requests:

- Proposition to review the ABX device: Rejected for the reason the device is fragile and rare. Well, I received devices for review similarly rare, more fragile, and all went well, every time.

- Proposition to record the output via the high precisions Cosmos ADC that Josh owns: Rejected for the reason of “no time”.

- Proposition to perform more measurements of the ABX device via the same Cosmos ADC: no reply

- Proposition to perform ABX test from the output of one DAC with a cable splitter: no reply

- Etc…

The few more information we got, and calling for more questions, are:

- Josh measured a SNR (how? weighted?) of 105dB, meaning that the ABX device has 10 times more noise than some of the DACs reviewed. What’s the nature of that noise, random, correlated?

- Josh measured IMD (which one?) of -108dB and TD+N of -112dB. How can the TD+N be better than the SNR, or the noise said to be at -108dB?

- We’ve been told that the box “does add some distortion, but that it's uniform across the inputs”. What does that mean? Do we get also 10 times more noise from the box than from the DACs?

Maybe Josh has “no time”, but says he’s still using the box “regularly”. I’d be him, I’d be eager to ensure I’m not running my tests in a vacuum. Obviously I am not Josh.

Provided this test really happened, since we have 0 proof of it, not even a picture of all DACs together which would have made a nice family photo, and actually we’ve not even been granted a picture of the inside of the box, and asked to look it up for ourselves on the web…, there are growing and tempting reasons to doubt all of it, honestly.

And sorry, but Josh saying:

“My intention wasn’t to score points for either camp or to make sweeping claims. I hope that came through.”

Contradicts:

“To be clear, I’m sure there are some DACs that sound the same, and if I had enough DACs on hand, I’d run into that result.”

The unwritten but implicit conclusion is “All DACs sound different and it can be heard with 100% accuracy” even using a device that masks all their tiny differences.

That’s a bold statement, and what appears to be the initial intention is a planned conclusion of the test before going for it. Not only the lack of proper documentation is an issue, but the absence of support to the community to understand and learn (repeat) from this experience is the elephant in the room.

Trying to stay positive and constructive, besides what many have shared, when it comes to mandatory complementary tests, I would add the below, which should have been included from the beginning:

- SINAD, SNR and DR measurements from each pair of DACs at the output of the box, as attenuated for the necessary level matching, both DACs driven. That is because the box has so much noise and distortion that lowering the gain for one DAC inevitably provides the other DAC with an advantage (it has more DR and distortion free-range). But in what range?

- Crosstalk between inputs, especially above 5kHz. Since the two DACs are running in parallel, there’s probably disruption from one input to the other. Some analog devices are bad at this, and some ears are sensible to high frequency distortion.

- Intermodulation crosstalk, for the same reason as above.

- IMD, Multitone and THD(+N) vs level are, for the same reasons, absolutely necessary to highlight the differences and potentially rule out differences from the box when being driven by different voltage input.

That’s a basis for more. Next obvious step is to run a comparison test “a la” Archimago, and the community would have been willing to help. Start with two DACs. How long does it take to record the output of the audio from the comparison box via the Cosmos ADC? Maybe one hour, if the box is regularly used, is that really impossible?

To me, I’m sad to see this review is no more than anybody claiming they can easily make a difference between two OP amps.

Totally agree to Your point of view. What brings up my couriosity is the motivation to do that kind of test and publishing it. We know, samples for such tests cost a lot of money, as well as that 'miracle-AB-box', so there is some investment to begin with. If this was really done, I would have expected some participation into the discussion here. But: no.
And no (as suggested and alway presented in Your reviews) images of DUT, ABX-box, no files ... nothing.

Fake?

 

[KSTR]

Even I know this is not how you layout the decoupling caps.

Decoupling is not that relevant here as the chip is static CMOS and operated in a non-switching state (except for the moment when attenuation is changed).

We are talking this level of non-linearity here:

That is the nonlinearity (==precision) of the resistance steps in rheostat mode (thus with wiper influence), not of the signal passing through in potentiometric mode (EDIT: actually we don't know which mode is used, it could well be rheostat mode with a fixed upper resistor, looking at the layout). AD states 0.014% for the 2kR type with 1Vrms and output unloaded which is realistic for a string of all equal on-silicon resistors. With loaded output things change quite a bit, attenuation will become less predictible and the distortion will go up.
Josh did calibrate levels correctly with load (HP amp input) attached and measuring at the HP output, so that variable was under control (assuming that L/R mathcing was also checked/established, besided the A/B matching).

 

[voodooless]

Decoupling is not that relevant here as the chip is static CMOS and operated in a non-switching state (except for t moment when attenuation is changed).

And yet, this is not how you would have done it..

 

[KSTR]

And yet, this is not how you would have done it..

Yep, certainly not ;-)

 

[Mort]

Hey, folks. Been a while. I promised myself years ago no more forum posts because I always wind up spending way too much time posting and not enough getting other things done. But I think nearly 4 years is enough self-timeout time. Hopefully I can remember how to use the bbcode correctly.

Amir emailed me about this thread and asked about borrowing my ABX comparator to review/measure. We had a bit of discussion via email about this. I did give him my permission to provide my responses here but think it's worthwhile to share them myself because - as a user of this unit - I can provide my own experience.

With that said, get your comfy pillow under your butt and get ready for a long post...

Email 1:




Email 2:




I think that covers it.

Thanks,
Erin


View attachment 547500View attachment 547501View attachment 547502View attachment 547503View attachment 547504View attachment 547505View attachment 547506View attachment 547507View attachment 547508View attachment 547509


View attachment 547510

Thanks for that summary of your experience with electronics. Very informative.

 

[mhardy6647]

He published the results. That alone means he felt they were valid and thus invites scrutiny.

That’s not a criticism, either.

But subsequently resisting the inevitable scrutiny is not a good way to be taken seriously by a community that takes evidence and proof extremely seriously.

I reckon many if not most of us have been through the peer review process -- likely as both reviewer and 'reviewee'.