DAC audible differences are finally proven with ABX?

[mhardy6647]

What causes doubt for me is extremely high success rate.

Yup.

 

[mhardy6647]

Obviously I am not Josh.

You're not joshing.
Sorry -- couldn't resist. We could use a little levity.

 

[pjug]

I wonder how many of the people accepting the story without questioning have tried an ABX to know how hard it is to get 100% even when you are clued in and there are small but definite audible differences.

 

[pablolie]

I reckon many if not most of us have been through the peer review process -- likely as both reviewer and 'reviewee'.

I'd be useless in my job if I didn't take feedback (which is a positive word for "criticism") seriously. And you can't have an ego. Technology teams only move forward with consensus. To get there, one MUST accept how the orginal premise can be improved. My colleagues don't want to tell me I am WRONG, they just want to make my work BETTER. And being taken seriously by experts (which they show when they bother to review my stuff) is a COMPLIMENT, not scathing negative criticism (because if you get that, it does mean you better take time in learning the subject matter better - which is also helpful).

There's absolutely nothing negative or inpolite or pigheaded in NOT accepting everything that's written/suggested/proposed. Like many have said, it's the ONLY way to arrive at a result that is defensible. In fact, it's having an ego that stands in the way of constant self-improvement that is pig-headed.

 

[pablolie]

I wonder how many of the people accepting the story without questioning have tried an ABX to know how hard it is to get 100% even when you are clued in and there are small but definite audible differences.

So true. In fact, the last time I tried to establish audible differences between very competent audio electronics (Benchmark AHB2 vs NAD M22) I spent way too many hours bitterly trying to establish which one I should keep. After HOURS I could tell which one I was listening to about 75% of the time, but I could not establish which I liked better or why. The differences vanished when I just listened to enjoy music. I shall never ever do that again. Just like trying to establish dfferences beween 320k OggVorbis and FLAC (also happy with an 80% success rate there), I'll never waste my time doing that again, either.

And then here... 6 different DACs. >97% success rate? Nah. It's laughably one-sided, sorry. It'd be a LOT more credible if the comparisons between at least two of the DACs fell pretty much flat. Then I'd try to listen...

"High-End" DAC Blind Listening Results - PART II: Results & Analyses

A blog for audiophiles about more objective topics. Measurements of audio gear. Reasonable, realistic, no snakeoil assessment of sound, and equipment.

archimago.blogspot.com

 

[ahofer]

I’m baffled that people here are treating this test as some kind of sinister agit-prop triumph for subjectivists that challenges the integrity of the audio science world and needs to be mobilized against.

The minor differences the tests claims to have successfully discerned and identified, insofar as the test write-up makes clear what they were (and it doesn’t make it very clear *at all*), are seemingly so trivial and marginal that the significance of the test, even if was robust and valid, remains obscured inside a dense fog bank of uncertainty and extremely indeterminate import.

The author has some further explaining to do, and has promised he will.

I didn’t see a lot of the “sinister agit-prop” stuff, I think mostly people are engaging and asking questions because this is both a good faith effort AND goes against a mountain of contrary evidence. There hasn’t been any ridicule.

Amir’s record on the Archimago DAC challenge is new to me and worth considering as well.

 

[kemmler3D]

Totally agree to Your point of view. What brings up my couriosity is the motivation to do that kind of test and publishing it. We know, samples for such tests cost a lot of money, as well as that 'miracle-AB-box', so there is some investment to begin with. If this was really done, I would have expected some participation into the discussion here. But: no.
And no (as suggested and alway presented in Your reviews) images of DUT, ABX-box, no files ... nothing.

Fake?

I'm reluctant to start accusing or anything, because I think that lowers our chances of ever getting corroboration from Josh, and we don't have anything that positively suggests fakery per se.

I would also say... if you were going to fake this kind of result, you'd think the correct % would actually be lower, to make it more believable.

I think we are left with "truly extraordinary hearing" or "flawed test setup / procedure" as likely alternative explanations.

But I also think we can't say "fake" is ruled out at this point.

And, again, for anyone reading this and thinking ASR folks are negative nancies who don't want there to be differences in DACs... my favorite possible outcome is we find an un- or under-documented mode of distortion in DACs and fix it in future DACs. That would ultimately be the point of all this, right?

 

[krabapple]

I wonder how many of the people accepting the story without questioning have tried an ABX to know how hard it is to get 100% even when you are clued in and there are small but definite audible differences.

This is where positive controls would have come in handy.

 

[krabapple]

And, again, for anyone reading this and thinking ASR folks are negative nancies who don't want there to be differences in DACs... my favorite possible outcome is we find an un- or under-documented mode of distortion in DACs and fix it in future DACs. That would ultimately be the point of all this, right?

The 'joy' in establishing a true audible difference is that deciding a preference for sound then becomes a valid thing.

And what audiophile doesn't love doing that?

 

[amirm]

That is the nonlinearity (==precision) of the resistance steps in rheostat mode (thus with wiper influence), not of the signal passing through in potentiometric mode (EDIT: actually we don't know which mode is used, it could well be rheostat mode with a fixed upper resistor, looking at the layout).

The variable resistor is the load for the source/DAC (voltage divider). Its non-linearities directly distort the output of the DAC as evidenced by how the same happens with distortion created in dummy loads for amplifier testing. An active load is just not going to be clean and certainly not for state of the art DACs.

 

[amirm]

I am not following how the linearity of the digital pot is a problem, as long as the setting used gets to 0.1dB matching and the resistance is stable.

Two factors are in play here:

1. Non-linearities in the "resistance" provided. An active load like this as I noted, is going to have distortion and hence the reason we don't use them for amplifier testing. They are OK for DC and non-critical applications, but not for high resolution/high fidelity applications.

2. The step you use, unless identical in both paths, will present different non-linearities.

If these were not an issue, this would be used as a volume control in products but they are not. Instead we have specialized VCAs (voltage controlled amplifiers) specially designed to be linear and keep distortion low. But even those devices can't keep up with state of the art DACs and will have level linearities.

Note that even standard resistors will have non-linearities.

 

[pablolie]

...
Amir’s record on the Archimago DAC challenge is new to me and worth considering as well.

I saw him mentioned on the Archimago post I mentioned, but can't find the actual post even though it is praised by participants. Not that is has anything to do with this particular thread.

I find it a bit sad some people can't accept any peer scrutiny before generically accusing this website of a one-sided bias that clearly doesnt exist. I admit *I* was one to loudly dismiss the results, but many in here were genuinely curious to explore the results more despite their utterly ridiculous one-sidedness (my words and opinion, not ASR's). So sue me, but do NOT judge ASR by my personal dismissal, there were a LOT of genuinely curious inquiries.

But scientific method requires peer review. And if you bow out of it, your paper is dismissed. Simple. Try passing a master's or doctoral thesis without answering review questions sometime. Try submitting a paper to the IEEE journal like that. Come on, it's bare low-level basics. There are standards for obvious reasons.

 

[pjug]

Two factors are in play here:

1. Non-linearities in the "resistance" provided. An active load like this as I noted, is going to have distortion and hence the reason we don't use them for amplifier testing. They are OK for DC and non-critical applications, but not for high resolution/high fidelity applications.

2. The step you use, unless identical in both paths, will present different non-linearities.

If these were not an issue, this would be used as a volume control in products but they are not. Instead we have specialized VCAs (voltage controlled amplifiers) specially designed to be linear and keep distortion low. But even those devices can't keep up with state of the art DACs and will have level linearities.

Note that even standard resistors will have non-linearities.

I don't know how much the resistance non-linearities come into play for line level signals and the digital pot (certainly something to consider), but your (2) is a good point I had not considered. If a single setting operation is controlling two pots in two source channels then I do think that sounds like it might be an issue depending on how it was checked. I wonder if @Josh83 was just checking the matching of L in A against L on B, and also R in A against R in B, when he says he made sure sources were within 0.1 dB.

edit: does anyone have a sense of how well our brains can tell L and R being out of balance (relatively between device A compared to in device B) in this kind of ABX test? Is it similar to the 0.2 dB threshold (or whatever it is) we have with loudness or is it something completely different? Also note that if L channels are +/-0.1dB and R channels are within 0.1 dB the L and R could be out of balance on the attenuated device by +/- 0.2 dB.

 

[hardisj]

It seems like now the ABX comparator itself is being considered as "not good enough". As someone who uses it regularly, I have found it to be quite helpful. Though, I do wish I could somehow do blind testing with digital sources. Specifically, HDMI, for when I test preamps.

That said, I would absolutely welcome a better version. I'll put my money where my mouth is, too.

I'd be happy to commission someone here to build *the ultimate ABX box* with better circuity and the addition of digital I/O. Also, silent switching. Take the current Van Alstine and let's beef it up to the point of future-proofing for the next decade. I'd gladly pay $2000-3000 for such a device. I'm dead serious.

I like the unit being remote controlled but having the results captured via SD card or via computer would be icing on the cake. Hell, even a way to directly tie in a SOTA sound card for capture, if we are gonna get wild. But, those would be pipe dreams.

I'll put up the money once the design is agreed upon. I don't know where the starting point would be but the challenge/offer is there if someone wants to take it on feel free to email me and let's talk: [email protected]

Oh, the Van Alstine's ABX box has too many speaker selectors. So this could be cut from 3 to 2.

Sincerely,
Erin

 

[Jazigo]

Super interesting topic! From my perspective - working in a company that produces RF hardware - all electronics are different even from serial number to serial number. But, the production threshold testing from unit to unit can be so low that audibility may never be in question. However, during R&D, the analog guys is constantly trying to verify data in the frequency domain and in the time domain to figure out issues. I am not an expert in analog signals - I stick to digital logic, but that makes me wonder: Are we missing some valuable data from only measuring in the frequency domain or have that been ruled out?

 

[Sir Sanders Zingmore]

Interesting how the perceived impact of the switch seems to have well, switched.
Usually it’s the “subjectivists” who say DBTs are flawed because “the switch makes everything sound the same”

Now the “objectivists” are saying the test is flawed because “the switch makes everything sound different”

 

[NTK]

Interesting how the perceived impact of the switch seems to have well, switched.
Usually it’s the “subjectivists” who say DBTs are flawed because “the switch makes everything sound the same”

Now the “objectivists” are saying the test is flawed because “the switch makes everything sound different”

Relays sequence/sounds can give cues as to either A or B is being played. See

Boston Audio Society - ABX Testing article

 

[pma]

I'd gladly pay $2000-3000 for such a device. I'm dead serious.

At such cost per pcs, one would need to sell at least 10 units to cover development costs. I’m dead serious.

 

[solderdude]

It seems like the potmeter is not buffered in any way.
A question that pops up is whether or not this is the 20k, 50k or 200k version (should be in the suffix on the chip)
I would assume the 20k version.
Each step of the potmeter attenuates 0.033dB so when 3 steps are used 0.1dB increments can easily be made.
Max attenuation would be -48dB.

In the later version relays always click to 'mask' when there is no switching. It would take a keen ear to use that 'info' to ace an ABX test.
Me thinks that if there were tells they would either have to be clipping behavior, phase issues (what if one of the used XLR to RCA happened to be put together in a different way) or audible ISO created during the resampling process that might (in certain looped segments) become audible.

It would help if those actual test files were known (not just the source files). I don't think the ABX box was the cause for being able to tell DACs apart but rather the used test signals.

Alas we will never know unless @Josh83 uploads the actual files used for his test.

It seems like now the ABX comparator itself is being considered as "not good enough".

I would not draw that conclusion though. I can see people wanting more functionality or balanced in/out etc and perhaps a more secure (so one cannot fabricate results afterwards) way of reporting the 8 attempts (I assume this would be displayed as input and chosen value for each attempt ?).
Even though the box could have been designed 'better' in many ways I guess it is functional and won't change the 'sound' in any audible detectable way.

 

[mcdn]

It seems like now the ABX comparator itself is being considered as "not good enough". As someone who uses it regularly, I have found it to be quite helpful. Though, I do wish I could somehow do blind testing with digital sources. Specifically, HDMI, for when I test preamps.

That said, I would absolutely welcome a better version. I'll put my money where my mouth is, too.

I'd be happy to commission someone here to build *the ultimate ABX box* with better circuity and the addition of digital I/O. Also, silent switching. Take the current Van Alstine and let's beef it up to the point of future-proofing for the next decade. I'd gladly pay $2000-3000 for such a device. I'm dead serious.

I like the unit being remote controlled but having the results captured via SD card or via computer would be icing on the cake. Hell, even a way to directly tie in a SOTA sound card for capture, if we are gonna get wild. But, those would be pipe dreams.

I'll put up the money once the design is agreed upon. I don't know where the starting point would be but the challenge/offer is there if someone wants to take it on feel free to email me and let's talk: [email protected]

Oh, the Van Alstine's ABX box has too many speaker selectors. So this could be cut from 3 to 2.

Sincerely,
Erin

The line level part is easy enough to design if you are happy with the performance of the TI PGA2311 volume control (-107dB THD+N). Using an ADC for calibration and a switched resistor chain we can get to 0.02dB level matching between DUTs, _and_ also level match each DUT's right and left channels if you want, to eliminate that possible cue. Controlling things through a microcontroller allows for automation of the level matching and also randomising relay switching order and delays.

The caveat is that it would have active parts including an output buffer, so some might claim it could be a source of distortion, but it would all be -105dB or better.

A/B switching power amps outputs is a question of bigger relays and some care over potential speaker/amp damage, but with an MCU randomising delays and ordering should be cue-proof.

Ping me and I'll share a preliminary design.