That's why he does 2 runs in a row.
A question to
@Josh83 could be what the actual test signal is ?
The ability of being able to tell DACs apart audibly is highly dependent on the used test signals.
Can you upload the used test signal that allows the DAC's to be pretty reliably' differentiated ?
I listed the songs in the article: two Van Morrison songs (“Rough God Goes Riding” and “Ancient Highway”), two Steely Dan tracks (“Negative Girl” and “Cousin Dupree”), one Jenny Lewis song (“Head Underwater”), and a transient-rich test signal, “Drum Solo Stereo,” from the Audio Check album available on Qobuz.
In line with what I almost always do for serious level-matched listening, I was using short, repeated segments from those tracks that I felt were revealing of differences and so I was making a true apples-to-apples comparison, rather than listening to different parts of the songs through A versus B versus X.
In case you have Cosmos ADC, would you be willing to share ADC record samples of the test music signals? With volume settings as during the test?
I don’t have time to do that now, but I may try at some point. However, for me the reason for buying the ABX box was to eliminate variables and additional AD/DA conversions. Otherwise, recording the DACs’ outputs directly into the Cosmos, then doing the ABX testing in DeltaWave would seem like a superior ABX test to including the ABX box in the same process.
@Josh83 Thanks for swinging by the thread and answering the questions in detail! I think this is a really cool experiment and brings up some interesting questions. Coming back to some of the points being discussed:
Delay, filters and source:
- I think the delay click you mention and the corresponding switching time (maybe around half a second?) make it very unlikely that any delay between the DAC outputs is relevant when testing using this method.
- Since the filters were closely matched, this should also be fine and fair and further mitigates any delay difference concerns
- Since the Mutec MC 1.2 only offers one optical out and one BNC, it would be good to know how exactly you connected the DACs to the Mutec
Analysis and possible causes or error:
- As pointed out by @Alexander Gödde in this post, it seems like the same model comparator but a potentially older unit was used in this dagogo series of posts. What is most puzzling about this is that the author of that post managed to ABX some DACs and line level signal cables (?!), but failed to achieve any positive result with amplifiers.
With the SINAD performance of average amplifiers being pretty much an order of magnitude worse than decent DACs (around 80 dB vs. >100 dB SINAD) and amplifiers driving a much more demanding load in the form of speakers, this result is extremely counterintuitive. As pointed out by Alexander, this points to a hidden problem with the line level signal paths in the comparator. It would also be possible that the speaker level signal path is somehow "obfuscating" the results between amps while the line level path is "more transparent", but this possibility doesn't invalidate the arising concern.
- As mentioned in my initial post, I have conducted some ABX tests myself and would judge those as significantly less challenging than what you did here. Nontheless, I have rarely scored a perfect 10/10. There is simply an element of uncertainty and maybe listening fatigue to them, which makes me question the abundantly clear 110/112 result you achieved in this test. Not in the "he's cheating" way, but in the "the differences would have to be overwhelmingly audible to repeatedly score that high" way. As multiple DAC comparisons on ASR and online have shown that differences - if present - must be miniscule and even actual null tests showing only the tiniest of differences to the original digital waveform following a full DAC->ADC loop, this is from my point of view not compatible with an essentially perfect ABX score. The fact that some DAC pairings you tested used similar (but not identical) top of the line chips from the same manufacturer makes the result even more unusual. For me, this points the "something in the line-level signal chain causes unexpected differences" hypothesis.
- I can fully understand that you don't want to send the box across the continent with it being so rare and hard to come by. However, from a scientific point of view this is an unfortunate thing: As it stands right now, nobody can cross-check your test setup except for yourself and - maybe more important - replication experiments are currently impossible. With replication being a big part of good science, this is disappointing. I hope some solution can be found in the future. Maybe a member owning the right equipment is willing to travel to you, or you are willing to travel to them with the box. Maybe there are easier and less cumbersome solutions to verify and replicate the experiment.
Further testing:
- I think the suggestions by fellow members to test the comparator with a single DAC + a Y-cable are the best next step. This would have to be done blind, though - so a friend would need to set up maybe three comparison pairs behind a curtain without you interacting, ensuring that it's not known to you which DACs are connected or if anything on the input side changed at all. That's definitely a bit cumbersome to do, but at least it doesn't cost any additional money.
- I also think an ABX test with amps would be a good next step, taking into account that the dagogo author failed that using his comparator.
- If you happen to measure the box again using your Cosmos ADC, it would be good to do so with two or three different attenuations in the range you were using during the ABX test to level match the DACs. I'm sure other members have additional ideas on what to check.
I really hope that you're going to continue to converse with fellow members and especially Amir on this result and are open to checking it further, even though that might be tiring and not generate any additional clicks for audiophilestyle. Because as it stands now, it leaves a lot of very interesting questions open to debate, which on the one hand is fascinating but on the other hand it's always unsatisfying to not have a definitive answer.
Thank you for the detailed response. As far as the Mutec, one DAC received RCA SPDIF and the other received AES. I’ve personally never detected or suspected audible differences between digital inputs, so that didn’t concern me, and I assumed most objectivists wouldn’t object, given that “bits are bits.”
Can you explain more what you’re suggesting with the same DAC and the Y-cable? Just to make sure I follow — you’d want the same DAC split to both inputs, so the test is whether I can distinguish it from itself? Given that the Multitone tests were nearly identical for inputs A and B with the same DAC, I wouldn’t expect that I could.
For amps, I’ve yet to try that feature of the ABX box, and I only have one power amp and one integrated amp (which measure fairly differently) so I don’t have the ability to do multiple comparisons as with DACs. (It’s worth noting that I’m selling some of my DACs, many of which I kept on hand only long enough to do the ABX testing.)
As far as the Dagogo results, I can only speak for my own tests. I’ve never invested much in cables and wouldn’t expect to pass a cable ABX myself.
Given the questions about the Van Alstine box, I hope Dan Kuechle will weigh in. Since there are so many EEs here on ASR, perhaps someone can design a simple, clean two-in, one-out XLR ABX box for DACs and send it to me. I’d happily pay for it.
Very well considered post, RandomEar, thanks for this.
Josh83's results (thank you too for posting here Josh) so strongly contradicted my priors I was skeptical but as RandomEar suggests, the strength of the result is objectively the most odd.
In the article, Josh83's "subjective scale" judgement of 80% to 95% (best case) similarity is far from the less than 1% difference I would expect. But, it is fully justified by the strength of his results. This does not seem right. But, the discussion does seem very right here, thanks again Josh83 and OP Leporello.
I’ll write more about this in a follow-up, where I’m going to talk about why I think the subjectivist/objectivist dichotomy is an oversimplification, but my conclusion is definitely not that anyone should see my results as implying that they would hear these differences or find them meaningful enough to be worth investing in a new DAC.
Indeed, I don’t believe in telling people that they’re not a “real audiophile” if their equipment isn’t “good enough.” I want to bring more people, especially young people, into audiophiledom, not push them out over what they can or can’t afford.
