I really like
@Josh83 ‘s The Best Version Of series. In those articles he combines relevant data with subjective listening impressions and it’s always clear which is which (not to mention he’s talking about the mastering of music albums, not the sonic characteristics of gear, so a totally different subject really).
With that said, of course his ABX results were suspect from the start - not because I think he was lying (I have no idea if he was lying or not), but simply because a 98% score when comparing DACs (as opposed to transducers) is already straining credibility on its face before one inquires further.
Of course, a result that strains credibility could still turn out to be true - highly unlikely is not the same thing as impossible.
Regardless, it seems to me that the fundamental problem here has been identified and clearly stated by
@amirm and
@pma : the ABX device produces nonlinearities both mechanical (relay sound differences) and electrical (distortion and volume-dependent and possibly impedance-dependent HF droop), whose magnitude either rivals or dwarfs any possible differences between the DACs.
I understand what
@solderdude and some others are saying about the distortion and the frequency response nonlinearities being unlikely sources of problems with the test since the distortion might not be audible and even if it is, it is likely to be identically audible for both A and B - and frequency response differences are similarly likely to be extremely small given the apparently small range of volume settings Josh used in order to level-match the DACs under test.
But I don't think those flaws of this comparator unit can be dismissed so easily, and the reason is that my understanding of what Josh picked up on was shallower vs deeper soundstage, and if memory serves a statement on his part that he listened for things like reverb decay on certain instruments and such, which I view as closely related to the perception of soundstage (owing to the common theme of perception of "air" and "ambience" and such). If we entertain the idea that someone can reliably pick up on really fine details of instrument decay and ambience cues between two DACs that measure more or less identically, then of necessity we must also assume that this person must be able to reliably pick up on very small differences in audio gear performance that are likely to contribute to that perception of instrument decay and ambience. You can't consider the possibility of the former unless you also consider the equal possibility of the latter. And we know that distortion and small variances in higher-frequency response absolutely are among the factors that can contribute to the perception of air, ambience, and dimensionality - and if I'm not mistaken we also know Josh apparently heard the effects of distortion that while high measurement-wise is of questionable audibility when listening to music. So this is why I think pma's point in particular has merit here, not just as the general principle he articulates (you don't test gear using worse-performing gear), but also because the
specific issues with the ABX device cannot be separated from the sonic characteristics Josh says he detected - they cannot be separated by type, and they cannot be separated by magnitude.
And this is before we even get to the relays. My point is that if you don't feel the ABX device's measured nonlinearities are sufficient to explain the differences Josh says he heard, then you must also be doubtful that he really heard those differences at all, or you must be doubtful that the differences he did hear were really coming from the DACs themselves.
And all that leaves is that he either cheated or lied in some way - or (IMHO more likely) that the relays (and/or possibly the FR nonlinearities of the box) were the tell, whether conscious/intentional or unconscious/unintentional.
And so finally, I do agree that it would be good - and IMHO important - to produce a recording of the relays that is of sufficient quality to capture the audible difference Amir reports in his review. Not because I personally don't believe Amir - I absolutely believe him. I believe him not because I think he's the one human being on earth who's free from bias, but rather because the difference he reports is so obvious and of such greater magnitude than any other possible or actual nonlinearity in this whole situation that it would be kind of insane for him to report it if it weren't true, since it would be so incredibly easy to falsify his claim in that case.
But it doesn't matter if I believe him - what matters is the objective data. A recording would, as solderdude says, put that one doubt to bed. Of course it won't convince the folks at AudiophileStyle, but who cares? What matters is
our standards here. And while I don't doubt Amir and I don't think anyone else does either, the absence of a recording of those relays is precisely the kind of thing we would jump all over if the shoe were on the other foot. So I would respectfully request such a recording if Amir is willing to do one, even if we have to wait some time for it and even if it requires another member loaning/sending him a sufficiently good mic.
