Hi! Good idea for a thread, considering the discussions currently floating in public opinion (also referring to Jim Lill's video on preamp influence on sound, which despite somewhat imprecise methodologically speaking, raises useful points).
Had some fun and very obsessive moments ABX testing this. I mix and master music for a living for quite some years now, so I was very interested in taking the test, since critical listening and blind tests are daily tasks. I find it important to keep a constantly updated sense of where the threshold of perception sits at for a given audio technique, since science informed decision-making necessarily looks different in either side of said threshold. For example, 320 kbps mp3s are consistently identifiable in ABX testing by well trained ears (with the caveat of most differences being present in the high end, where listener age matters most), but mostly undistinguishable for most people. Therefore, knowing exactly how small the difference is for most program material, I have no problem sending clients 320 mp3s for early mix evaluation. This forum has been important for solidifying knowledge on much of what I'm describing.
On to the tests. I did one round of training on foobar to get a general impression of the difference between files. It was clearly obvious, on first listen, that the difference would either only be relevant to the upper percentile of listeners or be mostly undetectable. Starting the test, I set up a 2 or 3 bar time range, making sure playback always restarted when switching X/Y (along the time domain, program material with physical instruments has inifinitely more timbre variation than electronics). After listening through most of the spectrum for differences, including low end transients, which can sound longer or less sharp when a lot of group delay is present, as well as imaging, I decided to focus on the very high end of the spectrum. That's where I expect most differences to be. The ride cymbal is perfect for that - sharp transients with complex frequency and time-domain information + lots of high end content. I believed I was noticing an extremely small difference, where on file B the ride's transient was more solid but the cymbal's tail had a touch less high end. Ran 4 tests, spaced out over a few hours, since the test is quite exhausting. Results (foobar files listed below): 9/16, 8/16, 10/16, 11/16, 10/16. After finishing the first test I was quite tired and decided that I should take some breaks during the next few tests, since the last few trial rounds always felt less precise. I kept feeling like I was getting progressively better at the test and learning the program material, yet the results never reached sufficient statistical significance, so I eventually called it a day. No matter how much I wanted to pass a test, even if I had reached 12/16 on a single one, it wouldn't matter much without being able to consistently replicate it.
In frustration, today I decided to try a fifth test focusing on different sounds. Looked around for differences and my brain decided that I could hear the side channel (the difference between L and R channels) "compressing" ie losing amplitude during loud vocal passages in a specific section and ran a test focusing on that. The result was a glorious 3/16.
Final remarks: Yesterday I was determined to believe I could hear a difference in the ride cymbal, but that I was being held back by fatigue. In full honesty I have serious doubts I ever perceived a difference present in the files, no matter how clear that seemed at the moment. I am fairly sure that hallucinating differences is extremely common on untested critical listening of this level and, in fact, think it might unfortunately have a much bigger role in audio quality judgements on an everyday basis than actual testing with solid methodology. Most of my conclusions stemming from this test are not about DACs, but about methodologies and ethics in audio testing. Uncontrolled confounding variables simply have way too large a magnitude of influence to be overcome by listening experience in many situations eg: trained listeners trying to listen beyond room influence when comparing speakers in different spaces. And, as timbre and psychoacoustic literature has shown, within confounding variables, context specifically is show to have an unexpectedly large influence in the way we perceive timbre.
I also purposefully didn't search for very low amplitude moments in the test signal, where noise floor or low amplitude conversion artifacts could be present. If you have to wait for the last few seconds of a fade-out or reverb tail to distinguish two pieces of equipment, I don't believe that can be pointed out as a timbre difference, considering the musical program material humans use audio equipment for. Unless that low level artifact clearly sounds worse or less faithful, that's in the realm of testing lab equipment in my opinion.
I am currently away on holidays, so I listened on my mobile setup (AKG K712 Pro connected to a JCALLY JM6 Pro). Will test again in the studio with a better setup. It's possible that this DAC can't reproduce the files faithfully enough. I tried the test on my Bose Quietcomfort NR headphones (using the JCALLY and a cable), and the lack of definition in the high end, either from the ADC/DCA, DSP processing, or the headphones themselves, made it impossible to even approach the ride cymbal differences - the high end sounds much more imprecise and scattered in time. To finish off, assuming my results are valid, I still would really like to abx test signals routed through multiple ADC/DAC stages, as that's something I strive to avoid in my setup.
