Archimago has a new DAC listening test

[Somafunk]

I see Archimago has a new listening test using 3 mystery dac’s, the difference in cost between them is 100x so it’ll be interesting to see if if there is much, if any difference when I listen. (I’m betting I can’t hear any difference at all).

Perhaps this fact might encourage you to listen for yourself: the difference between the least expensive and most expensive device is >100 times!

Link to Archimagos’ 2024 “High End” DAC Blind Listening Survey

 

[kemmler3D]

100x, so 20dB? Not hard to find DACs that do 90dB or 110dB SINAD, yep turns out that was a mention of price, not performance... still that's not so hard to do.

Either way, it's truly hard (without jacking up levels) to hear noise or anything at -90dB...

I'm tempted to bother trying this, on the other hand, I feel like it would be a ton of effort to hear any differences... my strategy would be to find the longest, quietest reverb tail, loop that, and jack up my headphone volume like mad. Then maybe there would be a hint of noise or HD to go on...

 

[tmtomh]

Thanks for this! And of course thanks to @Archimago for setting it up!

I didn't do an exhaustive test - I listened to only the first couple of minutes of each sample, and I didn't get up and stand very close to my speakers to see if I could hear any differences in low-level noise/hiss during the first, silent part. And I didn't try the reverb-tail test that @kemmler3D describes - which upon reflection I guess I should go back and try because I might actually hear a difference there (maybe), but oh well...

With those caveats stated, I heard a subtle but distinct difference from Sample A to Sample B, and then from Sample B to Sample C. When I switched from A to B, the initial bass impact felt "faster" with A, with more of a leading edge of impact than B had. And when I switched from B to C, the sort of background sound right after the bass impact sounded slightly more distinct on C than it had on B.

Of course, as is often the case, when I went back to A to double-check the difference I'd heard between it and B - and when I went back to B to check the difference I'd heard between it and C - I found the differences had disappeared. I hypothesize that the bass coming in on A initially seemed more impactful simply because it was the first time I'd heard it and I was taken by surprise. Conversely, I hypothesize that the background sound on C sounded more distinct because I'd already heard it before in B, and I was able to get used to it and sort of aurally decipher it more.

I don't have a proper blind ABX setup on my main system (8351b's fed digitally by Apple Music app on a Mac mini), so I did a crude version: using the Remote app on my phone, I queued up the "album" of Arch's three samples and set the app to shuffle and repeat. Then I covered the part of the screen that would show me the track title, and every minute or so I tapped the next-track button twice. That way it went to a random track, and by tapping it twice I created the possibility that it would sometimes go to the same track it had just played.

Each time, just before I advanced to the next track, I guessed at which sample I was listening to and peeked, before covering up the track title part of the display gain.

I didn't do this a ton of times, but still, my guessing results were surprising - not because I guessed wrong, but rather because I guessed wrong every single time. I mean, what are the odds of that?

 

[Somafunk]

100x, so 20dB? Not hard to find DACs that do 90dB or 110dB SINAD, so that's not too crazy at all.

But - it's truly hard (without jacking up levels) to hear noise or anything at -90dB...

The test has nothing to do with sinad,

 

[BDWoody]

100x, so 20dB? Not hard to find DACs that do 90dB or 110dB SINAD, so that's not too crazy at all.

I think that's the difference in price. (?)

 

[JustJones]

I think that's the difference in price. (?)

I think so.

From his Website.

Perhaps this fact might encourage you to listen for yourself: the difference between the least expensive and most expensive device is >100 times!

 

[gvl]

With my tinnitus kicking in strong I’m afraid I’m past the point where I could imagine I was hearing differences between DACs.

 

[wwenze]

Standard "subjective" defense:

"But the recording device is the bottleneck"

Also the mentioning of jitter somewhere

 

[wwenze]

Thanks for this! And of course thanks to @Archimago for setting it up!

I didn't do an exhaustive test - I listened to only the first couple of minutes of each sample, and I didn't get up and stand very close to my speakers to see if I could hear any differences in low-level noise/hiss during the first, silent part. And I didn't try the reverb-tail test that @kemmler3D describes - which upon reflection I guess I should go back and try because I might actually hear a difference there (maybe), but oh well...

With those caveats stated, I heard a subtle but distinct difference from Sample A to Sample B, and then from Sample B to Sample C. When I switched from A to B, the initial bass impact felt "faster" with A, with more of a leading edge of impact than B had. And when I switched from B to C, the sort of background sound right after the bass impact sounded slightly more distinct on C than it had on B.

Of course, as is often the case, when I went back to A to double-check the difference I'd heard between it and B - and when I went back to B to check the difference I'd heard between it and C - I found the differences had disappeared. I hypothesize that the bass coming in on A initially seemed more impactful simply because it was the first time I'd heard it and I was taken by surprise. Conversely, I hypothesize that the background sound on C sounded more distinct because I'd already heard it before in B, and I was able to get used to it and sort of aurally decipher it more.

I don't have a proper blind ABX setup on my main system (8351b's fed digitally by Apple Music app on a Mac mini), so I did a crude version: using the Remote app on my phone, I queued up the "album" of Arch's three samples and set the app to shuffle and repeat. Then I covered the part of the screen that would show me the track title, and every minute or so I tapped the next-track button twice. That way it went to a random track, and by tapping it twice I created the possibility that it would sometimes go to the same track it had just played.

Each time, just before I advanced to the next track, I guessed at which sample I was listening to and peeked, before covering up the track title part of the display gain.

I didn't do this a ton of times, but still, my guessing results were surprising - not because I guessed wrong, but rather because I guessed wrong every single time. I mean, what are the odds of that?

With a 2-device test, getting wrong every single time as opposed to 50% is still considered a positive detection, just that the result is the opposite of what was expected. I'm not sure how to interpret it in an ABX test however, since that would mean somehow (assuming X = A) you keep hearing X = B consistently yet you hear difference between X and A, which is illogical. Probably a systemic test issue.

With a 3 device test the expected random correct score is 33%, and the result interpretation gets more complicated depending on the combination of chosen answer vs correct answer.

Anyway, odds are an interesting thing. I just lost 10 games in a row in a game with 50% winrate, so the odds should have been 0.1%. But in a deterministic universe, the odds was 100%.

 

[Dunring]

I can only tell the color settings on many SMSL DACs. Most treble rolloff is at 15khz and I only tested to 15.5khz so no point, just go with the one with least distortion. Even most amplifiers all sound the same, years back some had a "house sound" to them. Noise floor is pretty noticable when I compared an SMSL SP200 to a Topping L30 ii using 10 ohm Truthear Zero blue IEMs. Rather make noise floor the unicorn I chase these days (might try the Corning optical USB cable next).

 

[Ron Texas]

100x isn't hard to do. There are DAC's for under $120 with a SINAD good enough to be transparent and exotic DAC's with five figure price tags which are expensive mainly because they are handmade one at a time.

 

[Doodski]

I can only tell the color settings on many SMSL DACs. Most treble rolloff is at 15khz and I only tested to 15.5khz so no point

Huh?

(might try the Corning optical USB cable next).

Seriously?

 

[tmtomh]

With a 2-device test, getting wrong every single time as opposed to 50% is still considered a positive detection, just that the result is the opposite of what was expected. I'm not sure how to interpret it in an ABX test however, since that would mean somehow (assuming X = A) you keep hearing X = B consistently yet you hear difference between X and A, which is illogical. Probably a systemic test issue.

With a 3 device test the expected random correct score is 33%, and the result interpretation gets more complicated depending on the combination of chosen answer vs correct answer.

Anyway, odds are an interesting thing. I just lost 10 games in a row in a game with 50% winrate, so the odds should have been 0.1%. But in a deterministic universe, the odds was 100%.

Good points! I would suspect my experience aligns with your final sentence - I ended up guessing based on "feel" since I didn't hear any differences, and I'm confident that had I continued to guess beyond the 7 or 8 tries I did, I would've eventually guessed right. I get what you are saying (which is why I was so surprised to be always wrong), but I really don't think I was picking up on any pattern of audible differences - I think my sample size of guesses just wasn't large enough to "even out" to the expected result.

 

[MacClintock]

Does anybody know a good (or in fact any) ABX tool for a Mac? There is this lacinato, but for some reason I do not manage to install that properly. Any suggestions?

 

[Sokel]

Does anybody know a good (or in fact any) ABX tool for a Mac? There is this lacinato, but for some reason I do not manage to install that properly. Any suggestions?

There is a foobar version for Mac as far as I know.
What I don't know is if the ABX plug-in works for Mac but I can't see why not.

Edit: Yep,works fine:

foobar2000 for Mac

foobar2000

www.foobar2000.org

(first add-on component in the list)

 

[shuppatsu]

I used Foobar's ABX comparator, which does not have a 3-way ABCX test. But even sighted I could not discern any differences listening to isolated segments of the three tracks. I therefore did not bother doing 2-way ABX testing.

 

[KSTR]

Perhaps this fact might encourage you to listen for yourself: the difference between the least expensive and most expensive device is >100 times!

No, why on earth would a large price difference -- actually a ratio, not a difference -- make a difference for anything? I wrote it sometime ago already, if a dongle DAC manufacturer decided to give away their product for free then that ratio would always go up to infinity, even when compared to another dongle DAC in the $10 range.

If the specs were 100x (40dB @kemmler3D) worse, then it would make more sense to me.

At any rate, listening tests of recordings of DACs using another DAC are not guaranteed to be flawless, we're not comparing DACs directly. It is assumed that the ADC used is flawless which is at least under the control of the initiator and the ADC he uses is a known-good one. But the playback DAC that we will be using?

Of course, it is is still a valid and valuable test and Archimago is a serious guy wrt test procedures etc. For example, as far as I can see, he recorded the DACs at a higher sample rate than the playback rate which is IMHO required for meaningful results.

 

[Brian Hall]

I couldn't really hear much difference if any. I ranked them C, B, A.

When playing the samples, my DAC showed they were 96 khz.

Is this really a valid test? Aren't we just hearing our own DACs version of what was recorded?

 

[Basic Channel]

I can only tell the color settings on many SMSL DACs. Most treble rolloff is at 15khz and I only tested to 15.5khz so no point, just go with the one with least distortion. Even most amplifiers all sound the same, years back some had a "house sound" to them. Noise floor is pretty noticable when I compared an SMSL SP200 to a Topping L30 ii using 10 ohm Truthear Zero blue IEMs. Rather make noise floor the unicorn I chase these days (might try the Corning optical USB cable next).

Most treble roll of is 15k for what? Avg human hearing?

There’s no way the noise floor between those two would be “pretty noticeable”, and the truthears would surely distort at the required listening levels anyway.

 

[Sokel]

I don't want to spoil but I wouldn't even try.