Blind testing two DACs - Proper Process

[davmol]

May I ask which pieces of music you used for the comparison?

I listened mainly to these playlists on Tidal and it Spotify. I tried to listen to a reasonably wide range of genres. The philharmonic does pop/rock was the closest I could get to classical without falling asleep, so I'll have to try harder in future

ASR Critical Music Tracks for Speaker & Room Testing

Listen to ASR Critical Music Tracks for Speaker & Room Testing on TIDAL

tidal.com

Hi-Res FLAC - 24bit/192kHz

Playlist - Hi-Res FLAC - 24bit/192kHz - 75 items

tidal.com

Electronic for Beginners

Playlist - Electronic for Beginners - 20 items

tidal.com

 

[HeebieGenie]

The amount of anecdotal support for more expensive DACs and amps being noticeably better than cheaper ones is so overwhelmingly large that I have always thought in the past "Surely this HAS to be true, right?"

I have never heard a true difference in sound quality, only volume. But, people just tell me "You don't have the ears to hear the difference, I would pass blind tests"

 

[krabapple]

The amount of anecdotal support for more expensive DACs and amps being noticeably better than cheaper ones is so overwhelmingly large that I have always thought "Surely this HAS to be true, right?"

And now you know that line of reasoning is mistaken generally, right?

 

[antcollinet]

You don't have the ears to hear the difference, I would pass blind tests"

And yet they never try.

 

[HeebieGenie]

And now you know that line of reasoning is mistaken generally, right?

Yes, I tend to believe that nowadays. I really think it's placebo of some kind, or psychoacoustics.

 

[NTTY]

"bad" though the mini is - it still achieves 85+dB Sinad (0.006% THD+N). Still pretty unlikely you'll be able to hear any difference between that and a "better" DAC.

And an interesting investigation because of that.

And that's only at full scale. Things can get messy when far from 0dBFS.

This is the worst (high-end) DAC I’ve seen, at full scale:

Significant distortion as per today's standard, but not something we can hear.

Let's try @-40dBFS:

Yep, that's 2% distortion... And it was 14% @-60dBFS, with a non linearity reaching a whooping 3dB! (signal played too loud). Really nasty...

In a blind test (record of music played by two DACs including this one, with high precision interface (input gain adjusted to less than 0.02dB), then played in foobar with the ABX plugin), compared to another well behaved DAC, it was not possible to identify this ugly one (with Tracy Chapman - Fast Car (don't shoot) and Diana Krall - Peel me a grape (don't shoot again)).

That's when I usually tell my friends to give a try to the Klippel audio test, it's fun to know what level of distortion we can hear.

Cheers

 

[Old_School_Brad]

And that's only at full scale. Things can get messy when far from 0dBFS.

This is my worst (high-end) DAC at full scale:

View attachment 418060

Significant distortion as per today's standard, but not something we can hear.

Let's try @-40dBFS:

View attachment 418061

Yep, that's 2% distortion... And it was 14% @-60dBFS, with a non linearity reaching a whooping 3dB! (signal played too loud). Really nasty...

In a blind test (record of music played by two DACs including this one, with high precision interface (input gain adjusted to less than 0.02dB), then played in foobar with the ABX plugin), compared to another well behaved DAC, it was not possible to identify this ugly one (with Tracy Chapman - Fast Car (don't shoot) and Diana Krall - Peel me a grape (don't shoot again)).

That's when I usually tell my friends to give a try to the Klippel audio test, it's fun to know what level of distortion we can hear.

Cheers

Yep, it’s quite an eye-opener when you realize how bad things have to get before it becomes audible. It kind of makes the whole SINAD race feel pointless -well, that's not entirely fair of me to say, but let's call it more of a showcase of engineering excellence. In practical terms though...

 

[NTTY]

SINAD is indeed a showcase, especially when used to rank items, it’s inevitable. And it could be done at different output scales. It is, by the way, since Amir, by his own convention, looks at it at 4Vrms output (balanced) and would reduce digitally the sine tone to match this target, if a DAC would output one or two dB more, and even if it wouldn’t be what the end user would get from the device in real life (listening to a master that reaches 0dFBS).

That being said, and because it mixes THD and Noise into one measure, we know that when SINAD is good (let’s say above 90dB) then the device does not suffer from significant noise or harmonic distortion. At a lower SINAD, there’s indeed a need to understand what is the source of the problem. Harmonic distortion would remain hidden into music until very high levels, not so much about noise.

 

[antcollinet]

And that's only at full scale. Things can get messy when far from 0dBFS.

This is the worst (high-end) DAC I’ve seen, at full scale:

View attachment 418060

Significant distortion as per today's standard, but not something we can hear.

Let's try @-40dBFS:

View attachment 418061

Yep, that's 2% distortion... And it was 14% @-60dBFS, with a non linearity reaching a whooping 3dB! (signal played too loud). Really nasty...

In a blind test (record of music played by two DACs including this one, with high precision interface (input gain adjusted to less than 0.02dB), then played in foobar with the ABX plugin), compared to another well behaved DAC, it was not possible to identify this ugly one (with Tracy Chapman - Fast Car (don't shoot) and Diana Krall - Peel me a grape (don't shoot again)).

That's when I usually tell my friends to give a try to the Klippel audio test, it's fun to know what level of distortion we can hear.

Cheers

Thats odd - wouldn't we normally expect non linear distortion products to drop in proportion to the test signal? WTH is that DAC doing?

 

[NTTY]

Thats odd - wouldn't we normally expect non linear distortion products to drop in proportion to the test signal? WTH is that DAC doing?

With 1bit DACs, yes, with older ones (R2R here), less so. But this one was quite a joke indeed, high-end of the time, at least by its price.

 

[davmol]

Just picked up a pair of Linton 85th speakers and for a bit of if fun I decided to repeat this test between the WiiM Ultra and the Fiio K11 R2R. I found a perfect voltage match at 94 on the Fiio and 40 on the WiiM. It was not quite as seemless switching this time, because there is a delay moving between usb out and line out on the Wiim Ultra, but I couldn't tell any difference between the two DACs when level matched.

My favourite part of this particular test was when I decided to switch the K11 from NOS to OS, just to see if there was an audible difference. After about ten minutes of switching back and forth and listening to different tracks, we were convinced that the NOS setting sounded a little fuller / a little more bass, and we preferred it that way. Then I realised that during the whole time we had been testing, the ultra had been set to line-out, so we had actually been listening to the WiiM DAC and not the Fiio! Imagination is powerful

 

[davmol]

Been a while but went back to do some DAC comparisons today. The reason being that I was using the WiiM Ultra with USB-out to Topping DX5II, but I decided to switch it out with the Fosi ZH3, mainly for aesthetic reasons, but also because with the (seemingly never ending) firmware updates for the Topping, I felt it was better to leave it in my office permanently attached to a PC.

When I started listening with the ZH3, I was taken aback that it seemed quite different (worse) than the DX5II, almost being harsh in treble and lacking a little in bass. So, needing to satisfy myself once again that this due was nothing more than volume changes, I broke the glass on my emergency audiophile-reality-check-toolkit (1-to-3 optical splitter; 1-to-4 RCA switching box, and a multi-meter) and got to work. I also decided to add in the Fiio K11 R2R just for nostalgia, given it is still the audiophile forum users' budget pick for "most musical DAC", and so much so that I was recently told if I don't hear how good it is then I need my hearing testing

Anyway, low and behold the ZH3 is significantly quieter out of the box than the DX5II, which is a chunk quieter again than the K11 R2R. I played around with their pre-amp volume controls to match voltage outputs as closely as I could, at which point the ZH3 and DX5II sounded identical. I could still notice a tiny volume difference when switching between the Fiio and the Topping, which last time around I resolved by using a second synced WiiM device as an additional volume control. However, this being a follow-up test, I had already satisfied myself that the differences were just volume related and by now the ZH3 sounded as good as the others, so I couldn't be bothered to go to that extreme again, knowing the outcome would be the same.

So I guess it's no surprise people like the K11 R2R based on the fact it seems quite a bit louder than other DACs, and it does make me wonder if DAC manufacturers might intentionally bump up the output to present a "better DAC". The second thing it made me think of is the comment I often see on audiophile forums where people will dismiss the validity of rapid A/B and ABX tests in favour of leaving a DAC in place for a longer period of time, and then switching back at a later date, which will "emphasise the difference". Well, if they don't level match the DACs before switching back then, as per my experience with the ZH3 vs the DX5II, of course they will hear a difference, but it is still just related to volume / output differences between the two DACs.

Anyway, below are readings from the multimeter, along with a picture of the "DAC Stack" test setup that includes a sulky Golden Retriever who is sick of the hifi getting more attention than him:

 

[obsessive9119]

My favourite part of this particular test was when I decided to switch the K11 from NOS to OS, just to see if there was an audible difference. After about ten minutes of switching back and forth and listening to different tracks, we were convinced that the NOS setting sounded a little fuller / a little more bass, and we preferred it that way. Then I realised that during the whole time we had been testing, the ultra had been set to line-out, so we had actually been listening to the WiiM DAC and not the Fiio! Imagination is powerful

I had a similar experience when I was fiddling with crossfeed and other DSPs in AU Lab. I was switching back and forth between a couple of different settings, tweaking things and was really happy with what I’d landed on…. and then realised I had the whole effect chain muted and had literally been making it up.