Blind testing two DACs - Proper Process

[davmol]

I will rewrite the instructions this weekend to a new version to take in all this feedback (and then ask for more). In terms of switching, it should hopefully be very simple. The DACs will plug into a switching box, the other party will have the remote control for the switching box and they will not be visible to me. I will ask for A or B or X and they will switch to the correct channel, so it should be simple. I can go fast slow as needed, or until my wife's patience runs out.

It is a 4 way switching box and I'm considering using splitter cables to plug each DAC into two ports e.g. A is on channels 1 & 2, with B on channels 3&4. This might help eliminate any tell from switching, such as an audible click or pause. Then, for example, if X=A and I'm listening to A, when I request to hear X, there will still be a switch occurring, even though it's still the same DAC. Hope that makes sense. I know splitters will upset the cable police but I'm happy to live with that. This is all still theory and I'll be checking all of this actually works, so it will be a while yet before I do any actual testing.

 

[davmol]

FYI, the DAC has arrived. I plugged it in and thought OMG, this sounds awful! Then I realised I was using my Sundara closed without EQ

Nice little device, it sounds good, can't say anymore than that yet. Unlike the Topping there is no option in Windows to set the Fiio's sample rate. I checked the instructions and it says that in OS mode it will upsample everything to 384khz, so I'll need to set the Topping to the same when I get around to testing.

 

[Matt_Holland]

It is a 4 way switching box and I'm considering using splitter cables to plug each DAC into two ports e.g. A is on channels 1 & 2, with B on channels 3&4. This might help eliminate any tell from switching, such as an audible click or pause. Then, for example, if X=A and I'm listening to A, when I request to hear X, there will still be a switch occurring, even though it's still the same DAC. Hope that makes sense. I know splitters will upset the cable police but I'm happy to live with that. This is all still theory and I'll be checking all of this actually works, so it will be a while yet before I do any actual testing.

This is a great idea!

Have you decided how you will level match the DACs?
The Topping has a 2.5Vrms mode and the Fiio is 2.7Vrms according to specs. A theoretical difference of 0.6dB

 

[davmol]

I plan to level match with a multimeter and both DACs have volume control so will see how close I can get them.

Also just found the Fiio does have sample rate settings in Windows 11, it's just hiding in the old school menu and not the shiny initial system sounds menu (access by clicking "More Sound Settings").

 

[yo!]

Loudness has to be spot on! The slightest louder signal will get your attention and sounds 'better', 0.2 dB over full spectrum could be already enough.

Good designed DACs will differ audibly, if at all, in the analogue branch. There is more room for interactions that could be heard. The sampling rates above 48 kHz don't make a difference, and jitter or similar things are no longer an issue for a long time, if they ever were. A small movement of the head in front of the speakers or the headphones being moved a few millimeters can cause tonal effects that are almost gigantic in comparison.

This reminds me of a Sony ES amplifier with some 'super direct clean' button or something like this. Yes, it was audible. They simply added a high shelf to make the unit sound more airy in 'direct' mode

 

[davmol]

So, as recommended, I did a quick little sighted A/B test using the pre-out from the Fiio K11 R2R into the Topping A50 III amp, and I also had the D50 III into the A50 III at the same time. I used an app called Voicemeeter to simultaneously output to both DACs at the same time over their respective USB connections. I level matched only by ear and then played various tracks from Spotify and Tidal with various sample rates at 48/96/192khz (24 bit), all while flipping back and forth between the two DACs using the input selector switch on the back of the A50 amp. I couldn't detect an audible difference between the two DACs, the playback was seemless with no gaps and I couldn't pick anything up at all other than they both sounded good. This was with the Fiio in OS mode and I used my Sundara closed yesterday and my Ananda open backs today.

When I put the Fiio into NOS mode I do feel like something was a very slightly different but I wasn't sure what I was hearing. I switched the D50 filter from my standard linear phase fast roll off (mode 3) to using minimum phase slow roll off (mode 7) at which point I could no longer detect a difference, say maybe the NOS is acting like a filter (or replicated by a slow roll off filter)?

Anyway, this was just a quick sighted test to see if I personally can hear enough of a difference between the DACs to justify a full blind test and I can't, so not much point continuing because I would just be guessing. However, I will first try them with a different source (that is not USB from WIndows) and will again do a basic level matched test just to be doubly sure that I can't pick up any difference. Is it worth me trying this simple AB with my amp and speakers, or are headphones the better bet for identifying such reported differences in DACs as "wider soundstage" etc? Also my speaker amp is class A, so I'm sure that will please some people

I genuinely wanted to hear a difference but I'm still enjoying this process and if, after trying other sources, I still find nothing different about the R2R DAC then I'll move on to compare the Chord Mojo 2, because that seems to be another favourite that everyone says sounds very different to DS DACs, so I will be interested to hear it.

 

[Blumlein 88]

So, as recommended, I did a quick little sighted A/B test using the pre-out from the Fiio K11 R2R into the Topping A50 III amp, and I also had the D50 III into the A50 III at the same time. I used an app called Voicemeeter to simultaneously output to both DACs at the same time over their respective USB connections. I level matched only by ear and then played various tracks from Spotify and Tidal with various sample rates at 48/96/192khz (24 bit), all while flipping back and forth between the two DACs using the input selector switch on the back of the A50 amp. I couldn't detect an audible difference between the two DACs, the playback was seemless with no gaps and I couldn't pick anything up at all other than they both sounded good. This was with the Fiio in OS mode and I used my Sundara closed yesterday and my Ananda open backs today.

When I put the Fiio into NOS mode I do feel like something was a very slightly different but I wasn't sure what I was hearing. I switched the D50 filter from my standard linear phase fast roll off (mode 3) to using minimum phase slow roll off (mode 7) at which point I could no longer detect a difference, say maybe the NOS is acting like a filter (or replicated by a slow roll off filter)?

Anyway, this was just a quick sighted test to see if I personally can hear enough of a difference between the DACs to justify a full blind test and I can't, so not much point continuing because I would just be guessing. However, I will first try them with a different source (that is not USB from WIndows) and will again do a basic level matched test just to be doubly sure that I can't pick up any difference. Is it worth me trying this simple AB with my amp and speakers, or are headphones the better bet for identifying such reported differences in DACs as "wider soundstage" etc? Also my speaker amp is class A, so I'm sure that will please some people

I genuinely wanted to hear a difference but I'm still enjoying this process and if, after trying other sources, I still find nothing different about the R2R DAC then I'll move on to compare the Chord Mojo 2, because that seems to be another favourite that everyone says sounds very different to DS DACs, so I will be interested to hear it.

Wider soundstage is mostly puffery. We only need about 20 db difference to hear maximum stereo effect. Maybe more with headphones.

Otherwise if you have some good headphones they probably will be more revealing of small differences.

NOS probably has a slight upper frequency roll off below 20 khz. So does the mode 7 filter. So that is possibly what you were hearing.

Do please match levels with a voltmeter if you have one. It literally won't be more trouble than by ear. Get the readings within 1.2% of each other or as close to that as possible. Use a 400 hz tone for this.

 

[davmol]

I will match with meter when I a/b them again with an alternative source at the weekend.

 

[davmol]

Just for info, once I'm on a mission I struggle to move on from it until I have completed something meaningful, so I'm still thinking about how I can complete this testing process to my own satisfaction.

With that preamble covered, I just realised I have ESS DACs, and I now have the Fiio R2R, and I'm also looking at the Chord Mojo 2 (FPGA), but I haven't yet considered an AKM based DAC. I would like to include an AKM based DAC in this testing and will be on the lookup for a good choice with the criteria "relatively cheap, objectively measures well, and has received positive subjective reviews"? On that basis I was thinking about the SMSL SU-1, given that I've seen a great review on ASR and also a number of really good reviews on some of the more traditionally subjective / expensive gear reviewing YouTube channels. I'm happy to consider other AKM based DACs up to £150 new or used, or possibly up to £300 if from Amazon, so I can return it if I decide it offers no benefit and don't want to keep it.

I realise there are people in the "must spend £1000+ to get good performance from a DAC" but that's not in my scope for now, as I'm just looking to satisfy my own curiosity about objectively competent DACs of different architecture, where subjective reviewers have said they have distinctly different sound signatures (e.g. it seems common for subjective reviewers to say AKM's "velvety sound" is warmer, more musical, more natural, than the "analytical" ESS DACs etc).

Another follow up question - are there any established metrics / clearly defined language to consider and use for describing audible differences, should I happen to encounter them along in this journey? I would like to increase my scientific vocabulary / audio terminology.

Thanks!

 

[JeremyFife]

So, as recommended, I did a quick little sighted A/B test using the pre-out from the Fiio K11 R2R into the Topping A50 III amp, and I also had the D50 III into the A50 III at the same time. I used an app called Voicemeeter to simultaneously output to both DACs at the same time over their respective USB connections. I level matched only by ear and then played various tracks from Spotify and Tidal with various sample rates at 48/96/192khz (24 bit), all while flipping back and forth between the two DACs using the input selector switch on the back of the A50 amp. I couldn't detect an audible difference between the two DACs, the playback was seemless with no gaps and I couldn't pick anything up at all other than they both sounded good. This was with the Fiio in OS mode and I used my Sundara closed yesterday and my Ananda open backs today.

When I put the Fiio into NOS mode I do feel like something was a very slightly different but I wasn't sure what I was hearing. I switched the D50 filter from my standard linear phase fast roll off (mode 3) to using minimum phase slow roll off (mode 7) at which point I could no longer detect a difference, say maybe the NOS is acting like a filter (or replicated by a slow roll off filter)?

Anyway, this was just a quick sighted test to see if I personally can hear enough of a difference between the DACs to justify a full blind test and I can't, so not much point continuing because I would just be guessing. However, I will first try them with a different source (that is not USB from WIndows) and will again do a basic level matched test just to be doubly sure that I can't pick up any difference. Is it worth me trying this simple AB with my amp and speakers, or are headphones the better bet for identifying such reported differences in DACs as "wider soundstage" etc? Also my speaker amp is class A, so I'm sure that will please some people

I genuinely wanted to hear a difference but I'm still enjoying this process and if, after trying other sources, I still find nothing different about the R2R DAC then I'll move on to compare the Chord Mojo 2, because that seems to be another favourite that everyone says sounds very different to DS DACs, so I will be interested to hear it.

Nice work. Hardly anyone goes to this trouble, and shares their findings. Great to read about.

Do try to level match more accurately, it's an obvious source for inconsistencies and risks adding confusion.

Thank you for sharing

 

[antcollinet]

Nice work. Hardly anyone goes to this trouble, and shares their findings. Great to read about.

....

Thank you for sharing

Seconded.

One day I'll put my money where my mouth is too.

 

[Soniclife]

I couldn't detect an audible difference between the two DACs, the playback was seemless with no gaps and I couldn't pick anything up at all other than they both sounded good.

Few people reading this will be surprised by this, but congrats on doing the work, not enough do. I'd reflect on which sources of information predicted this, or something different, there is more important stuff to learn, and it applies to everything, not just hifi.

 

[davmol]

And so it begins...
Bottom to top:
R2R, ESS, AKM, FPGA

Spoiler alert, after a quick level match by ear, I can't tell any difference (nor can my wife). Excited to see how close I got level matching by ear when I use the multi meter later this week

I'll be using these all week to give a good A/B and see if I can detect anything at all with some more careful listening.

 

[Purité Audio]

Which switching unit are you using , any issues with it?
Keith

 

[krabapple]

If you really want to go whole hog, start with deliberately level MISmatched outputs, to a degree easily ABX'd, then gradually decrease the mismatch. If the difference is simply output level, you'll find the match point where they become indistinguishable to you.

 

[davmol]

Which switching unit are you using , any issues with it?
Keith

It's a cheap box from Amazon, had decent reviews:
https://www.amazon.co.uk/gp/product/B0CQ4SZF7B/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8

Build is as expected at this price: not too solid, not too flimsy. Overall as a switcher it is almost seamless, there is a quiet but audible "pop" on changeover, although if music was loud enough / lost of drums etc, you may not even notice it changing. The fact it comes with a remote control is a bonus and it can be used either as 4-in/1-out, or 1-in/4-out. It doesn't have any volume adjustment per channel, so is something to consider if you need to level match devices that don't have their own. In my case the SU-1 has no volume control but all the other devices do, so I can get away with matching to the SU-1.

Long term reliability aside (being an unknown with this no-name stuff), I'm quite pleased so far for the price.

 

[davmol]

Update:

• Listened for around 6 hours on headphones yesterday via the A50III amp while working, plus around 30 minutes through speakers via a Musical Fidelity A1 clone amp.
• Regularly switched between DACs, listened to a range of electronic, rock, pop, blues. I don't listen to much classical or jazz but will give some a try at some point.
  • If there are any suggestions of tracks that might demonstrate clear differences between the DACs then let me know. I tried a couple of "audiophile soundstage" type playlists on Tidal today but no differences found.
• For a moment thought I was hearing a slight difference in timing between Mojo 2 & SU-1, vs the K11 & D50III, almost like the former were more "relaxed" or laid back. I had a think for a bit and realised the first 2 DACs were connected by an optical splitter to the Wiim Mini and the latter 2 DACs by an optical splitter to the Ultra, with the audio running in sync mode. I decided to run all 4 DACs to the Wiim Mini, via various optical splitters and the difference went away. There must have been a tiny bit of latency that was momentarily noticeable when quickly switching between the two Wiim outputs.
  • It was interesting that the sample rate drops to 48khz when running in sync across the Ultra and Mini.. When I run audio from just one Wiim then it will happily play 96/24, even through a bunch of optical splitters. I did try 192/24 but it was a bridge too far in this case, so I'm sticking with 96/24 and one Wiim from here on.
• I still go through occasional moments thinking I hear "smoother" bass, with the Mojo 2 in particular, but then when I replay the sections (where I thought I heard it) on the other DACs, it sounds just the same. Maybe I just want the Mojo to sound smoother for the extra wedge of cash it costs, along with all the marketing guff!
• I had fun with the speakers, turned up nice and loud and I couldn't hear a difference. If I cycled quickly through the DACs via the remote switcher there was barely even a break in playback. The cheap little switcher is quite impressive.

My view as of today is that I still hear no discernible difference between these DACs when listening to music. I will continue listening into the weekend and then get out the multi-meter to check and align levels and then carry on again into next week, just to be 100% sure there is no clear difference that I can hear that would justify a blind test.

 

[Soniclife]

Update:

• Listened for around 6 hours on headphones yesterday via the A50III amp while working, plus around 30 minutes through speakers via a Musical Fidelity A1 clone amp.
• Regularly switched between DACs, listened to a range of electronic, rock, pop, blues. I don't listen to much classical or jazz but will give some a try at some point.
  • If there are any suggestions of tracks that might demonstrate clear differences between the DACs then let me know. I tried a couple of "audiophile soundstage" type playlists on Tidal today but no differences found.
• For a moment thought I was hearing a slight difference in timing between Mojo 2 & SU-1, vs the K11 & D50III, almost like the former were more "relaxed" or laid back. I had a think for a bit and realised the first 2 DACs were connected by an optical splitter to the Wiim Mini and the latter 2 DACs by an optical splitter to the Ultra, with the audio running in sync mode. I decided to run all 4 DACs to the Wiim Mini, via various optical splitters and the difference went away. There must have been a tiny bit of latency that was momentarily noticeable when quickly switching between the two Wiim outputs.
  • It was interesting that the sample rate drops to 48khz when running in sync across the Ultra and Mini.. When I run audio from just one Wiim then it will happily play 96/24, even through a bunch of optical splitters. I did try 192/24 but it was a bridge too far in this case, so I'm sticking with 96/24 and one Wiim from here on.
• I still go through occasional moments thinking I hear "smoother" bass, with the Mojo 2 in particular, but then when I replay the sections (where I thought I heard it) on the other DACs, it sounds just the same. Maybe I just want the Mojo to sound smoother for the extra wedge of cash it costs, along with all the marketing guff!
• I had fun with the speakers, turned up nice and loud and I couldn't hear a difference. If I cycled quickly through the DACs via the remote switcher there was barely even a break in playback. The cheap little switcher is quite impressive.

My view as of today is that I still hear no discernible difference between these DACs when listening to music. I will continue listening into the weekend and then get out the multi-meter to check and align levels and then carry on again into next week, just to be 100% sure there is no clear difference that I can hear that would justify a blind test.

Great write up, quite a few yep, been there moments.

 

[johny_2000]

I would need to hear something like an R2R DAC or an AKM chipset, or FPGA etc, in order to hear all the additional soundstage depth, width, height... weight...

Most FPGA-based DACs are designed to color the sound. This is the philosophy these companies choose to differentiate themselves from integrated-chip based DACs such as ESS, AKM, Cirrus Logic, Rohm, Texas Instruments, etc.

PS Audio DACs: "Use of twin FPGAs rather than an off-the-shelf DAC chip provides immense processing power, resulting in a complete lack of digital glare, and allows the owner to download our free operating system updates as they are released. With every free upgrade to the operating system, new levels of resolution and audio magic will enrich and make new this remarkable D to A converter.",

Chord DACs:
"At the heart of every Chord DAC is an FPGA chip (field-programmable gate array) loaded with custom code to optimize analog conversion for timing and low noise and deliver and incredibly natural, musical sound."

Audio-GD DACs:
"The internal hardware design is fully controlled by complex software. A huge advantage is the fact the software in the FPGA can easily be upgraded offering new features or improve the performance. Such design is much flexible and future proof!
A lot people may know the even harmonic , just like the tube design, proper even harmonic especial 2x harmonic made the sound tube like and smooth.
But a lot people even some designers don't know the proper odd harmonic especial 3x harmonic can make the sound lively.
If you like to looking for the good specs, R2R 1 may not as good as the same price range ESS9028 Sigma Delta design, but its sound is analog and smooth."

and a few other companies with a similar approach. They have some fans, but it's not about transparency and high fidelity.

 

[antcollinet]

Most FPGA-based DACs are designed to color the sound. This is the philosophy these companies choose to differentiate themselves from integrated-chip based DACs such as ESS, AKM, Cirrus Logic, Rohm, Texas Instruments, etc.

PS Audio DACs: "Use of twin FPGAs rather than an off-the-shelf DAC chip provides immense processing power, resulting in a complete lack of digital glare, and allows the owner to download our free operating system updates as they are released. With every free upgrade to the operating system, new levels of resolution and audio magic will enrich and make new this remarkable D to A converter.",

Chord DACs:
"At the heart of every Chord DAC is an FPGA chip (field-programmable gate array) loaded with custom code to optimize analog conversion for timing and low noise and deliver and incredibly natural, musical sound."

Audio-GD DACs:
"The internal hardware design is fully controlled by complex software. A huge advantage is the fact the software in the FPGA can easily be upgraded offering new features or improve the performance. Such design is much flexible and future proof!
A lot people may know the even harmonic , just like the tube design, proper even harmonic especial 2x harmonic made the sound tube like and smooth.
But a lot people even some designers don't know the proper odd harmonic especial 3x harmonic can make the sound lively.
If you like to looking for the good specs, R2R 1 may not as good as the same price range ESS9028 Sigma Delta design, but its sound is analog and smooth."

and a few other companies with a similar approach. They have some fans, but it's not about transparency and high fidelity.

And yet when they are measured, they often measure as transparent (see Chord DAC review linked below). Or just perform badly with loads of noise, and/or distortion. What you are quoting there are just marketing stories, that have little correlation with the actual sound from the devices.

PS audio measures badly - but that is from noise, and from distortion coming from use of a transformer as a filter : not "colouration"

Chord DAVE Review (DAC & HP Amp)

This is a review, detailed measurements and listening tests of CHORD DAVE combination DAC and headphone amplifier. It is on kind loan from a member and costs US $14,000. Having been frustrated with the user interface of lower end Chord products with their color Morse codes, I was really looking...

www.audiosciencereview.com

Review and Measurements of PS Audio PerfectWave DirectStream DAC

This is a review and detailed measurements of the PS Audio PerfectWave DirectStream DAC. It is on kind loan from a member. The unit that I have has the network "bridge" streaming option and costs US $6,899. Without that option it costs US $5,999. The DS DAC is quite heft with a textured matt...

www.audiosciencereview.com