Blind testing two DACs - Proper Process

[johny_2000]

And yet when they are measured, they often measure as transparent (see Chord DAC review linked below). Or just perform badly with loads of noise, and/or distortion. What you are quoting there are just marketing stories, that have little correlation with the actual sound from the devices.

PS audio measures badly - but that is from noise, and from distortion coming from use of a transformer as a filter : not "colouration"

Chord DAVE Review (DAC & HP Amp)

This is a review, detailed measurements and listening tests of CHORD DAVE combination DAC and headphone amplifier. It is on kind loan from a member and costs US $14,000. Having been frustrated with the user interface of lower end Chord products with their color Morse codes, I was really looking...

www.audiosciencereview.com

Review and Measurements of PS Audio PerfectWave DirectStream DAC

This is a review and detailed measurements of the PS Audio PerfectWave DirectStream DAC. It is on kind loan from a member. The unit that I have has the network "bridge" streaming option and costs US $6,899. Without that option it costs US $5,999. The DS DAC is quite heft with a textured matt...

www.audiosciencereview.com

The Totaldac d1-six is an "R2R" DAC. Translation: "we wanted to build our own DAC because we know more than people who build DAC chips."

 

[peng]

Most FPGA-based DACs are designed to color the sound. This is the philosophy these companies choose to differentiate themselves from integrated-chip based DACs such as ESS, AKM, Cirrus Logic, Rohm, Texas Instruments, etc.

PS Audio DACs: "Use of twin FPGAs rather than an off-the-shelf DAC chip provides immense processing power, resulting in a complete lack of digital glare, and allows the owner to download our free operating system updates as they are released. With every free upgrade to the operating system, new levels of resolution and audio magic will enrich and make new this remarkable D to A converter.",

Chord DACs:
"At the heart of every Chord DAC is an FPGA chip (field-programmable gate array) loaded with custom code to optimize analog conversion for timing and low noise and deliver and incredibly natural, musical sound."

Audio-GD DACs:
"The internal hardware design is fully controlled by complex software. A huge advantage is the fact the software in the FPGA can easily be upgraded offering new features or improve the performance. Such design is much flexible and future proof!
A lot people may know the even harmonic , just like the tube design, proper even harmonic especial 2x harmonic made the sound tube like and smooth.
But a lot people even some designers don't know the proper odd harmonic especial 3x harmonic can make the sound lively.
If you like to looking for the good specs, R2R 1 may not as good as the same price range ESS9028 Sigma Delta design, but its sound is analog and smooth."

and a few other companies with a similar approach. They have some fans, but it's not about transparency and high fidelity.

Such marketing talks don't seem logical. If they did color the sound by design, they should be able to show us the color(s) that should be measurable and their claims would then be much more believable. I wouldn't mind such "colored" devices but only if they took the Nelson Pass's kind of approach with his hormonic generator.

Pass H2 Harmonic Generator - Positive Feedback

Pass H2 Harmonic Generator

positive-feedback.com

To quote something he said:

"Some purists argue that this is some sort of violation of the recording, that playback should be as accurate as possible. It's a valid opinion, but it's not my purpose to argue that point here.

My own view is that this is entertainment—not Dialysis—and the customer should use whatever makes him happy. After all, he is the guy who pays for it."

Those manufacturers bs about their designed sound quality is not logical because they don't really know what sound quality in terms of "color", "warmth", "crisp" etc their customers preferred.

As NP also stated on his website:

Audio distortion and feedback - Pass Labs

Audiophiles seem to revel in minor controversies – vinyl vs CD’s, tubes versus solid state, capacitor, wires, magic dots… and negative feedback. At one extreme, the position is that “feedback makes amplifiers perfect”. At the other extreme, “feedback is a … Continue reading →

www.passlabs.com

"Anecdotally, it appears that preferences break out roughly into a third of customers liking 2nd harmonic types, a third liking 3rd harmonic, and the remainder liking neither or both. Customers have also been known to change their mind over a period of time."

It would seem silly to pay lots of extra $ for such devices that offer the intended "color" that is however not changeable. Barring the use of NP's kind of harmonic "generators" that is user adjustable, even the use of room correction software, and/or tone controls would seem more appealing to someone who want to play with options that are users adjustable.

 

[Soniclife]

Customers have also been known to change their mind over a period of time

A good argument for doing it in software, plus you get as much or little of each flavour as you want, and for free.

 

[Mnyb]

Awesome tread .

My 2c the chip DAC’s are very similar if engineered .

R2R migth be the odd man out if to much voodoo is aplied .

FPGA chord are known for striving for real performance. PS Audio and GD the usual snake oil.

If ive found difference in chip DAC’s i would question my methodology.

If rhe R2R is different i would not be surprised depending on filter selection. NOS without filters is a broken implementation and should not be used.

 

[davmol]

Update:
Measurements have been completed with multimeter, using a 400hz 0dB test tone from Spotify (checked Hz with meter and was correct). It took a bit of trial and error to get the DACs arranged in a way that they could be closely aligned:

• The SMSL SU-1 does not have volume control but could be matched closely with the Fiio K11 R2R.
• The volume control of the Topping D50III and the Chord Mojo 2 are stepped differently to the K11 (but fairly close to each other).
• In the end, to get best alignment, I had to revert back to having 2 DACs per WiiM device, with both WiiM devices set at max volume..
  • The SU-1 and K11 R2R were connected to the Ultra, and the D50III and Mojo 2 connected to the Mini.
  • I could then adjust the pre-gain on the Ultra, reducing it by 0.7db, to more closely align across all the DACs' volume controls.
• The DACs were then each adjusted to align as closely as possible with the SU-1 (that has no volume control).
• Primary voltage measurement was at the output of the 4-in/1-out switch, because this is the input for the Topping A50III amp.
• Secondary check was the DAC output before the switch, just for info and to check loss through the switch.

Here are the initial results:

The measurements were very close but when listening I felt the K11 had a lower volume than the rest, which seemed odd given the above measurements.
I decided to investigate by measuring the voltage output from the A50III amp, at both the 6.35mm jack (using a 6.35mm to RCA cable) and the balanced 4.4mm jack (using a 4.4mm to 3.5mm balanced headphone cable). The A50III volume control was set to max to make sure I could repeat the measurements if needed.

Here are the results of the amp output (DACs are in the same order as the columns in first table):

*The high gain balanced values should be treated with caution because my multimeter could only go to 1d.p. when above 10v reading.

It seems that when the amp is in medium or high gain, the K11 R2R voltage drops significantly, which made me happy that I wasn't imagining the volume difference. To investigate some possible reasons for this voltage drop, I tried swapping around RCA cables and swapping the inputs into the switching box, but it didn't make a difference, it was only ever the K11 that had the drop in voltage after the amplifier. Any thoughts why this might be the case?

Based on the analysis of the jack voltages, I decided that to ensure the closest volume matching I should use low gain on the A50III, along with the 4.4mm balanced output, which is still plenty loud enough for me to listen to uncomfortable levels on my Hifiman Ananda and Sundara Closed.

Summary & Conclusion
The issue with the differences in volume steps across the DACs meant that I was as close as possible in my previous volume matching by ear and at that point I could not discern a difference. By separating the DACs into groups and using the pre-gain on the Ultra, I was able to close the volume gap across all 4 DACs to negligible amounts. To me, the 4 DACs sound absolutely identical, so the only decision I have left now is which DAC I'm going to keep, based on features / aesthetics etc. They're all really nice DACs with their own pros and cons, so I will ponder this question for a few more days and then report back.

 

[Blumlein 88]

The input impedance of the A50III varies with gain settings, and the in out combinations. I don't know why this would effect the FIIO unless it has higher output impedance or some other limitation into lower impedances. But you are reading the lower output with the combinations with the low input impedance.

These are left to right SE in/SE out, Bal in/SE out, SE in/Bal out, and Bal in/Bal out.

 

[Old_School_Brad]

Update:
Measurements have been completed with multimeter, using a 400hz 0dB test tone from Spotify (checked Hz with meter and was correct). It took a bit of trial and error to get the DACs arranged in a way that they could be closely aligned:

• The SMSL SU-1 does not have volume control but could be matched closely with the Fiio K11 R2R.
• The volume control of the Topping D50III and the Chord Mojo 2 are stepped differently to the K11 (but fairly close to each other).
• In the end, to get best alignment, I had to revert back to having 2 DACs per WiiM device, with both WiiM devices set at max volume..
  • The SU-1 and K11 R2R were connected to the Ultra, and the D50III and Mojo 2 connected to the Mini.
  • I could then adjust the pre-gain on the Ultra, reducing it by 0.7db, to more closely align across all the DACs' volume controls.
• The DACs were then each adjusted to align as closely as possible with the SU-1 (that has no volume control).
• Primary voltage measurement was at the output of the 4-in/1-out switch, because this is the input for the Topping A50III amp.
• Secondary check was the DAC output before the switch, just for info and to check loss through the switch.

Here are the initial results:
View attachment 404131
The measurements were very close but when listening I felt the K11 had a lower volume than the rest, which seemed odd given the above measurements.
I decided to investigate by measuring the voltage output from the A50III amp, at both the 6.35mm jack (using a 6.35mm to RCA cable) and the balanced 4.4mm jack (using a 4.4mm to 3.5mm balanced headphone cable). The A50III volume control was set to max to make sure I could repeat the measurements if needed.

Here are the results of the amp output (DACs are in the same order as the columns in first table):
View attachment 404140
*The high gain balanced values should be treated with caution because my multimeter could only go to 1d.p. when above 10v reading.

It seems that when the amp is in medium or high gain, the K11 R2R voltage drops significantly, which made me happy that I wasn't imagining the volume difference. To investigate some possible reasons for this voltage drop, I tried swapping around RCA cables and swapping the inputs into the switching box, but it didn't make a difference, it was only ever the K11 that had the drop in voltage after the amplifier. Any thoughts why this might be the case?

Based on the analysis of the jack voltages, I decided that to ensure the closest volume matching I should use low gain on the A50III, along with the 4.4mm balanced output, which is still plenty loud enough for me to listen to uncomfortable levels on my Hifiman Ananda and Sundara Closed.

Summary & Conclusion
The issue with the differences in volume steps across the DACs meant that I was as close as possible in my previous volume matching by ear and at that point I could not discern a difference. By separating the DACs into groups and using the pre-gain on the Ultra, I was able to close the volume gap across all 4 DACs to negligible amounts. To me, the 4 DACs sound absolutely identical, so the only decision I have left now is which DAC I'm going to keep, based on features / aesthetics etc. They're all really nice DACs with their own pros and cons, so I will ponder this question for a few more days and then report back.

Fantastic summary and description. This is exactly why we do this -to ground ourselves in reality and uncover these kinds of oddities.

Thank you for putting this together.

 

[antcollinet]

I don't know why this would effect the FIIO unless it has higher output impedance or some other limitation into lower impedances.

An output impedance from the Fiio of around 150ohm would make that difference, with a change of 10K down to 2.5K on the A50.

Doesn't seem that unlikely.

 

[davmol]

So it didn't take much longer to decide on what I'm keeping on the desk. I'm sticking with the A50III / D50III stack, along with the WiiM Mini for when I'm not using my PC because it covers the following bases:

• Small footprint but with a high quality metal build, which is heavy enough to stay put while plugging / unplugging headphones.
• Good range of jacks and gains, for various headphones and IEMs
• Remote control for the DAC (the two custom configuration buttons are a bonus)
• PEQ, albeit only on USB, but it is handy to assign headphone profiles to the remote and press a button to change, rather than opening an app.
• All the inputs I need, including LDAC Bluetooth, which comes in handy sometimes if using my phone or tablet for videos etc.
• The DAC display prioritises volume (in pre-amp mode), which I like because I can find constant sample rate changes on the screen to be distracting.
• The WiiM mini allows me to stream without the PC being on, and also plugs the gap of the D50's missing PEQ over Optical. I also like that when streaming I can now play / pause / change tracks without having to pick up my phone or open an app on PC.

If WiiM ever decide to release a Fiio R7 style headphone focussed version of the Ultra then I could be tempted to buy one, if it ticks all these boxes and comes in an attractive all-in-one package. I'm at least now comfortable that I'm not going to find a DAC / AMP combo that sounds better than what I have now, so I can put that question to bed and squander my money on more headphones instead ;-)

All of this testing really just means that I (any my wife) can't hear any differences between AKM / ESS / FPGA (Chord) / R2R DACs when they are level matched. If anyone else thinks they can, or just wants to check if they can, as I did here, then it was really quite inexpensive to pick up a switching box and an optical splitter, so why not give it a go and report back. I found it to be quite fun and I'm a little sad that I couldn't hear any differences at all, because I'd have quite liked to have tried some blind testing!

Also before anybody asks...yes I gave plenty of burn-in time, and have photos to prove it.....

 

[antcollinet]

Also before anybody asks...yes I gave plenty of burn-in time, and have photos to prove it.....

Nice

 

[Soniclife]

Also before anybody asks...yes I gave plenty of burn-in time, and have photos to prove it.....

Ah good, you remembered to cast out the stair-step demon, they all sound different if you forget this crucial ritual.

 

[jools]

You might like to include a $3 DAC - see the World's Cheapest DAC review. For one thing it will give your carefully made setup a chance to present an actual audible difference - and at very little cost although the extra burn-in candles will probably come to more than the DAC. And I found it a very useful comparison because the difference between something that measures that badly and an excellent DAC is surprisingly small. It kind of sounds OK... which is not surprising as vinyl sounds OK too. And it shows why the differences between moderately good DACs and state of the art are tiny/inaudible. There really is no night and day...

 

[Old_School_Brad]

You might like to include a $3 DAC - see the World's Cheapest DAC review. For one thing it will give your carefully made setup a chance to present an actual audible difference - and at very little cost although the extra burn-in candles will probably come to more than the DAC. And I found it a very useful comparison because the difference between something that measures that badly and an excellent DAC is surprisingly small. It kind of sounds OK... which is not surprising as vinyl sounds OK too. And it shows why the differences between moderately good DACs and state of the art are tiny/inaudible. There really is no night and day...

That's an excellent idea with a reference you're confident will be audible.

 

[davmol]

I have seen comments saying the DAC in the Wiim Mini is not good enough to use it as a standalone device, so maybe at some point I will test it versus the Ultra and maybe versus something like my laptop's inbuilt DAC. Might be interesting to see if I can hear any differences. I already couldn't tell a difference between three D50 and the Wiim Ultra so probably won't bother using both. Will just use the Ultra, mini and laptop.

 

[antcollinet]

I have seen comments saying the DAC in the Wiim Mini is not good enough to use it as a standalone device, so maybe at some point I will test it versus the Ultra and maybe versus something like my laptop's inbuilt DAC. Might be interesting to see if I can hear any differences. I already couldn't tell a difference between three D50 and the Wiim Ultra so probably won't bother using both. Will just use the Ultra, mini and laptop.

"bad" though the mini is - it still achieves 85+dB Sinad (0.006% THD+N). Still pretty unlikely you'll be able to hear any difference between that and a "better" DAC.

And an interesting investigation because of that.

 

[Old_School_Brad]

I have seen comments saying the DAC in the Wiim Mini is not good enough to use it as a standalone device, so maybe at some point I will test it versus the Ultra and maybe versus something like my laptop's inbuilt DAC. Might be interesting to see if I can hear any differences. I already couldn't tell a difference between three D50 and the Wiim Ultra so probably won't bother using both. Will just use the Ultra, mini and laptop.

The Wiim Mini is perfectly good -you’d have to find something much worse to be certain you could hear a difference.

 

[Boris]

That D50III/A50III duo is a very nice set up that should provide many years of listening pleasure. Enjoy!

 

[Jimi Floyd]

So it didn't take much longer to decide on what I'm keeping on the desk. I'm sticking with the A50III / D50III stack, along with the WiiM Mini for when I'm not using my PC because it covers the following bases:

• Small footprint but with a high quality metal build, which is heavy enough to stay put while plugging / unplugging headphones.
• Good range of jacks and gains, for various headphones and IEMs
• Remote control for the DAC (the two custom configuration buttons are a bonus)
• PEQ, albeit only on USB, but it is handy to assign headphone profiles to the remote and press a button to change, rather than opening an app.
• All the inputs I need, including LDAC Bluetooth, which comes in handy sometimes if using my phone or tablet for videos etc.
• The DAC display prioritises volume (in pre-amp mode), which I like because I can find constant sample rate changes on the screen to be distracting.
• The WiiM mini allows me to stream without the PC being on, and also plugs the gap of the D50's missing PEQ over Optical. I also like that when streaming I can now play / pause / change tracks without having to pick up my phone or open an app on PC.

View attachment 404332

If WiiM ever decide to release a Fiio R7 style headphone focussed version of the Ultra then I could be tempted to buy one, if it ticks all these boxes and comes in an attractive all-in-one package. I'm at least now comfortable that I'm not going to find a DAC / AMP combo that sounds better than what I have now, so I can put that question to bed and squander my money on more headphones instead ;-)

All of this testing really just means that I (any my wife) can't hear any differences between AKM / ESS / FPGA (Chord) / R2R DACs when they are level matched. If anyone else thinks they can, or just wants to check if they can, as I did here, then it was really quite inexpensive to pick up a switching box and an optical splitter, so why not give it a go and report back. I found it to be quite fun and I'm a little sad that I couldn't hear any differences at all, because I'd have quite liked to have tried some blind testing!

Also before anybody asks...yes I gave plenty of burn-in time, and have photos to prove it.....
View attachment 404333

Thanks a lot for doing your experiment and reporting here!

 

[VientoB]

If you are after proving the results to anyone else definitely go with the full double blind abx people are suggesting.

But before going to that trouble you might just get the levels matched, ensure filters used are similar in terms of impact on frequency response(this needs to be clear in point 2 of your setup). Default settings might not be the same.

just switch between the 2 using the rca switcher. If you can't tell a difference there after a bit of listening and quick switching with no gap in playback , you're not going to pick it in a properly blinded test.

From what I've gleaned on here either the difference when detectable is so huge nobody could miss it (should be clear from measurements) or there is an artifact /tell that once you hear it, you listen for each time.

Yeah I think this is a very significant point.

 

[Roland68]

So it didn't take much longer to decide on what I'm keeping on the desk. I'm sticking with the A50III / D50III stack, along with the WiiM Mini for when I'm not using my PC because it covers the following bases:

• Small footprint but with a high quality metal build, which is heavy enough to stay put while plugging / unplugging headphones.
• Good range of jacks and gains, for various headphones and IEMs
• Remote control for the DAC (the two custom configuration buttons are a bonus)
• PEQ, albeit only on USB, but it is handy to assign headphone profiles to the remote and press a button to change, rather than opening an app.
• All the inputs I need, including LDAC Bluetooth, which comes in handy sometimes if using my phone or tablet for videos etc.
• The DAC display prioritises volume (in pre-amp mode), which I like because I can find constant sample rate changes on the screen to be distracting.
• The WiiM mini allows me to stream without the PC being on, and also plugs the gap of the D50's missing PEQ over Optical. I also like that when streaming I can now play / pause / change tracks without having to pick up my phone or open an app on PC.

View attachment 404332

If WiiM ever decide to release a Fiio R7 style headphone focussed version of the Ultra then I could be tempted to buy one, if it ticks all these boxes and comes in an attractive all-in-one package. I'm at least now comfortable that I'm not going to find a DAC / AMP combo that sounds better than what I have now, so I can put that question to bed and squander my money on more headphones instead ;-)

All of this testing really just means that I (any my wife) can't hear any differences between AKM / ESS / FPGA (Chord) / R2R DACs when they are level matched. If anyone else thinks they can, or just wants to check if they can, as I did here, then it was really quite inexpensive to pick up a switching box and an optical splitter, so why not give it a go and report back. I found it to be quite fun and I'm a little sad that I couldn't hear any differences at all, because I'd have quite liked to have tried some blind testing!

Also before anybody asks...yes I gave plenty of burn-in time, and have photos to prove it.....
View attachment 404333

May I ask which pieces of music you used for the comparison?