Blind testing two DACs - Proper Process

[davmol]

Hi, I have a Topping D50 III DAC and have just ordered a used Fiio K11 R2R DAC. I have previously not heard a difference between DACs, in the form of my D50, a Qudelix 5k, the Wiim Ultra internal and a Geshelli J2 and am of the opinion that there is no audible difference if the DACs are well designed and not intentionally coloured. However, I was told in a forum that the reason I don't hear differences is that they all use the same chipset and I would need to hear something like an R2R DAC or an AKM chipset, or FPGA etc, in order to hear all the additional soundstage depth, width, height... weight... BMI? that I am missing with my cheap DACs.

Even though I am not a believer, I do like to confirm things myself and and the K11 R2R is so cheap that I figured why not test for fun and sell it on. I want to check the process for a good blind test. My setup will be as follows:

• K11 R2R using RCA out into a remote control RCA switching box.
• D50 III using RCA out into RCA switching box (not using balanced out because K11 does not have that, even though shouldn't make a difference).
• Switching box into my Topping A50III amplifier.
• Test headphones (with no EQ) will be HiFiMan Ananda and Beyerdynamic DT1990 Pro.

My outline plan for the process is as follows:

1. Power supplied to all devices from the same GAN USB plug and using same USB cables of same length (Amazon Basics).
2. Ensure DACs are in base settings with no EQ etc and using same sample / bit rates.
3. Same RCA cables will be used with same length (Amazon Basics).
4. Play a test tone / white noise and measure / align / record each DAC output voltage at end of RCA cable, using a multimeter.
5. Plug both sets of RCA cables into the switching box with use another Amazon Basics RCA cable from the output.
6. Measure and confirm same output voltage from the switching box when using each source input (align and record again if needed).
7. Connect up to the A50III and use single ended 6.35mm jack (Beyer is not balanced and I want to try both headphones).
8. Close cabinet doors to hide all equipment and sit facing away from any equipment / away from eyeline of support crew (wife..)
9. Get my wife to assign each DAC input as A or B and draw up an order of testing (20 samples, e.g. A-A-A-B-A-B-B-A-A-B-B etc)?
10. She will control the remote for the switching box and I will have no sight of any equipment.
11. I will guess D50 or K11 and she will record it on the sheet. I plan to repeat the test 3 times using for the Ananda, and 3 times for the DT1990, maybe a day apart.

So how does this process look to the experts here on blind testing? Is there anything I can improve and how might I deal with the settings on each DAC to make sure I am testing apples with apples i.e. how do things like NOS and OS modes on the R2R affect the testing, or any of the filters on the D50 III? I will select tracks from the ASR critical listening playlist on Spotify.

It will probably be a week or two before I can conduct the testing and my wife couldn't give a toss about audio so I might need to sub in a friend instead if she really isn't interested

 

[Purité Audio]

I would only use one pair of headphones, but level matched and unsighted looks all good.
I found my wife also quickly tired of my unsighted comparisons.
What I might do in the future is compare a ‘ blameless’ dac with some progressively poorer examples to try and reach some conclusions as to how much distortion ( and which type) has to be added before I can actually discern a difference.
Keith

 

[olieb]

I am not an expert on blind testing, but I would have two questions:
How are you going to match levels if the outputs of the DACs are not matching already?
Did you think about doing an ABX test? (If you listen to e.g. A-A-A-B-A-B-B-A-A-B-B, even in the situation that you do perceive a difference, how are you supposed to know which is which?)
EDIT:
to be more precise: After listening to "A" (from above) what would be the question and what could be your answer?
But maybe I did not get it and the sequence already specifies the Xs from an ABX test?
In any case: Amazing effort.

 

[BDWoody]

So how does this process look to the experts here on blind testing? Is there anything I can improve

Nice to see someone want to put their own ears to the test, so definitely looking forward to hearing about how it goes.

I'm not an expert by any means, but do know that the 'double blind' part could use a little attention. If the person doing the switching knows what they are switching, you are open to the Clever Hans issue. Not sure the best way to do that, but if you want to tighten it up, that's one place. Maybe mix up the cables ends so neither of you knows until afterwards what was plugged into A or B, but others will likely have better input.

I'd also suggest an AB/X protocol.

 

[JSmith]

JSmith

 

[davmol]

How are you going to match levels if the outputs of the DACs are not matching already?

Is this not the purpose of measuring and aligning output voltage using the multimeter, or you do mean something else?

 

[davmol]

I'll watch that video tonight. From ab/x perspective that means I will listen to a and b and then hear a random one and try to identify it?

 

[jools]

You won't be able to hear all that wonderful soundstage and PRAT through Amazon BASIC cables.... (Joking, for the avoidance of doubt - but that's what 'they' will say.)

 

[JeremyFife]

Sounds interesting, it's great to see tests like this being carried out (I'm too lazy / could never convince my wife to take part!)

Do your DACs offer different filter settings? Make sure that you have recorded which setting is in use. Are the default settings on each DAC equivalent? If not perhaps select equivalent settings.

 

[BDWoody]

I'll watch that video tonight. From ab/x perspective that means I will listen to a and b and then hear a random one and try to identify it?

You'll have two known samples, A and B, and one unknown sample that you will try to identify as either A or B, so yes, pretty much what you said.

You can switch between them all as much as you like for each trial.

 

[Jimbob54]

If you are after proving the results to anyone else definitely go with the full double blind abx people are suggesting.

But before going to that trouble you might just get the levels matched, ensure filters used are similar in terms of impact on frequency response(this needs to be clear in point 2 of your setup). Default settings might not be the same.

just switch between the 2 using the rca switcher. If you can't tell a difference there after a bit of listening and quick switching with no gap in playback , you're not going to pick it in a properly blinded test.

From what I've gleaned on here either the difference when detectable is so huge nobody could miss it (should be clear from measurements) or there is an artifact /tell that once you hear it, you listen for each time.

 

[Soniclife]

Nice to see someone want to put their own ears to the test, so definitely looking forward to hearing about how it goes.

I'm not an expert by any means, but do know that the 'double blind' part could use a little attention. If the person doing the switching knows what they are switching, you are open to the Clever Hans issue. Not sure the best way to do that, but if you want to tighten it up, that's one place. Maybe mix up the cables ends so neither of you knows until afterwards what was plugged into A or B, but others will likely have better input.

I'd also suggest an AB/X protocol.

The double blind aspect is the area that I thought needed improvement.

I would suggest getting everything set up and level matched, then test swapping back and forth sighted, try to identify what's different. Also look for something that's a tell that's not pure sound quality, if you spot something you will need to address it.

When your are ready to test get your wife to assign the cables randomly, note the config, close the cabinet then leave the room, ideally there is no contact between you till after the test, if your house allows that easily. When you have guessed she swaps again randomly, a coin toss should work.

Do you know how well the new DAC measures?

 

[olieb]

Is this not the purpose of measuring and aligning output voltage using the multimeter, or you do mean something else?

Sure, but if you measure a difference you have to adjust levels ("aligning"). Matching should be to 0.1dB or better. How will you do that?
In the digital domain? -> change of the signal (not "bit perfect" anymore)
Analog attenuator? -> difference in signal chain

 

[Jimbob54]

Sure, but if you measure a difference you have to adjust levels ("aligning"). Matching should be to 0.1dB or better. How will you do that?
In the digital domain? -> change of the signal (not "bit perfect" anymore)
Analog attenuator? -> difference in signal chain

That's what you have to do for any levelling though. Ideally both dacs are close enough in output levels at max and can attenuate digitally so one can be nudged a little to match the other .

 

[audio_tony]

Play a test tone / white noise and measure / align / record each DAC output voltage at end of RCA cable, using a multimeter.

Multimeters typically have a limited frequency response so using white noise is not recommended.

Just use a single tone - 1kHz is likely optimal as most decent multimeters are still fairly accurate at that frequency on the AC range.

If your measurements prove to be inconsistent, then switch to a lower frequency like 500Hz.

 

[Sokel]

That's what you have to do for any levelling though. Ideally both dacs are close enough in output levels at max and can attenuate digitally so one can be nudged a little to match the other .

That was always a problem as theoretically (since that's the root all the advises go) the attenuated DAC is not judged fairly just by the reduced SNR alone.
Now,imagine that there are two DACs,VC is done digital so it will be surely at the -20s or -30s and one of them has an ESS hump which is right on the money at that level and it reaches distortion at the -60dB ballpark or even higher.

IMO,such a test cannot be fair like that.Maybe both at near max through a pre-amp?

 

[antcollinet]

change of the signal (not "bit perfect" anymore)

Change to non bit perfect not audible in any case. If there is a doubt go to digital attenuation on both.

 

[DVDdoug]

and am of the opinion that there is no audible difference if the DACs are well designed and not intentionally coloured.

You might want to try a quick non-blind test first. If you don't hear a difference there's not much point in blind testing. If you hear a difference, or if you think you might be hearing a difference, a blind ABX test can help to determine if you can statistically-reliably hear a difference. And that's the goal of an ABX test... To determine if you can hear a difference. Then if you can hear a difference it's usually a matter of opinion which one sounds best (assuming no obvious defect like noise or distortion).

If you go-ahead with the blind test it's important that you can't tell if it's switched or not. She may have to mute whenever she switches, or doesn't switch.

Get my wife to assign each DAC input as A or B and draw up an order of testing (20 samples, e.g. A-A-A-B-A-B-B-A-A-B-B etc)?

An easy approach is to flip a coin. She can do that to get an unknown-random sequence in advance, or she can do it during the test.

As DBWoody says.... In an ideal ABX test you get get to ask for A, B, or X, as many times as you wish and you can listen as long as you wish before making a choice for each trial..

Here is an ABX probability chart. But... I sort-of feel like if you can't get it right 100% of the time you're probably not going to know which one you're listening to if you listen to one on Monday and the other on Tuesday, and it's not enough of a difference to worry about.

 

[davmol]

Thanks for all the info so far. I have a few follow up questions:

1. Both DACs can do up to 384khz/32bit with usb source. What would be the best selection to minimise "you should have tested at x khz/ x bit to really hear the difference".

2. I don't really understand the expected output of NOS and OS filters on the k11 r2r DAC. Which is most suitable for this comparison / testing, or should it not matter either way? Or for those that would confidently say r2r DACs sounds very different ("better / more musical" etc), is the biggest difference heard with the NOS or OS filter applied?

3. There are some comments about identifying and eliminating "tells", so I would assume this means everything that is not part of the music / playback e.g. "more bass" would not be a tell, but some identifiable artefact / distortion would be?

I may not hear a difference but I'm working on the basis that I will and at least I have a plan for this or future real or imagined differences.

 

[Blumlein 88]

I don't know what you plan on spending for a switch box. You might make things cleaner and easier to test if you instead purchased an audio interface to record each dac. Then use abx software to test with each recording.