Blind testing two DACs - Proper Process

[davmol]

Do you know how well the new DAC measures?

I haven't seen any measurements, it's quite a new device but here are the output specs for the K11 R2R's RCA, according to Fiio:

 

[davmol]

I don't know what you plan on spending for a switch box. You might make things cleaner and easier to test if you instead purchased an audio interface to record each dac. Then use abx software to test with each recording.

I already own the switching box, I use it for multiple line-in for my Wiim ultra:
Rca switching box - Amazon link

 

[Blumlein 88]

I already own the switching box, I use it for multiple line-in for my Wiim ultra:
Rca switching box - Amazon link

Well that is the way to go then.

 

[Matt_Holland]

I already own the switching box, I use it for multiple line-in for my Wiim ultra:
Rca switching box - Amazon link

By using this cheap switch you will never convince the audiophiles of the validity of the test. Even though it may be audibly transparent.

I’m looking forward to this and great job designing the test and dedication the time.

My one question was around level matching and how you are going to achieve this, but others have mentioned this too. Have you decided how you will do it?

 

[antcollinet]

1. Both DACs can do up to 384khz/32bit with usb source. What would be the best selection to minimise "you should have tested at x khz/ x bit to really hear the difference".

Redbook (16/44.1) is already pretty much transparent (audibly perfect) to almost everyone almost all of the time(except when some tricks are played). Go to 24/48 and that is absolutely transparent for all people all of the time. So use 24/96 and you have absolutely knocked it out of the park. Again though, not so you'll convince dyed in the wool "everythign matters" audiophiles. But then no blind test will, because "blind testing is too stressful so you can't hear real differences"

2. I don't really understand the expected output of NOS and OS filters on the k11 r2r DAC. Which is most suitable for this comparison / testing, or should it not matter either way?

NOS filters are a broken implementation - not compliant with the sampling theorem, and can result in audible artefacts. I assume (but don't know) that the OS filters on the k11 will work well enough to be transparent for most people. Certainly the spec sheet you have posted suggest that with a SINAD of about 72dB (Distortion dominated) it is low enough to be undetectable for almost everyone.

So if you are *wanting* to hear a difference, use the NOS Filters. However....

Or for those that would confidently say r2r DACs sounds very different ("better / more musical" etc),

No, it is not better/more musical - it is just more distorted with imaging artefacts. Even then though there is no guarantee you'll hear a difference. Most people are unable to hear distortion (klippel test) below about -45dB.

3. There are some comments about identifying and eliminating "tells", so I would assume this means everything that is not part of the music / playback e.g. "more bass" would not be a tell, but some identifiable artefact / distortion would be?

No, distortion is not a tell, since this is a weakness/difference in the DAC that you are trying to hear. Unless the distortion were coming from the test box or resulted from bad setup or interconnect for example.

Tells might be:
Inaccuracy in level matching
A difference in sound of the switch depending on which way it is switching
A timing difference in the music (if it steps back in time in one direction and forward in time in the other. If it is not possible to synchronise the sound, then it should be muted briefly during the switching.

Behaviour of the person doing the switch. This can be a real problem and the reason double blinding is used. It can be as subtle as a change in breathing, a smile or other facial expression, speed of change differences etc. One of our members significantly (and deliberately, as a demonstration) biased a test result simply by changing the angle of the pen in his jacket breast pocket depending on which device was playing.

Probably many others.

 

[Soniclife]

Thanks for all the info so far. I have a few follow up questions:

1. Both DACs can do up to 384khz/32bit with usb source. What would be the best selection to minimise "you should have tested at x khz/ x bit to really hear the difference".

2. I don't really understand the expected output of NOS and OS filters on the k11 r2r DAC. Which is most suitable for this comparison / testing, or should it not matter either way? Or for those that would confidently say r2r DACs sounds very different ("better / more musical" etc), is the biggest difference heard with the NOS or OS filter applied?

3. There are some comments about identifying and eliminating "tells", so I would assume this means everything that is not part of the music / playback e.g. "more bass" would not be a tell, but some identifiable artefact / distortion would be?

I may not hear a difference but I'm working on the basis that I will and at least I have a plan for this or future real or imagined differences.

My (limited) understanding of NOS DACs is higher sample rates get them closer to accurate, so you my want to try low and high rates.

Your new DAC might just be inaccurate enough to spot in a blind test, which is interesting. If you can hear a difference it would be good to get it measured.

A classic tell for this type of thing would be one dac having a slight delay so when switching to it you hear the delay.

 

[Wmusic]

Pay attention using blind tests: if you don’t hear any differences you cannot affirm anything.

 

[davmol]

Behaviour of the person doing the switch. This can be a real problem and the reason double blinding is used. It can be as subtle as a change in breathing, a smile or other facial expression, speed of change differences etc.

I'm fairly sure that I can set it up so that I can be one side of the door and she can be on the other. We've been together for 24 years so smiling won't be an issue

I'll go with 24/96 and OS and if I don't hear any difference then I'll try NOS.

I realise I won't convince people that don't want to be convinced, in either direction, but I'm still interested for myself. I may well not find a difference between these two DACs to justify any blind testing but that is a result in itself, considering many tell me there is a marked difference between R2R and DS DACs.

Even if I don't need to do a blind test, I'd still like to have a procedure in place that will allow me to conduct a home test with a reasonable degree of control, which I can then use to compare other dacs, and possibly amps, if necessary in the future.

 

[SIY]

Thanks for all the info so far. I have a few follow up questions:

1. Both DACs can do up to 384khz/32bit with usb source. What would be the best selection to minimise "you should have tested at x khz/ x bit to really hear the difference".

2. I don't really understand the expected output of NOS and OS filters on the k11 r2r DAC. Which is most suitable for this comparison / testing, or should it not matter either way? Or for those that would confidently say r2r DACs sounds very different ("better / more musical" etc), is the biggest difference heard with the NOS or OS filter applied?

3. There are some comments about identifying and eliminating "tells", so I would assume this means everything that is not part of the music / playback e.g. "more bass" would not be a tell, but some identifiable artefact / distortion would be?

I may not hear a difference but I'm working on the basis that I will and at least I have a plan for this or future real or imagined differences.

One other small point- assuming you did randomization properly (and that does NOT mean 'looks random to me," it's doing coin flips and if there's three heads in a row (say), that's what you go with) is timing. If your wife goes into the room and the coin flips lead to "don't swap," unplug and plug anyway. Otherwise, there's a tell when she comes out after one minute instead of three minutes.

Yes, definitively start by comparing level matched but without implementing double blinding yet. If you can't hear the difference (and the vast likelihood is that you won't), test over.

 

[Blumlein 88]

I like to use the random function in a spreadsheet to make a table of A or B beforehand instead of coin flips. Either will work.

 

[davmol]

One other small point- assuming you did randomization properly (and that does NOT mean 'looks random to me," it's doing coin flips and if there's three heads in a row (say), that's what you go with) is timing. If your wife goes into the room and the coin flips lead to "don't swap," unplug and plug anyway. Otherwise, there's a tell when she comes out after one minute instead of three minutes.

Yes, definitively start by comparing level matched but without implementing double blinding yet. If you can't hear the difference (and the vast likelihood is that you won't), test over.

There shouldn't be any cable swapping because both DACs will go through the RCA switching box, so it should just be a case of the other party involved selecting the channel that corresponds with the randomised X. I think one issue to solve will be outputting audio simultaneously to both DACs becauseI highly doubt Windows will do that without some lag or tell. I'll give the Windows dual audio output workaround a chance first, but assuming it has issues then maybe something like this will be an option:
Amazon - USB to Dual Toslink

 

[antcollinet]

We've been together for 24 years so smiling won't be an issue

 

[Matt_Holland]

There shouldn't be any cable swapping because both DACs will go through the RCA switching box, so it should just be a case of the other party involved selecting the channel that corresponds with the randomised X. I think one issue to solve will be outputting audio simultaneously to both DACs becauseI highly doubt Windows will do that without some lag or tell. I'll give the Windows dual audio output workaround a chance first, but assuming it has issues then maybe something like this will be an option:
Amazon - USB to Dual Toslink

Any possibility of getting Roon and two compatible streamers? That way you can sync the two streamers for the same content. Roon may also have fine enough volume control to achieve the level matching. I also wondered whether Roon could be used to achieve better double blind conditions because the assistant could remain outside of the the listening room at all times. Does the switch box work from its remote control at distance? If not maybe there’s a way to feed both DACs into a mixer and apply a mute in Roon to the stream/DAC not under test which would provide completely seamless switching and allow the assistant to do everything outside of the listening room.

 

[antcollinet]

There shouldn't be any cable swapping because both DACs will go through the RCA switching box, so it should just be a case of the other party involved selecting the channel that corresponds with the randomised X. I think one issue to solve will be outputting audio simultaneously to both DACs becauseI highly doubt Windows will do that without some lag or tell. I'll give the Windows dual audio output workaround a chance first, but assuming it has issues then maybe something like this will be an option:
Amazon - USB to Dual Toslink

That might work - but if the buffers in the two devices are different sizes, that will lead to different lag. In that case, higher sample rates will reduce the lag and the lag difference.

 

[BDWoody]

Any possibility of getting Roon and two compatible streamers?

When I went through this for myself, I used two CCA pucks sync'd through Roon with toslink out into two DACs. Worked great for me.

 

[davmol]

The used fiio r2r dac I picked up on eBay is coming with a fiio sr11 streamer which I'm going to sell on but I think is roon ready, and I have a wiim ultra, so I can investigate the Roon option, thanks.

@Matt_Holland yes the switch box is infrared and works from distance.

 

[xaviescacs]

Also take into account that taking the same test more than once raises the probability of getting a significant result just by chance, so you have to correct the p-values accordingly. I know, it's frustrating, but otherwise is just gambling. Assuming you don't learn anything from test to test, which keeps each test independent from each other, this could be the way to go.

 

[danadam]

An easy approach is to flip a coin. She can do that to get an unknown-random sequence in advance, or she can do it during the test.

I like to use the random function in a spreadsheet to make a table of A or B beforehand instead of coin flips. Either will work.

Random sequence of "0"s and "1"s:

RANDOM.ORG - Integer Generator

This page allows you to generate random integers using true randomness, which for many purposes is better than the pseudo-random number algorithms typically used in computer programs.

www.random.org

(refresh that page to generate a new sequence)

However, I was told in a forum that the reason I don't hear differences is ...

and am of the opinion that there is no audible difference if the DACs are well designed and not intentionally coloured.

If you don't hear the difference in the test you will be told in that other forum that you have reverse confirmation bias

 

[DanielT]

My suggestion. Test first without worrying too much about level matching. IF , probably not but IF you at this first round of tests - with enough tests so you can rule out chance - dot them then move on to round 2 in which you then have to level match.

Regardless of when, I say when not if, you fail to separate them, there will be those who find reasons why this is the case. It will be blamed on bad speakers, cables, switches, that you didn't make an effort to listen for differences and so on. Be aware of that if you post your results on a subjective audiophile site.

You might want to try a quick non-blind test first. If you don't hear a difference there's not much point in blind testing. If you hear a difference, or if you think you might be hearing a difference, a blind ABX test can help to determine if you can statistically-reliably hear a difference. And that's the goal of an ABX test... To determine if you can hear a difference. Then if you can hear a difference it's usually a matter of opinion which one sounds best (assuming no obvious defect like noise or distortion).

If you go-ahead with the blind test it's important that you can't tell if it's switched or not. She may have to mute whenever she switches, or doesn't switch.


An easy approach is to flip a coin. She can do that to get an unknown-random sequence in advance, or she can do it during the test.

As DBWoody says.... In an ideal ABX test you get get to ask for A, B, or X, as many times as you wish and you can listen as long as you wish before making a choice for each trial..

Here is an ABX probability chart. But... I sort-of feel like if you can't get it right 100% of the time you're probably not going to know which one you're listening to if you listen to one on Monday and the other on Tuesday, and it's not enough of a difference to worry about.

Pretty much what I'm into. Test first. It can be a "quick and dirty" blind test, which to begin with does not follow the blind test protocol to the letter.
In any case it is better to start somewhere with practical tests. The practical tests can then, if necessary, be improved and fine-tuned.

 

[Soniclife]

There shouldn't be any cable swapping because both DACs will go through the RCA switching box, so it should just be a case of the other party involved selecting the channel that corresponds with the randomised X.

That's going to make the test very hard, rapid switching between 2 choices as you listen makes small differences much easier to hear reliably.