How Should I A/B Test DACs?

[tejman]

I am fairly new to the enthusiast audio community and have been doing a lot of reading on ASR.

Upgrading up my desktop DAC/Amp setup and I want to see what differences I can actually hear. I want to do some testing. I will be using my own equipment and borrowing from friends. Here is what I have to work with:

My hardware:
- DT770 Pro 250ohm (headphones)
- Etymotic ERP4-T (earphones)
- SMSL AD18 (dac/amp)
- FiiO E17K (dac/amp)
- PC soundcard (dac via 3.5mm line out)
- JDS Labs Atom (amp)
- Micca MB42x (pair)
- Micca COVO-s (pair)
- All the needed analog and digital/optical cables

Borrowed hardware:
- SMSL Sanskrit 10th mk2 (dac)
- Topping D50s (dac)
- Topping E30 (dac)

Stuff I may need for testing?:
- Multimeter
- Analog RCA switch
- Optical splitter

I understand I need to focus on testing one aspect of the audio chain at a time. Because I already have headphones and a great amp, im focusing on DAC testing. I will prolly end up testing amps, headphones and speakers down the road.

This thread seems like a good template to follow. Except, PC > dacA + dacB (via 2 USB cable or optical splitter) > RCA switch (optional?) > Amp > HP

1) Major difference, I don't plan on doing a "blind" test cuz I want to do this on my own on my own schedule. (I may be able to have a roommate randomly plug 2 DACs into a switch but I would still be the one doing the switching and will prolly be able to easily see which is which during the test).

2) Should I use a single type of input to my DACs (i.e. USB)? I would use optical normally but since my portable DAC only has USB maybe its better to be consistent? Also, correct me if im wrong, but USB would obviate the need for splitting the optical signal coming out of my PC (allowing me to use a switch to change inputs with one button).

3) Do I need a switch if I can't do fully "blind" testing? Should I just change it myself? I figure a switch would make it easy to switch DACs mid song, but im not sure if that is better than just listening to a song or a short playlist completely and then switching DACs. I would be using RCA cables and/or 3.5mm to RCA cables for input and RCA for the output to a single amp.

4) One of my main concerns is level matching the output of the DACs. Can I get away with an ear test and pink noise? If so, how? If not, what is the procedure for doing this with a multimeter? Depending on the cost for the appropriate meter, I would consider doing this.

5) Source audio? Working on ~5 track playlist. I usually listen to music via Spotify (premium), but thinking I should download some high fidelity versions of the songs I choose. Any reputable sources or specific quality to look for?

Any other feedback in general would be great! Maybe this is not worthwhile if not doing properly/blind? Maybe there is a much simpler/pragmatic way to test?

 

[Taddpole]

Maybe this is not worthwhile if not doing properly/blind?

Agreed

 

[tejman]

Agreed

Fair enough. Ignoring the more scientific approach, are there any guildelines to casually testing/comparing 2 dacs (or amps/hp/spk for that matter)? Other than keeping other hardware the same of course. Like leveling output of each dac/amp to compares equal volumes?

 

[Doodski]

You need a method to match the signals coming out of the DACs so they are at the same levels. A switch box is nice but that doesn't level match.

 

[tejman]

You need a method to match the signals coming out of the DACs so they are at the same levels. A switch box is nice but that doesn't level match.

Gotcha. From what I have read, the options seem to be ear test with pink noise, test signal measured with SPL near headphones, or test signal measured with multimeter. The latter being the most accurate.

I don't have a SPL or multimeter but would rather invest in a multimeter if needed. But should I even bother and just do an ear test?

 

[Doodski]

Gotcha. From what I have read, the options seem to be ear test with pink noise, test signal measured with SPL near headphones, or test signal measured with multimeter. The latter being the most accurate.

I don't have a SPL or multimeter but would rather invest in a multimeter if needed. But should I even bother and just do an ear test?

You can't get it accurately matched by ear. You'll need to measure and adjust. If you are going to put in the effort get a meter.

 

[sigbergaudio]

If you don't do it blind, ABX style it will be impossible to not be biased by visual cues, conscious or unconscious assumptions about which is best, etc - so it's not really worth the effort.

And if you actually are able to set up an ABX test, it's almost certain that you won't be able to reliably tell the difference. Does that mean there isn't an audible difference? Opinions differ. But at least you won't be able to determine that through an ABX-test if you've level matched and both DACs are functioning properly and/or the manufacturer didn't intentionally introduce significant coloring of the sound.

So the point is, you won't be able to reach a conclusion about which is better that way (if that is indeed your goal).

 

[mnemonix]

Fair enough. Ignoring the more scientific approach, are there any guildelines to casually testing/comparing 2 dacs (or amps/hp/spk for that matter)?

You are choosing to ignore the single most important factor affecting whether the results of your listening tests have any worth whatsoever since you will literally never know the reason you are hearing or not hearing what you think you are. I wouldn't bother wasting your time if I were you.

 

[tejman]

Ok so I gather from y'alls responses that any sort of "test" short of a well set up ABX test is not a worthwhile endeavor? I am prepared to do the all the required set up but I don't think I can achieve a "blind" test right now. Did I understand correctly?

If that is the case, is the advice you would give to the prospective DAC buyer, just go with objective reviews online like this site instead of comparing products themselves? Does this also apply to other hardware like amps, headphones and speakers?

I am happy to say this type of subjective assessment is just not feasible.

I felt as though there was a significant difference listening to music from my AD18 headphone output vs the E17K portable dac/amp. I bought the Atom amp (not used yet) and am hoping to see if it provides a similar improvement. I know the DAC in each device can change things too. Hence why I started down this path.

 

[tejman]

So the point is, you won't be able to reach a conclusion about which is better that way (if that is indeed your goal).

Are you saying this even if one were to ABX testing? If so, then what reason is there to buy any device over another? I assume there must be some minimum of quality where you can hear obvious differences. But maybe my anecdotal experience is leading me astray.

 

[Doodski]

Ok so I gather from y'alls responses that any sort of "test" short of a well set up ABX test is not a worthwhile endeavor? I am prepared to do the all the required set up but I don't think I can achieve a "blind" test right now. Did I understand correctly?

Yes.

If that is the case, is the advice you would give to the prospective DAC buyer, just go with objective reviews online like this site instead of comparing products themselves? Does this also apply to other hardware like amps, headphones and speakers?

For DACs the measurements here at ASR are a good basis to choose your DAC.
For amps it is the same.
Headphones and speakers you will need to hear them but the specs and measurements will give you a good basis.

 

[Doodski]

If so, then what reason is there to buy any device over another?

For better specs and cleaner output that is closer to the original waveform without adding anything or taking away from the original source material.

 

[tejman]

Thanks so much for your help!

Headphones and speakers you will need to hear them but the specs and measurements will give you a good basis

So in these cases, do the same principles apply with regards to needing A/B testing? or is it that headphones have differences that are more perceptible?

Do you think the obvious difference that I seem to hear between my AD18 and E17k (same headphones) is placebo? Or is there a certain threshold of quality after which you can't tell the difference anymore?

 

[tejman]

For better specs and cleaner output that is closer to the original waveform without adding anything or taking away from the original source material.

And this can all/mostly be determined through objective measurements (like the ones on this forum) rather than subjective listening? And are headphones different in this sense?

 

[Doodski]

So in these cases, do the same principles apply with regards to needing A/B testing? or is it that headphones have differences that are more perceptible?

Yes, you need to make the speakers have the same output volume. The same for headphones.

Do you think the obvious difference that I seem to hear between my AD18 and E17k (same headphones) is placebo? Or is there a certain threshold of quality after which you can't tell the difference anymore?

Transducers always sound different. When you hear a difference it is real and you'll know when it is better in most cases. The differences are usually obvious.

 

[Doodski]

And this can all/mostly be determined through objective measurements (like the ones on this forum) rather than subjective listening? And are headphones different in this sense?

Yes, you can choose a DAC based on measurements.
Headphones and speakers are different because there are so many variables. But generally if you look at specs and measurements you can see what the variations are in controlled settings. Use your ears for transducers.

 

[tejman]

Transducers always sound different. When you hear a difference it is real and you'll know when it is better in most cases. The differences are usually obvious.

Im a little confused here. I thought the transducers were the parts actually create the air pressure. If so, wouldn't those be confined to the headphones? if I am trying the same headphones with two different dac/amp outputs, how would I hear difference in the transducers?

Again, thank you for taking the time to educate a newbie!

 

[Doodski]

Im a little confused here. I thought the transducers were the parts actually create the air pressure. If so, wouldn't those be confined to the headphones? if I am trying the same headphones with two different dac/amp outputs, how would I hear difference in the transducers?

Again, thank you for taking the time to educate a newbie!

I guess I misread your comment and thought you where comparing transducers too.

 

[Killingbeans]

If that is the case, is the advice you would give to the prospective DAC buyer, just go with objective reviews online like this site instead of comparing products themselves?

Forget the reviews. Look at the data. If it doesn't show anything being horribly broken, shift your focus to features and aesthetics. And if they satisfy you... and you really, really need a DAC, go for it!

Does this also apply to other hardware like amps, headphones and speakers?

Amps, yes. Headphones+heads and speakers+rooms is a whole other can of worms. Not to mention the wrecking ball called 'recordings'

I assume there must be some minimum of quality where you can hear obvious differences.

Yes, but the bar is set way lower than most people are willing to admit.

 

[FrantzM]

Hi
Tejman

From your post I could not fully grasp which DACs you were going to A/B... If the DACs you are trying to test are these:
- SMSL Sanskrit 10th mk2 (dac)
- Topping D50s (dac)
- Topping E30 (dac)
It will be likely educational. There won't (can't ) be any differences in sound between these if you go blind and match the levels. Focus on features and a beefy amp .. and there again, same. Transducers (Headphones, speakers) do sound different, no doubt but electronics? ... Don't worry too much about those. a $99 amps such the JDS Labs Atom is all you need.. if your DAC doesn;t have an amp....
Read this post