How Should I A/B Test Headphone Amps?

[Bob-23]

Congratulations! When ab-testing devices I also found no differences or, at best, very minor ones where I thought there were big ones...

Sometimes it's not that easy dealing with results like these, though.

For some years I've been introducing a friend's son (aged 13 now) into the basics of soldering and amplifier electronics, mainly in the winter time. (I'm also a hobbyist, not a professional.) Having changed 1 1/2 ago from cds to computer audio, I bought a dac. Then, in the following soldering session, I showed to my friend's son 'proudly' the new dac and told him, full of conviction, what a great improvement in sound quality that new dac meant, compared with the 'typical rubbish' computer dac.

It was only about half a year ago, that I decided to ab-test the computer dac vs. my new one (with Phil Allisons AB-Switch-Box which I had built longer ago). I was kind of 'shocked': I couldn't hear any difference...

...I told my friend's son, that I was wrong. Telling him was not really an easy exercise for me, but it might have been an important lesson for him... Convictions have to stand the test, and if they don't, they have to be dropped, even if it's a bit painful.

I have 2 questions:
Does your Box switch seamlessly, without audible interruption?
And, if it's not to personal: What age group are your participants in the test?

 

[hutt132]

Congratulations! When ab-testing devices I also found no differences or, at best, very minor ones where I thought there were big ones...

Sometimes it's not that easy dealing with results like these, though.

For some years I've been introducing a friend's son (aged 13 now) into the basics of soldering and amplifier electronics, mainly in the winter time. (I'm also a hobbyist, not a professional.) Having changed 1 1/2 ago from cds to computer audio, I bought a dac. Then, in the following soldering session, I showed to my friend's son 'proudly' the new dac and told him, full of conviction, what a great improvement in sound quality that new dac meant, compared with the 'typical rubbish' computer dac.

It was only about half a year ago, that I decided to ab-test the computer dac vs. my new one (with Phil Allisons AB-Switch-Box which I had built longer ago). I was kind of 'shocked': I couldn't hear any difference...

...I told my friend's son, that I was wrong. Telling him was not really an easy exercise for me, but it might have been an important lesson for him... Convictions have to stand the test, and if they don't, they have to be dropped, even if it's a bit painful.

I have 2 questions:
Does your Box switch seamlessly, without audible interruption?
And, if it's not to personal: What age group are your participants in the test?

The switch box I got isn't completely seamless, as there's a very faint pop when pushing the button. It's mostly audible during quiet parts/songs, but I couldn't hear it when there was a lot going on in a song.
The age group is 4 in mid 20s and 1 in 50s. I had another friend come over today to compare the Jot to the Vali tube amp. She was faced away while I was setting it up and during the experiment. During the first few minutes she didn't even know I was switching it. Then she tapped my finger when she wanted me to switch it and couldn't tell a difference either.

I've been looking at the Phil Allison design you linked, and I want to make two boxes now (one with rca and one with 3.5mm) to mitigate any audible interruptions.
Do you have links to relays and a push-on/push-off switch that you recommend? I have a tiny bit of soldering experience, but I'm not sure what the best place is to buy any parts or solderable breadboards.

 

[flipflop]

Got the 3.5mm switcher today, had a few of my audiophile buddies come over, and proceeded to question our reality for the rest of the day.

I'll start off by describing my A/B testing setup. I played flac to my Modi Multibit via toslink, which fed two headphone amps at the same time with RCA splitters. Both amps' outputs fed a MC102 3.5mm switcher that switched between each output to feed my HD58X headphones. I used a Fluke 117 multimeter to level match each amp to 0.08v while playing a 400Hz test tone with the headphones connected.

A variety of genres were tested such a pop, electronic, country, and classical. I sat with my eyes closed while my friend clicked the button to switch the inputs to eliminate bias, so I had no idea which amp was which and I was not picking up visual queues from his body language.

Amplifiers tested were the Schiit Jotunheim, JDS Atom, JDS O2, Massdrop O2, and Schiit Vali 2 (with 6BZ7 tube and Amperex ECC88/6DJ8 Orange Globe Logo O-Getter tube).

Comparison Results Prior to the A/B Setup
My initial impressions with each amp when I was comparing them by only unplugging my headphones from one amp and plugging them into the other as fast as possible were:
Atom sounded more bright/trebly than the rest
JDS O2 sounded more full/slightly more textured than the Jotunheim
Vali 2 sounded more textured than the Jotunheim
Orange Globe tube sounded better than the 6BZ7

Results After the A/B Setup
All the amplifiers sounded the exact same. Tube amp and all. Same quality. Same soundstage. I could not tell when my friend was pressing the button besides when there was a very faint pop sometimes when the signal was switched. I had to open my eyes to make sure he was pressing the button to switch them because I didn't believe it. My friends had the same results, they couldn't hear any differences. I even unplugged one cable from each amp to ensure the switcher was actually working.

While writing this I still can't seem to believe it. I don't know how I was so sure I perceived a difference when I compared the amps by manually plugging my headphones into each. The tube amp is what shocked me the most. There was zero perceived difference between every amp tested.

View attachment 30155

Good job on actually going all the way.
A negative result can be an eye-opener, but positive results are usually more fun. According to Schiit's own specs, the Vali 2 has an output impedance of 5.8 ohms in high gain mode. If you ABX it like that against another one of your amps, using low impedance headphones with an impedance swing, you might be able to hear a difference.

 

[Bob-23]

I've been looking at the Phil Allison design you linked, and I want to make two boxes now (one with rca and one with 3.5mm) to mitigate any audible interruptions.
Do you have links to relays and a push-on/push-off switch that you recommend? I have a tiny bit of soldering experience, but I'm not sure what the best place is to buy any parts or solderable breadboards.

I'm still lookin' for an ideal push-on/push-off switch, too, seems to be difficult, Conrad doesn't have one, you can either hear the respective position, as they sound different, or they are pushed-in/off in different (distinguishable) heights and/or they are much to loud at all, so that you hear that it's just being switched. Provisionally I built a normal switch into the little box... I haven't tested Segor's switches yet (Conrad's 10 minutes from my house, Segor takes an hour)... I like these sturdy boards, usually get them in a bit larger quantities via amazon.

The LED and the 1k resistor are not necessary, you can leave them out.

You need a 12 V DC-powersupply, I put one on the list, but maybe you have one at home, they are quite common.

Do you live in Europe? I could recommend Segor here in Berlin.

Here's a small list of the parts you asked for (for the headamp-version):

board:
0406/EP (1,40 Euro)
https://www.segor.de/#Q=0406/EP&M=1

relay:
FRS 1H12 (1,70 Euro)
https://www.segor.de/#Q=FRS1H12&M=1

RH-12 (4,90 Euro/golden contacts)
https://www.segor.de/#Q=RH-12V&M=1

diode:
1N4007 (10pcs. 0,40 Euro)
https://www.segor.de/#Q=1N4007&M=1

switch:
MS-244 (2,80 Euro)
https://www.segor.de/#Q=MS-244&M=1

powersupply:
SNT12V/0,6A (7,90 Euro)
https://www.segor.de/#Q=SNT12V%

https://www.amazon.de/Doppelseitig-Prototype-Lochrasterplatte-Verschiedene-Kompatibel/dp/B0734XYJPM

It's a perfect entry-level project! Have fun with it!

 

[Bob-23]

I corrected the link for the boards...

There's also Mouser, Farnell, Reichelt, Conrad and others where you can get the parts...

The relays have to be SPDT (single pole double throw/'1xUm' in German), 12V, 1A or more, and the (signal-)voltage you want to switch +, say, 50%. You may use relays with golden contacts, they don't oxydize, but are not really necessary unless you're in a very humid climate. I added a link, nonetheless.

Diode 1N4007 substitutes 1N4004.

Good luck!

 

[solderdude]

@hutt132 Now that you have seen all the anti-A-B arguments (you know where) which are typical for the sighted listening believers because all of the reasons why they believe A-B is a flawed testing method were mentioned, I will present you with a time consuming but worthwhile test method to debunk what they say. It involves statistical relevance and the aid of a second person and several weeks.

Use the setup you have and the 2 post polarizing amps you have, say the Vali2 and O2 or say the Jotunheim and A2 or Atom. The ones 'they' say sound very different from each other during prolonged and sighted listening tests.

Ask someone to switch the amps every now and then and keep notes on which they he/she set the switch in position A or B.
It is O.K. if the switch was only used 2 or 3 times over the coarse of a few weeks or every few days or so or even when you stepped out during an 'evaluation' and secretly the person switched (without you knowing). That really doesn't matter.

You just need to relax and listen to music. and write down date/time and which you think is playing every time you have been listening which again does not have to be every day. You can use different listening levels (use PC for volume control) and at any moment that is convenient for you.

When you have reached at least 20 'listening sessions (the more the better) compare notes for statistical relevance.

I did such a test once because of the arguments found on the web which are exactly like the ones in the before mentioned 'place'.

Have fun ... it is very educational and is liberating you of all internet opinions. It saves you time and frustration having to argue about it because you are one of the few that really knows and actually tested it.

 

[hutt132]

Quick update: My friend brought over some of his new equipment and we set up the a/b test again for various setups. The levels on each setup were matched to 0.172v. Each setup fed a Nobsound Little Bear MC102 a/b switcher connected to HD6XX headphones. Various genres of music were tested.

Setups:
1) Schiit Modi Multibit (pre-firmware update) -> Schiit Jotunheim
2) Asus Xonar Essence STX Soundcard (Analog Out) -> Schiit Jotunheim
3) Topping D50s -> Objective 2

Setups 1 and 2 were compared to setup 3. One person was listening with their eyes closed while the other was pressing the button on the a/b switcher. Both my friend and I could hear no difference when blind a/b testing.

 

[amirm]

For headphone amps, I suggest two types of content:

1. With lots of bass.

2. Something with high frequencies. And/or one track with some bass simultaneously but make sure there is not a ton of instruments playing at once.

For #1, listen for whether the bass is as strong, as clear and undistorted. In extreme cases some amps actually mute on bass notes! You need difficult loads to make this show issues and elevated volume. In extreme cases and as a "control" you can put in ear plugs and listen since that all but filters low frequencies.

For #2, focus on how bright the highs become. Try to ignore everything else. Again you would need to play at higher volumes and difficult loads.

The HD6XX should be a difficult load for some amps that are low on output voltage.

If you listen otherwise, our outcome is most likely correct. It is lack of power for the load combined with content that pushes that which determines differences.

 

[gustavjung]

Hi. Thanks for sharing this research!
I want to do the same comparing 2 DACs, Fiio K3 and Topping NX4 with HD598SE loaded.
Will the following comutation be ok?
A) PC - > Fiio K3 connected by USB - > AB switcher - > Headphones
B) Laptop - > Topping NX4 connected by USB - > AB switcher - > Headphones
So there will be 2 different sources(pc and laptop)
Or is it possible to connect DACs to 1 PC simultaneously?
Do I need to do level match after AB switcher?

 

[amirm]

Or is it possible to connect DACs to 1 PC simultaneously?

You can certainly connect both at once and Windows will recognize them. The issue then is to play to both at once. My Roon player can do that and that is how I perfom my AB tests. It is much, much better to switch between analog outputs of two DACs than switch the USB side. The latter will cause big time delay and potential issues with player software.

 

[amirm]

Do I need to do level match after AB switcher?

You are unlikely to find a level controlled AB switch that is anywhere as good as the best DACs. I roughly level match using independent DSP volume controls in Roon. It has accuracy of 1 dB which is useful but not perfect.

 

[gustavjung]

@amirm thank you so much for your help!

So do you mean that I don't need a multimeter for level matching? Can I use Right Mark Audio software or DAW using input of build-in sound card for this or Room player will do good as well?
Also did I understand it right, I can connect 2 DACs simultaneously by USB with 1 PC using Roon player and then connect DACs to AB switcher? I will do level matching before connecting DACs to AB switcher but with my headphones loaded using 2female-3.5->1male-3.5. Correct?

 

[gustavjung]

Sorry, I just figured it out. I don't actually need AB switcher for this test, I need only
two 3.5 male - > one 3.5 male and switch in a Roon player. Right?

 

[velasfloyd]

Hi!
Interesting. I will buy a switch and will compare as well. Hopefully I will save a lot of money with this.

But then I have the next question:

If every "good enough" dac and/or headamp sound the same for us. Why Amir would recommend a $500 or $1000 dac/amp?

lets say for example, Topping D90. Great Dac with super performance for $700. Amir recommended it. But then we cant hear any difference against topping D50s or DX3 pro.... they are $250. Yea, engineering is amazing and the level of the measurements are better... but why recommend something we cannot appreciate by our nature?
Yea, some will say because the inputs, outputs, form factor, size, color, looks, premium materials, durability... etc. And I agree, but all of that really can justify $300 or $500 or $1000 difference?

This forum is amazing and the value Amir and other put here is excellent, I thanks a lot. This is only a genuine question of mine. I would like to read your opinions on this

 

[solderdude]

Amir recommends something when the performance is up to his standards and doesn't look at the price point that heavily.
Good performer is good performer... being a $9.- dongle or high performance DAC.

You can buy the most expensive electronics but if one is using crappy speakers (and or has crappy acoustics) or uses below par headphones the result will still be disappointing.

 

[reallyoldcob]

Just tried using a "ONE Little Bear" 4 way switcher to compare three different headphone amps and they all sounded the same.
Headphone amps:
1. Apollo Twin MKIIII
2. Schiit Heresy (via Apollo Line Outs)
3. Audient Nero (via Apollo Monitor outs)

Headephones:
1. Focal Clear
2. Hifiman Sundara

Comparing the headphone amps less efficiently (unplugging and replugging), there is an obvious difference between the three amps. Is it possible that the Little Bear switcher circuit is creating a bottleneck for sound quality?

Comparing the Schiit Heresy before and after the Little Bear is a night and day difference.