ABX; you really need to do it to understand.

[Shadrach]

Despite many of the passing nods in reference to ABX testing I can’t help get the impression from many of the posts here on ASR, that while some are prepared to believe this reviewer, or that person, may not have been able to pick correctly enough to give statistical significance, they would somehow be different.
The reasoning runs from knowing exactly what your system sounds like to the belief that if your system is of high enough quality, be that judged by specifications, or subjective impressions, somehow you will hear the difference.
On other forums I’ve come across audiophiles who just won’t do ABX; they really don’t want to find out what they can and can’t hear and are happier in their ignorance.
Some people do a quick bit depth and sample rate ABX with say foobar on a windows machine and call it a day. I’ve had conversations with such people and somewhere in the conversation I find out that while they’ll accept they couldn’t differentiate between the differing formats they believe that when it comes to their hardware it will all be different.
If I, as some people do, were considering spending thousands of pounds on a piece of audio equipment, I would want to know with a reasonable degree of certainty that that particular piece of equipment did in fact sound very noticeably different to something I could buy for a few hundred.
It’s easy to believe that setting up any meaningful ABX test is too complicated and to time consuming to be worthwhile. In fact you can do a reasonable job with a large curtain and a bit of help (no, not the wife).
Find some local audio enthusiasts and make a weekend of it. I have tried in the past to persuade people to do this at a couple of the small DIY audio meets that used to be more common, without much success I might add.
Yes, it is going to hurt. Yes, you may well feel a bit of a fool.
On the plus side for yourself, you may end up saving thousands of pounds and go on a great holiday instead (yes, with the wife) You may also find you stop worrying about every micro detail of your system and start enjoying the music again.
The benefit for the rest of us is you may stop going on the Internet and writing a complete nonsense that those who have used ABX know is a complete nonsense. Seriously, you’ll be a better audiophile for the experience. The more you do it, the funnier it gets and when you’ve got to the point that you’ve built your own ABX switch box to take to dealer demos you’ll know you can withstand the slickest salesperson and the most enticing looking boxes.
Don’t just read about other peoples experiences; do it yourself. Make it a party. Have a bit of fun at your own expense.
This is a good demonstration of how to go about an ABX test.
http://matrixhifi.com/ENG_contenedor_ppec.htm
We found a thick blanket hung from the ceiling allowed us to do speakers as well. Sure, the blanket had an effect on the sound waves but the effect was pretty much the same for any speakers. (Don’t forget to shuffle the speaker positions around)

The conclusion that the group I used to belong to came to was it was possible to get positive results, strangely most differences were heard off axis and when not concentrating. However, even with many of the loudspeakers we listened to what we preferred often wasn’t the speakers we expected to like.

 

[daftcombo]

Some people do a quick bit depth and sample rate ABX with say foobar on a windows machine and call it a day.

Even that made my ears a lot more humble. Very difficult exercise.

I did sighted tests with several DAC, not even level-matched, trying to express differences with words. Failed too.

 

[invaderzim]

Even that made my ears a lot more humble.....

Doing a really basic A/B test after being blown away by a component was the most humbling experience for me. I thought the new component was amazing and would have written a massive review stating how wonderful it was.
After picking my cheap old component nearly half the time as the best one I realized that all those glowing reviews of the latest, greatest, most expensive piece were written by people just like me and, in reality, there likely wasn't a damn bit of difference for them either.

 

[Dimitrov]

What I would LOVE to see is someone knowledgeable in test design to create a thread detailing step by step how to perform a test - from the blinding process, how best to do it, the level-matching process, the number of trials, what kind of switch box to use - just a step by step tutorial for lay people.

I have never seen this.

I have wanted to do this stuff for a long time but I have zero clue how to do it properly and that's the problem. There doesn't seem to be a primer on any of this, so people will just say go do it, but I think it's more difficult to do than people probably realise.

You also want the results to be meaningful in some sense, and if you do the test poorly which is very possible for folks who don't know what to do, then the results can be ignored. So what I think would be of great service to the community is someone who has A LOT of experience doing this, who can detail exactly how to do it - with a bunch of people - so that we can try and replicate the experiment.

Teach lay folks how to conduct good listening tests, not just say "go to a DBT", but actually show as way of example how to do it - maybe even set up a Youtube video tutorial, I don't know. That would be great and everyone who is curious about this stuff would benefit.

What does everyone think?

 

[Blumlein 88]

What I would LOVE to see is someone knowledgeable in test design to create a thread detailing step by step how to perform a test - from the blinding process, how best to do it, the level-matching process, the number of trials, what kind of switch box to use - just a step by step tutorial for lay people.

I have never seen this.

I have wanted to do this stuff for a long time but I have zero clue how to do it properly and that's the problem. There doesn't seem to be a primer on any of this, so people will just say go do it, but I think it's more difficult to do than people probably realise.

You also want the results to be meaningful in some sense, and if you do the test poorly which is very possible for folks who don't know what to do, then the results can be ignored. So what I think would be of great service to the community is someone who has A LOT of experience doing this, who can detail exactly how to do it - with a bunch of people - so that we can try and replicate the experiment.

Teach lay folks how to conduct good listening tests, not just say "go to a DBT", but actually show as way of example how to do it - maybe even set up a Youtube video tutorial, I don't know. That would be great and everyone who is curious about this stuff would benefit.

What does everyone think?

For starters, sighted or unsighted, MATCH LEVELS PRECISELY to within .1 db. Otherwise you got nothing.

HOW?

Play a 1 khz tone, and measure at the speaker terminals the voltage. Voltages of both sources must be within 1% of each other.

Next be able to switch quickly. Ideally near instantly. Due to limits of echoic memory no more than a few seconds. How to do this depends upon what is being tested.

 

[daftcombo]

For starters, sighted or unsighted, MATCH LEVELS PRECISELY to within .1 db. Otherwise you got nothing.

HOW?

Play a 1 khz tone, and measure at the speaker terminals the voltage. Voltages of both sources must be within 1% of each other.

Next be able to switch quickly. Ideally near instantly. Due to limits of echoic memory no more than a few seconds. How to do this depends upon what is being tested.

I would say 0.1dB, not 1dB, and at every frequence, not only 1kHz, for a game in "normal mode".
What you suggest is just "rookie level".

 

[andreasmaaan]

I would say 0.1dB, not 1dB, and at every frequence, not only 1kHz, for a game in "normal mode".
What you suggest is just "rookie level".

He actually did say 0.1dB

 

[daftcombo]

He actually did say 0.1dB

Hey sorry, didn't see the dot. Still, at every frequence 20Hz-20kHz.

 

[Dimitrov]

For starters, sighted or unsighted, MATCH LEVELS PRECISELY to within .1 db. Otherwise you got nothing.

HOW?

Play a 1 khz tone, and measure at the speaker terminals the voltage. Voltages of both sources must be within 1% of each other.

Next be able to switch quickly. Ideally near instantly. Due to limits of echoic memory no more than a few seconds. How to do this depends upon what is being tested.

Okay, but matching levels is just one step (albeit an important step) in a multitude of different steps to performing a good listening test.

Switching how? If we test amplifiers, or DACs, or cables ... switching needs to be done almost immediately. What is the best way to switch ... that lay folks can do themselves?

What would the test be administered? From start to finish?

 

[Shadrach]

What I would LOVE to see is someone knowledgeable in test design to create a thread detailing step by step how to perform a test - from the blinding process, how best to do it, the level-matching process, the number of trials, what kind of switch box to use - just a step by step tutorial for lay people.

I have never seen this.

I have wanted to do this stuff for a long time but I have zero clue how to do it properly and that's the problem. There doesn't seem to be a primer on any of this, so people will just say go do it, but I think it's more difficult to do than people probably realise.

You also want the results to be meaningful in some sense, and if you do the test poorly which is very possible for folks who don't know what to do, then the results can be ignored. So what I think would be of great service to the community is someone who has A LOT of experience doing this, who can detail exactly how to do it - with a bunch of people - so that we can try and replicate the experiment.

Teach lay folks how to conduct good listening tests, not just say "go to a DBT", but actually show as way of example how to do it - maybe even set up a Youtube video tutorial, I don't know. That would be great and everyone who is curious about this stuff would benefit.

What does everyone think?

I think its a fair comment.
The link I gave does show how they went about it.
I may write a bit on ABX setup.
There are some recommendations based on how long you can remember what something sounded like. The thing is, different people have differing memory abilities, both visual and auditory. For audio enthusiasts what should be important is not whether or not they can pick correctly but how small the differences are should there be any. This would at least reduce hopefully those who fit say an audiophile fuse and then report small but significant improvements.
Stringent conditions for the average stereo listener are not so important imo. At work, the conditions had to be fairly exacting and after a week of intense listening and watching pilots and tank crews taking such tests, I got pretty tired. Given we were testing with ambient noise at 110 db and more this is hardly surprising.
The purpose of the tests was to try discover what was and what wasn't important in order for the listener to understand a set of brief instructions and later, to build the hardware that those instructions were communicated with.
Possibly the most interesting thing we found was people a) don't listen and b) interpret what they hear with reference to what they've heard before and what they expect to hear in the circumstances they are in. This gave an awful lot of simulated 'disasters'.

 

[Blumlein 88]

Hey sorry, didn't see the dot. Still, at every frequence 20Hz-20kHz.

The reason I suggest just 1 khz for less formal tests is first practicality. Secondly if two devices have different frequency response that will be a reason for them to be identifiable. Yes, you could just measure it and know that without listening. Some still want to hear the difference. Matching at 1 khz is an area you aren't likely to have un-flat response from any device except transducers. If you aren't happy with that I'd suggest matching with a pink noise signal.

 

[Blumlein 88]

Okay, but matching levels is just one step (albeit an important step) in a multitude of different steps to performing a good listening test.

Switching how? If we test amplifiers, or DACs, or cables ... switching needs to be done almost immediately. What is the best way to switch ... that lay folks can do themselves?

What would the test be administered? From start to finish?

Well, I think there are some ITU standards for this. But if you look at that you'll throw your hands up and say forget it. So my idea was to start with the simplest and most important steps. If you aren't doing academic research or developing a commercial/industrial product your needs are less stringent, but can still be useful.

Switching........how to do it? If you have a preamp, connect each DAC to different inputs and switch inputs on the front panel. Same might be possible for some cables.

Amplifiers are going to be more problematic in switching speed. You'll just have to take longer. That reduces the acuity of the differences that can be heard. But many people will listen to an amp this week, and have no problem comparing sound to one they heard last week. So this is better than that. If you set everything up and can use banana plugs for speaker connections you can do it pretty quickly. Using two people for switching (one per channel) can speed it up.

I'm sorry, I'm doing this piecemeal instead of a comprehensive guide, and the guide is what you asked for.

 

[RayDunzl]

Still, at every frequence 20Hz-20kHz.

Frequency is a continuum.

That would take forever.

And, if you have to attenuate Device A to match Device B at 1kHz, and boost Device A to match Device B at 2kHz, you've already identified a difference.

 

[daftcombo]

Frequency is a continuum.

That would take forever.

And, if you have to attenuate Device A to match Device B at 1kHz, and boost Device A to match Device B at 2kHz, you've already identified a difference.

I hear you, but if you can EQ one so that each and every frequency from 20Hz to 20kHz of both devices are the same, and if afterwards you can't spot one from the other in an ABX, it means that both devices CAN sound the same, hence keeping this EQ settings, you can have the more expensive for the price of the cheapest.

That was how Richard Clark worked for his 10.000$-reward amp test.

 

[pkane]

I hear you, but if you can EQ one so that each and every frequency from 20Hz to 20kHz of both devices are the same, and if afterwards you can't spot one from the other in an ABX, it means that both devices CAN sound the same, hence keeping this EQ settings, you can have the more expensive for the price of the cheapest.

That was how Richard Clark worked for his 10.000$-reward amp test.

Probably not what you are looking for, but one way to do arbitrary ABX or other type of testing is to capture/digitize the output of the device under test using a quality ADC. You can then compare multiple captures with different DUTs, or with and without a DUT, nearly instantaneously by switching between the recordings.

Foobar ABX comparator will help with this. My own DeltaWave software may also be useful to perform a very precise phase/amplitude match between recordings, as well as exactly the type of EQ you are asking about. It also has an ABX-style comparator tool built in, although for now it's limited to comparing two waveforms rather than comparing stereo recordings.

 

[andreasmaaan]

Building an ABX switch box for testing yourself against different pieces of hardware (amps, cables, etc) is quite difficult (for an electronics amateur like me anyway). Rod Elliott outlines one approach here. There is also a commercial box available from https://avahifi.com/products/abx-switch-comparator.

I would say that anyone interested in ABX testing hardware who hasn't tried an ABX using software should try with software first, to at least give themselves an idea of roughly where the audibility thresholds with their ears and their setups might be. Realising how difficult it is to hear 0.1% or even 1% nonlinear distortion, or 720° of phase shift, etc. etc. can be very enlightening.

 

[Shadrach]

I would start with the easiest option which is to install foobar 2000, assuming you have a Windows computer.
https://www.foobar2000.org/
Once you have foobar up and running, install this component.
https://www.foobar2000.org/components/view/foo_abx
There are ABX comparators for Linux computers but most don't come with a GUI (General User Interface) and requires compiling (Basically unziping a compressed package and instructing the operating system where to place the individual files)
foobar has a resampler so you can change say from flac to mp3 and various other formats.
I would suggest if you have them, or are prepared to download some you try comparing various remasters to 'original issues as a first step.
You should get lots of positive results.
10 trials seems to be the normal. The more trials the more accurate the results.
Essentially you need to 'play' with the ABX feature in foobar. If you take it all too seriously you'll get frustrated.
Remember despite what one reads on the Internet very few people can tell the difference between an mp3 and flac version of the same file; even those sitting in front of multi thousand systems.
Testing hardware is more challenging. As I mentioned above, read the stuff in the link in my first post at Matrix. It covers all the features of the test and the maths at the end.
You can buy ABX switch boxes, they tend to be expensive.
Finally, there are lots of knowledgeable contributors to this site (I doubt many have done an ABX) and what can seem a bewildering array of graphs and numbers. You will get told often that we/they can measure everything to do with audio (some are a bit more careful in this generalization) The thing is is they can't measure you and it is you that hears the music.
One last thing. The most measurably transparent piece of audio equipment I have is the JDS Labs Atom amplifier. It cost about £140.
I couldn't replace one single driver in my loudspeakers for that amount and I dread to think how much say the equivalent one of my amplifiers, say the Exposure 25 would cost these days, so I'm just as daft as the rest, but I know it.

 

[SIY]

I think that the way the question is structured, you've made what I call the Procrustean Fallacy, that is, assuming there is a single answer. There isn't.

Step one, before all else- define specifically what it is you're trying to test. Then a reasonable protocol can be developed for that specific question. It will NOT be the same for testing other things.

The things that others are mentioning here (e.g., level matching) are just basic controls, things that are necessary but not sufficient.

 

[andreasmaaan]

There are also a few nice ABX tools available online.

Here are the ones that I've used (from a designer called Blogohl):

• Distorder - nonlinear distortion audibility
• Phase Audibility - as the name suggests*

*I've had a small issue with this one on my system. Since the test is not "break and make", when X is the unfiltered signal there is no break, whereas when X is the filtered signal there can be a tiny break when switching. So to work effectively, the minimum possible system latency needs to be implemented (and even then there might be a long enough break to destroy the validity of the test).

 

[Shadrach]

Building an ABX switch box for testing yourself against different pieces of hardware (amps, cables, etc) is quite difficult (for an electronics amateur like me anyway). Rod Elliott outlines one approach here. There is also a commercial box available from https://avahifi.com/products/abx-switch-comparator.

I would say that anyone interested in ABX testing hardware who hasn't tried an ABX using software should try with software first, to at least give themselves an idea of roughly where the audibility thresholds with their ears and their setups might be. Realising how difficult it is to hear 0.1% or even 1% nonlinear distortion, or 720° of phase shift, etc. etc. can be very enlightening.

I've got lots of time for Rod Elliott. He's been one of the few rational voices in recreational audio over the years.
I've built a few of his projects.