ABX testing

[Simon 13th]

I enjoyed the chat the other day and didn’t want to repeat it but just wondered, has anyone ever done a valid ABX test on here of components that a person will say they can hear differences between.

I’m talking about someone who has listened to the gear in sighted tests, Knows the hifi, the room and puts the ABX test to the test.

Most of the ones online are pretty poor, no Controls, unscientific. Etc.

 

[Soniclife]

Not quite sure what your want but I recommend everyone to try a lossless Vs lossy abx first, as it doesn't involve fancy equipment, just software, and you don't need a second person to do it double blind, as the software handles it. It's a good test because it is passable, there are real differences, and you can completely control what is going on by doing the lossy encode from a file of your choosing.

 

[Sgt. Ear Ache]

Most of the ones online are pretty poor, no Controls, unscientific. Etc.

you mean compared to the controls in place for the standard subjective comparisons?

 

[sergeauckland]

My only comment about ABX tests is that whilst others may be able to do them, I can't. My audible memory is so short that by the time I've listened to X, I've long forgotten what A or B sounded like, and by the time I get back to A or B, I've forgotten what X was like.

My preference is for an AA BB AB BA test where the question is 'Are the two the same or different' That way, with close volume matching and fast switching, it's exquisitely sensitive to showing up differences. If A and B are indistinguishable, then any questions of which is better are meaningless, but if A and B can be reliably identified as different, then that can lead to other questions like why, or which is preferred, i.e. subjectively better.

I've done several such tests and in my case, sometimes the differences are clear and others much less so, to the point of just guessing.

S

 

[Purité Audio]

Completely agree, if I can’t tell any difference then that is good enough for me, but now I tend just to look at components measurements because level matching/unsighted comparison is a faff.
Speakers still sound different though.
Keith

 

[Jimbob54]

The person that should be doing the test is the one claiming the difference if measurements etc suggest they should be indistinguishable. At the point where they can say they truly have and can tell the difference, it might be worth others doing similar. Is that what you are getting at? Or are you asking if anyone here has done an ABX to support another members ABX findings?

There is another thread doing the rounds here which suggests credence should be given to credible subjective claims. To my mind, they gain credibility if the person can back them up with something resembling controlled tests themselves, not just the usual swapping out of old component with new and declaring a change.

 

[Simon 13th]

The question was whether there are any reliable ones.

I think the tests are largely pointless. Everyone knows that people are evaluating small audiophile type differences which are big in the scheme of things in an audiophile world, but small in a sense of not knowing the system, room and so on.

largely when people start banding about the plethora of unreliable ones online they talk about people who don’t even know the hifi so it’s hardly surprising they can’t discern it apart and any results are pointless.

then put the discussion on a forum with a range of intelligences and nobody actually properly agreeing any consensus, and the point is largely pointless.

I was interested in the home ones on people who have own gear. I did it once with some speaker cables and I was able to get everyone right after 7 or 8 my friend and I gave up. So reproduce-able on individual basis with people who know their gear, of good hearing, I suspect the statistical significance would be high.

 

[Sgt. Ear Ache]

why would "knowing your gear" matter? Do two things sound different from one another? Audiophiles claim differences all the time on gear that isn't their own gear.

And sorry, but I don't believe you did any proper blind tests and successfully identified any speaker cables. That's the sort of thing I'd need to see controlled by a neutral third party. Anyone can make claims about what they've done at home.

 

[BDWoody]

I haven't done one that would stand up to peer review, but I've blind tested myself with quite a few DAC's when I first found this site and thought everyone was crazy.

I had bought an Auralic Vega, and heard all the things Stereophile magazine told me I should hear.

Came here looking for confirmation about what a smart audiophile I was, and while I found that it did indeed measure very well, I started picking up more and more about the reality of bias and psychoacoustics, and decided to test myself.

Long story short, I sent the Vega back, and bought 3xJBL708P's and 2x705P's with about the same money, laughed a little, and haven't bothered with a controlled comparison since. Haven't felt the need.

Once you get an understanding of a few basics, the doubt about what might be missing goes away.

 

[Purité Audio]

If there really is an audible difference then it can be reliably differentiated, IME very few wish to take an unsighted comparison, they just have too much invested, psychologically and often financially.
Keith

 

[danadam]

My preference is for an AA BB AB BA test where the question is 'Are the two the same or different'

Isn't that essentially switching between A and X in ABX test?

 

[sergeauckland]

Isn't that essentially switching between A and X in ABX test?

Not quite. An ABX test involves listening to A, then listening to B, then to X and deciding if X was A or B. By the time I've heard X I've forgotten about A or B. An AA AB BA BB test requires just saying same/different. No memory is required with rapid switching.

It's possible with an ABX test just to say X isn't A so must be B, but in a randomised ABX test, B could well be A again. That's why ABX testing requires lots of single attempts and/or many participants to get statistical validity.

Whether ABX or my preferred AA AB BA BB testing, they both need a large sample to avoid lucky guesses, either way, false positives or negatives so properly conducted tests are so rare.

S

 

[eddantes]

Your host has done a video on the subject:

The search function on this site is effective if employed.

 

[JeffS7444]

I've done just enough ABX to marvel at how apparent sonic differences can simply disappear. The gold standard here is to match volumes within 0.1 dB, but per Stereo Review's Larry Klein, you can do this by ear, and the way you know you've got it is when you can no longer hear differences! If you lack the hardware to allow for instantaneous switches, you might try making digital recordings of each, then using free software ABX comparator.

 

[VintageFlanker]

I was interested in the home ones on people who have own gear. I did it once with some speaker cables and I was able to get everyone right after 7 or 8 my friend and I gave up.

Can you tell us more about this? Right now, please forgive me if I'm a touch skeptical...

By the way, I have what you're asking for. Controled blind ABx tests about cables (power and interconnects). Few volunteers audiophiles who are active on many French forums, claiming to hear night and day differences as you do. They failed, as always: Their own gears, their own room: not any real difference whatsoever... This was conducted and published by French magazine Canard PC Hardware:
https://drive.google.com/file/d/1dSRdTPvFiN6PQFGjsVo-8JquqHgyIsfL/view?usp=drivesdk
(Page 9 and 10)

For the record, Canard PC is one of the most trusted and respected (French) magazine in the field of hardware. They have objective approach, which is much more common with PC components, obviously. This test is very well-known there and no one yet managed to discredit it.

... But I guess you'll say they should be deaf or something.

 

[danadam]

Not quite. An ABX test involves listening to A, then listening to B, then to X and deciding if X was A or B.

That sounds like the original procedure from 1950: Munson; Gardner. "Standardizing Auditory Tests". The Journal of the Acoustical Society of America.

Then there is a modification, apparently called "modern ABX", from 1982: Clark. "High-Resolution Subjective Testing Using a Double-Blind Comparator". Journal of the Audio Engineering Society. As far as I understand, it gives control to the participant to play A, B or X as they see fit.

Here's @j_j's comment from HA:

I have to point out that the "abx" test of Munson was a sequential test, not a time-proximate test in which the subject had control over stimulus selection at all points in time. This test has been brought up from time to time, and various people have taken upon themselves to scold the entire AES about how this sequential test is such a bad test (as it is, the lack of time proximity is massively desensitizing), ignoring the fact that this is NOT the modern test called "ABX".

The ABX test from the Michigan bunch was time-proximate. This is a substantial improvement, and is what is commonly referred to in the present day as the ABX test.

Is the original procedure still used? Earlier in the same thread Arnold Krueger says about the 1950 procedure:

[...] I thought it was not just different but hideously different. I initially couldn't figure out why anybody with half a brain would do a listening test in such a difficult way. However I gave it a chance, read a few papers and figured out why it made sense, and it does make sense for certain purposes.

Unfortunately he didn't explain why it makes sense and what are those purposes.

 

[Blumlein 88]

That sounds like the original procedure from 1950: Munson; Gardner. "Standardizing Auditory Tests". The Journal of the Acoustical Society of America.

Then there is a modification, apparently called "modern ABX", from 1982: Clark. "High-Resolution Subjective Testing Using a Double-Blind Comparator". Journal of the Audio Engineering Society. As far as I understand, it gives control to the participant to play A, B or X as they see fit.

Here's @j_j's comment from HA:

Is the original procedure still used? Earlier in the same thread Arnold Krueger says about the 1950 procedure:

Unfortunately he didn't explain why it makes sense and what are those purposes.

My guess is he thought for threshold or masking tests it had more merit.

I rather like duo-trio testing. You have the reference, and then listen to a pair of choices picking which on sounds like the reference. It is how I actually do the Foobar tests. Having control over the choices makes that possible.

I also like triangle tests. You get three samples and listen to decide which of the three sounds different. Two are the same and one is different.

 

[pma]

My preference is for an AA BB AB BA test where the question is 'Are the two the same or different'

This is a good method that you have described. ABX tends to result in "everything sounds same" if the differences are small. So it is about the goal we have. If we want to catch very significant differences only, ABX is fine. In case we are interested in tiny improvements and their audibility, your method is better.

 

[Alex-D]

Even without a proper blind test I stopped hearing any difference in DACs I own after I volume matched them with 1khz tone and a cheap noise meter.

HP laptop DAC, Apple US USB-C dongle, GRACE balanced DAC, Dragonfly 1.2

Before the strict volume matching newer stuff was always better: Apple Dongle > Grace Bal > Dragonfly

 

[solderdude]