What to do about the ABX test?

[Sokel]

Sorry to point that out,but mob rules sometimes is not only applied to trolls or the old fashioned audiophile with the wrong training.
Sometimes is also applied to people asking more extensive test,other views (more complex sometimes),etc.

Generally whatever gets us out of our comfort zone or dares to blame a cheap or high praised device here gets the stick or oblivion.
Whoever does a blind test that proves that he heard no difference gets away with it with ease,not the case if the other way around.
That's not very scientific,is it?It is just convenient as it follows the narrative.
That's not open mind people.

 

[voodooless]

But in all seriousness, if there isn't a good new-member-oriented FAQ it could go a long way. Perhaps we could start drafting it collaboratively in Google Docs or similar.

Just brainstorming here: I guess what would be useful to start this is to assign somebody as an editor (preferably a native English speaker), then come up with a list of questions, and then divide those up towards willing helpers to craft basic answers. For the answers we need some ground rules of do’s and don’t, and also make sure that people don’t crossover subjects, or if they do, refer to each other’s questions. The editor than checks for consistency, language and spelling and a v1 version can then be jointly reviewed, one question at a time. Finally it can be released into the wild. It would also be useful to be able to link directly to a specific question, so one can just post a link to it.

 

[solderdude]

It depends on the test. The case you put forward is not a comparison between two products and not a fair test. Everyone would need to listen to the same system.

And that is impossible when doing an online test or some downloaded files or even when comparing 2 DACs for that matter.
Circumstances, drivers, and above all listening 'skills' vary a lot.
For determining audibility thresholds you need lab settings and people with training (under 30 preferably).

For home listeners that make all sorts of claims these tests are easily done wrong and this will give false info to the one doing the 'test' and what's worse they go online and tell everyone that will listen that they can hear what is said can't be heard.

Home AB(X) tests are basically 'audibility threshold' tests for the one doing that test only on their gears/ears. The only conclusion that can be drawn is that they passes the test and have a suspicion the test might be done properly or not.

 

[Cbdb2]

There's another matter overviewed sometimes.
There is a percentage in population that don't do well under tests.Any tests.
It's not about rebellious attitude or something like that,it's about the stress of the test.
Teachers,professors,even driving instructors,etc know that well and they are able to identify it.
So,sometimes demanding a test can be stressful.

(only for consideration)

Those are test that you can fail. Thats the problem. Audiophiles think there hearing is special and that a AB is testing there golden ears not the DUT so if they hear no difference they think they failed. And there egos won't let them hear the truth so the test must be flawed. My first test with an "audiophile" would be AB the same 2 components to see if the person can admit they can't hear a difference. Most would fail which shows they will say theres a difference even when they can't hear one.

 

[Shadrach]

Blind conditions are certainly crucial if you want to pin the basis of the preference on the audio alone

True, if one is trying to make a comparison between one and another by ABX. I wouldn't describe it as crucial or even advisable to establish a preference on listening blind. One doesn't listen blind at home I'll assume. As you pont out many other factors other than sound quality have an influence.

If I didn't believe that measurements told enough of the story to give me confidence that I had made as good a choice as circumstances allowed I would try an ABX between loudspeakers with comparable measurements to prove to myself I could, or couldn't hear a difference. Then I would include the other factors to establish a preference be that adherance to some predefined curve or the colour of the enclosure.

 

[Sokel]

My first test with an "audiophile" would be AB the same 2 components to see if the person can admit they can't hear a difference. Most would fail which shows they will say theres a difference even when they can't hear one.

My best friend is an audiophile (one of those with the expensive gear) deep inside the community,gathering with others,etc.
I join him all the time.You want to know what happens in these gatherings?They listen to stuff and most of the time they laugh (me with them) as no apparent difference is heard.
But that's private,and as you can suspect are very smart and very successful people who know exactly what is going on.
They just enjoy these big heavy boxes,the massive speakers,etc.

But they wouldn't admit that to the average internet Joe nor they involve in such debates (maybe one out of ten).
And the main reason is not the people who try to educate them.
Is the people who hate them for having the disposable budget to spare without care.

 

[tomelex]

ABX or any sort of tests we do are not relevent basically if your head is not locked in a vise when listening over speakers, there can be peaks and troughs in amplitude response by moving your head a few inches in some cases and thus yeah, you can hear a difference simply due to the point in space in the room your ears are. With headphones, you stand a chance of hearing a difference in gear if you keep your head still again and let others do the switching.

SO, make sure you are not hearing the room!!

And yes, we can all hear differences in gear when there is enough there to hear, and so what does that prove, other than someone has better hearing than someone else, does it prove which one gear is more "right" when there are such small differences. Heck, simple balance between channels is hard to keep below 0.1 dB in lots of gear, and speakers, forget about it. ahahaha

 

[Shadrach]

And that is impossible when doing an online test or some downloaded files or even when comparing 2 DACs for that matter.
Circumstances, drivers, and above all listening 'skills' vary a lot.
For determining audibility thresholds you need lab settings and people with training (under 30 preferably).

For home listeners that make all sorts of claims these tests are easily done wrong and this will give false info to the one doing the 'test' and what's worse they go online and tell everyone that will listen that they can hear what is said can't be heard.

Home AB(X) tests are basically 'audibility threshold' tests for the one doing that test only on their gears/ears. The only conclusion that can be drawn is that they passes the test and have a suspicion the test might be done properly or not.

Can't disagree with any of that.
I would hope that peoples motivation for trying ABX testing isn't to pass the test. What it is useful for imo even if not done in the ideal environment is it shows how difficult it can be to differentiate between one thing and another and this is a valuable piece of knowledge.

 

[MRC01]

No the test result is only valid for that individual under his circumstances, assuming the test was done properly. It says nothing about other people nor about audibility in general.

If a single person can hear the difference (passes a proper ABX test), then it proves the difference is "humanly audible".

Much like when Alex Honnold free climbed El Capitan, he proved it was possible. Not that every person could do it, but at least 1 could. It was "humanly possible". Before he did that some people believed or said it was "impossible" and nobody could prove them wrong. After he did it, they were proven wrong.

It's not just a theoretical distinction because engineers and companies who want to build the best audio gear (or codecs, etc.) need to know what is "humanly audible". Not whether the average person can hear a difference, but whether any person anywhere could hear a difference. Yet neither should they waste their time engineering bandwidth, noise, distortion beyond what is "humanly audible".

 

[amirm]

Just how many actual individual and unique ABX tests have you posted here on ASR? Seriously, not the same ancient key jangling sample from a decade ago...

Oh. Now the age of the test matters? Why? Human hearing has evolved in that decade???

When someone says high res sounds the same as CD and I show double blind test otherwise, from a public test no less, that all of a sudden has no value? It not only has value, it completely shifts the discussion.

If age of listening tests matters this much, then nothing in psychoacoustics can be used as much of dates back decades and decades.

Amir, the files are local. You can analyse the files to any extent you want prior to running those files through a Foobar ABX. You pick two files, and ABX them. That's how it works. We all do it and have done for years. Most of us do it honestly, many don't.

Nonsense. Did you not watch the video? The specific test I showed there had countermeasures that made it impossible to analyze the files. Even if you did analyze them in advance, it doesn't do you any good whatsoever when foobar randomizes them. You are talking like someone who has never taken or conducted such tests. I suggest doing that before continuing to make false accusations and claims. These tests are extreme hard to pass for a reason: they are well done.

 

[amirm]

If a single person can hear the difference (passes a proper ABX test), then it proves the difference is "humanly audible".

Much like when Alex Honnold free climbed El Capitan, he proved it was possible. Not that every person could do it, but at least 1 could. It was "humanly possible". Before he did that some people believed or said it was "impossible" and nobody could prove them wrong. After he did it, they were proven wrong.

It's not just a theoretical distinction because engineers and companies who want to build the best audio gear (or codecs, etc.) need to know what is "humanly audible". Not whether the average person can hear a difference, but whether any person anywhere could hear a difference. Yet neither should they waste their time engineering bandwidth, noise, distortion beyond what is "humanly audible".

Exactly right. This is why we (industry) use trained listeners. And very difficult content that brings out differences. We do this so that we can find and fix even the extreme cases to get full transparency. Sticking one's head in the sand with easy tests and general public makes no sense if you care to do an excellent job.

Above is critical as you say: when making claims that this and that is transparent, it better be for all people, all content and all conditions. Otherwise it simply is not transparent.

 

[xnor]

So what is a next best alternative?

Know your psychoacoustics and personal hearing limits (established through "boring" and "tedious" double blind testing).

How do regular ASR members pick their gear?

Blind tests are the best most discriminating method. I find I can detect with 100% reliability some very small differences when using two segments of 5 seconds or less and rapid switching. OTOH, some of those I score 50/50 if segments are 15 or 30 seconds long. I have found anything I only hear using the very short segments which both can fit inside my Echoic memory are so small they have zero relevance to normal music listening. So on one hand if you cannot hear something using short rapid switching listening tests it is a pretty sure bet you cannot hear it. On the other if the difference isn’t large enough to hear with 30 second segments it isn’t big enough to matter for music listening.

The main question to ask yourself is this: will any of this bother you given the requirements and usage scenarios?
If not and you get to choose then pick what makes you happy based on other criteria.

It should also be noted that a lot of people asking for recommendations are nowhere near that level of controlled critical listening. They probably have a lot of false notions about audibility both from the subjective and measurement side, reinforced through confirmation bias.
Even if a reality check might annoy them momentarily, it's better in the long run.

So what other things can we do or that some of you do that is useful? What is a more effective way to engage people who don’t understand things about what can and cannot be heard without chiming in over and over “hey, do an ABX test or it didn’t happen”?

"Hey, sighted impressions are notoriously unreliable. Whether you like it or not, biases inevitably distort your perception. Therefore, your claims cannot be accepted as such without actual evidence. Try to repeat testing in a more controlled setting. Have you heard of blind testing?"

Or:

"Nice story, bro."
<turn around and talk to someone else>

 

[DonR]

Exactly right. This is why we (industry) use trained listeners. And very difficult content that brings out differences. We do this so that we can find and fix even the extreme cases to get full transparency. Sticking one's head in the sand with easy tests and general public makes no sense if you care to do an excellent job.

Above is critical as you say: when making claims that this and that is transparent, it better be for all people, all content and all conditions. Otherwise it simply is not transparent.

Certainly avoids the obvious retort of "transparent for whom?"

 

[amirm]

My wish for countering extreme claims to newcomers is to NOT demand an ABX test. As noted, in many cases, such a test is not possible for feasible. Instead, simply do what I ask: please run the same test blind, and repeat 10 times and see if you get 8 out of 10 right. If so, please take a video and post it here with full detail so that we can examine and repeat as necessary to verify. Doing so means that you are only using you ears and nothing else. Take your time. You can do it quickly or take days of weeks. Until you do this, you have not provided any evidence that you "heard" anything.

Above is completely doable and defensible.

 

[krabapple]

True, if one is trying to make a comparison between one and another by ABX.

It needn't be ABX. ABX is just one kind of blind test. Harman's blind speaker preference tests aren't ABX.

I wouldn't describe it as crucial or even advisable to establish a preference on listening blind.

I would, if you want to eliminate bias from appearance , price knowledge, etc.

One doesn't listen blind at home I'll assume. As you pont out many other factors other than sound quality have an influence.

So? That only means, don't assume you are only responding to the sound, at home.

If I didn't believe that measurements told enough of the story to give me confidence that I had made as good a choice as circumstances allowed I would try an ABX between loudspeakers with comparable measurements to prove to myself I could, or couldn't hear a difference. Then I would include the other factors to establish a preference be that adherance to some predefined curve or the colour of the enclosure.

If you have results from a good measurement set, that is a good basis for performance-based choice of a speaker .

But of course knowing those results will influence your 'preference' if you are doing a sighted evaluation.

The only way to 'prove' that your preference is based only on the sound, is a blind protocol. That's just a fact of life. Live with it, and adjust your claims accordingly.

 

[antcollinet]

When someone says high res sounds the same as CD and I show double blind test otherwise, from a public test no less, that all of a sudden has no value? It not only has value, it completely shifts the discussion.

Really interested by this. While I am pretty certain I can't hear better than CD, I'd like to understand what level of hi-res is needed for full transparency - if the tests determined this.

48/24 good enough or 96/24 needed. Or even higher?

 

[antcollinet]

It needn't be ABX. ABX is just one kind of blind test. Harman's blind speaker preference tests aren't ABX.




I would, if you want to eliminate bias from appearance , price knowledge, etc.




So? That only means, don't assume you are only responding to the sound, at home.




If you have results from a good measurement set, that is a good basis for performance-based choice of a speaker .

But of course knowing those results will influence your 'preference' if you are doing a sighted evaluation.

The only way to 'prove' that your preference is based only on the sound, is a blind protocol. That's just a fact of life. Live with it, and adjust your claims accordingly.

Perhaps so - but is totally impractical for most people making a purchasing decision. Certainly not possible to set up in store, and most people aren't going to purchase multiple high value bits of kit so they can conduct blind testing at home to decide which they want, then take the hit on return costs for the items they choose not to keep - even if return is possible.

 

[krabapple]

And hare we go again....

Yes, just as with amps, there are *conceivable conditions* under which it is *humanly possible* to tell hi rez from CD rate. Do those conditions normally pertain? NO.

Yet 'golden ears' would have you believe they do. Even their wives can hear it.

 

[krabapple]

Really interested by this. While I am pretty certain I can't hear better than CD, I'd like to understand what level of hi-res is needed for full transparency - if the tests determined this.

48/24 good enough or 96/24 needed. Or even higher?

And here we go again....

Yes, just as with amps, cables, DACs, there are *conceivable conditions* under which it is *humanly possible* to tell hi rez from CD rate.

Do those conditions normally pertain when listening to home audio? NO.

Yet 'golden ears' would have you believe they do. Even their wives can hear it. It's on THEM to prove it under their typical listening conditions, if they assert such a claim.

 

[antcollinet]

And here we go again....

Yes, just as with amps, cables, DACs, there are *conceivable conditions* under which it is *humanly possible* to tell hi rez from CD rate.

Do those conditions normally pertain when listening to home audio? NO.

Yet 'golden ears' would have you believe they do. Even their wives can hear it. It's on THEM to prove it under their typical listening conditions, if they assert such a claim.

No-one has claimed otherwise. I am simply interested in what level of digital resolution is needed for absolute transparency in perfect listening conditions with trained listeners.