• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

What to do about the ABX test?

krabapple

Major Contributor
Forum Donor
Joined
Apr 15, 2016
Messages
3,169
Likes
3,717
Perhaps so - but is totally impractical for most people making a purchasing decision. Certainly not possible to set up in store, and most people aren't going to purchase multiple high value bits of kit so they can conduct blind testing at home to decide which they want, then take the hit on return costs for the items they choose not to keep - even if return is possible.


If you imagine this statement of the obvious rebuts what I wrote, you are seriously misreading what I wrote.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
If a single person can hear the difference (passes a proper ABX test), then it proves the difference is "humanly audible".

Sure... if the test was done correctly and repeatable then yes, in a laboratory setting it could be used as evidence for further examination.

We are, however, talking about golden eared people making claims here and not doing or wanting to do some more rigorous testing. In the end, you only do these kind of tests for yourself. It would be foolish to think it will have any 'weight' in this community unless it is all meticulously logged and laid out in a scientific way. When such is not done you will be mowed down by quite a few theoretical hardliners. :)

In the end, you should do well executed blind AB(X) tests and other audibility tests to better understand your own perception limits. Not that of someone else, nor for any community. Even if one fails in the end it is not a guarantee someone else might pass the same test.

It's not just a theoretical distinction because engineers and companies who want to build the best audio gear (or codecs, etc.) need to know what is "humanly audible". Not whether the average person can hear a difference, but whether any person anywhere could hear a difference. Yet neither should they waste their time engineering bandwidth, noise, distortion beyond what is "humanly audible".
Interesting from a scientific viewpoint. And perhaps for research and improvement. One would need access to someone who can, train them etc.

Then again, if only 1% of a potential market can hear 'something' that may be a so tiny difference then commercial parties won't be interested and probably find it good enough for marketing.
 
Last edited:

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,414
Likes
12,296
Location
UK/Cheshire
If you imagine this statement of the obvious rebuts what I wrote, you are seriously misreading what I wrote.
That "perhaps so" in my reply is acknowledging the validity of your statement. But that validity doesn't also result in a practicality.

You do seem to be particularly grumpy this evening. :p
 

Shadrach

Addicted to Fun and Learning
Joined
Feb 24, 2019
Messages
662
Likes
947
It needn't be ABX. ABX is just one kind of blind test. Harman's blind speaker preference tests aren't ABX.




I would, if you want to eliminate bias from appearance , price knowledge, etc.




So? That only means, don't assume you are only responding to the sound, at home.




If you have results from a good measurement set, that is a good basis for performance-based choice of a speaker .

But of course knowing those results will influence your 'preference' if you are doing a sighted evaluation.

The only way to 'prove' that your preference is based only on the sound, is a blind protocol. That's just a fact of life. Live with it, and adjust your claims accordingly.
I'm not interested in proving anything.:) My preferences are very rarely based on how something sounds because tomorrow it will sound different.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,376
Likes
234,506
Location
Seattle Area
Really interested by this. While I am pretty certain I can't hear better than CD, I'd like to understand what level of hi-res is needed for full transparency - if the tests determined this.

48/24 good enough or 96/24 needed. Or even higher?
I cover the one scenario in the video I post. That one is a case of 16 bits not being enough. Statistical analysis of large library of music shows best case to be 18 bits. Ideally we have 20 bits so that we can cover a full dynamic range of 120 bits. That way the lowest volume would have inaudible noise.

As for sample rate, 48 kHz would be better but is not as necessary as the bit depth. Note that higher bit depth also helps when you apply EQ/bass management to make sure what is preserved is still above 16 bits even if you believe that is all you need.
 

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,584
Likes
38,283
Location
Gold Coast, Queensland, Australia
In the end, you should do well executed blind AB(X) tests and other audibility tests to better understand your own perception limits. Not that of someone else, nor for any community. Even if one fails in the end it is not a guarantee someone else might pass the same test.

This is the important part. One person's ABX results mean nothing to anyone else but themselves.

Anyone can train themselves to improve their perception limits, in order to hear aberrations in samples of music/sounds, especially when you are technically aware of exactly what processes and artefacts will highlight those differences.

Amir's classic jangling keys ABX is such a sample- great for determining compression artefacts in high frequencies. I have some incredibly well recorded tracks with bar chimes which do exactly the same thing. I've trained myself to just listen to what compression algorithms do to the few seconds of those bar chimes. Once you've heard it, you can never unhear it. But to lord the ability to hear those differences as some sort of 'golden-eared expert' is just silly.
 

Multicore

Major Contributor
Joined
Dec 6, 2021
Messages
1,676
Likes
1,768
That may be a noble effort, but ultimately will not bring any new knowledge to users. So what actually should be done to communicate in a way that is understandable and engaging?
It's hard to give advice. So much depends on the exact specifics of the conversation and the people involved.

I might try asking questions, like: What do you think might account for the experience you described? Follow up answers with more questions that guide the interlocutor to making statements of belief or assumption. Then leave those standing, or state your different positions, or maybe try correcting them.
 

fivepast8

Member
Forum Donor
Joined
Jul 26, 2020
Messages
46
Likes
201
Location
Switzerland
Seems increased commentary in recent weeks about ABX tests. Much of it stemming from people who come to ASR to set us straight about trusting our ears. I do agree with some who have said that calls for ABX or it didn’t happen have become almost like a club to beat people over the head with, and nearly cultish in how some new posters have the call rain down upon them. Not that I haven’t been guilty of it myself.

Some comments by @restorer-john have caused me to think about this situation. We stand little chance of convincing, or engaging in meaningful discussion with people with this approach. Like restorer-john I think there is a lot more talk of it than participation in or use of ABX listening tests among most posters. For most audiophiles it is impractical for most situations.

Some who don’t like ABX tests complain they are stressful. Only if you feel challenged by it or think you’ll suffer loss of face. After you have done it a couple or three times it isn’t stressful. It is major league TEDIOUS and BORING. Most of us do them with Foobar ABX or similar software. That isn’t very useful for amps and not at all for speakers.

So what is a next best alternative? What is a friendlier way to get the point across? How do regular ASR members pick their gear?

Blind tests are the best most discriminating method. I find I can detect with 100% reliability some very small differences when using two segments of 5 seconds or less and rapid switching. OTOH, some of those I score 50/50 if segments are 15 or 30 seconds long. I have found anything I only hear using the very short segments which both can fit inside my Echoic memory are so small they have zero relevance to normal music listening. So on one hand if you cannot hear something using short rapid switching listening tests it is a pretty sure bet you cannot hear it. On the other if the difference isn’t large enough to hear with 30 second segments it isn’t big enough to matter for music listening.

I believe the #1 thing to emphasize with any comparative listening is you must match levels precisely. Set a comfortable listening level and measure voltage of test tones at speaker terminals so each component matches within 1%. You cannot do any useful listening comparisons without this step. This one thing even in sighted listening can cause people to experience the disappearance or large reduction in differences they thought they were hearing.

The #2 thing to make clear is that fairly small deviations in frequency response are audible. So checking that might eliminate any need to go further for differences you hear. There are some simple ways to test this.

So what other things can we do or that some of you do that is useful? What is a more effective way to engage people who don’t understand things about what can and cannot be heard without chiming in over and over “hey, do an ABX test or it didn’t happen”?

What helped me more than ABX was the learning that the brain plays tricks on us. There were a couple of videos that I saw on ASR that helped me realize how much information/context/cues the brain adds. Once you accept that you cannot trust your ears in sighted tests then it becomes much easier to follow the science.

For example in the video below. If I can't tell "far from bar" .... then how the hell can I tell silver cable from copper cable? Such simple videos made me realize that in order to tell if something made a real difference, I would have to do it blind... and a little bit of statistics helps one understand that you gotta do it several times to rule out luck.

 

MattHooper

Master Contributor
Forum Donor
Joined
Jan 27, 2019
Messages
7,201
Likes
11,817
A very thoughtful post by Blumlein 88 !

I admit I have riled up plenty of folk in the more subjectivist forums by arguing for the relevance of blind tests. But I usually try to be as diplomatic as possible - emphasize that I'm not telling everyone "you have to blind test your equipment!" (Because most don't want to, and it's impractical in many cases for many audiophiles). But rather, we should at least be able to recognize that real problems that blind tests exist to address: the ever present variable of our biases in influencing our perception.

The problem is, it doesn't matter how diplomatically this is raised, some are triggered by the very notion their subjective experience, the one they use to vet gear, might be wrong. This is understandable as a human default: after all, the entire world comes to us through our subjectivity! Through experience. It can feel destabilizing to start to doubt the basis of your everyday experience, especially when dearly beloved beliefs have been built on that subjective "experience."

I think personal aptitude is a variable here. I remember how much I enjoyed taking Psychology courses in university. It was fascinating to learn so much about cognitive errors, bias effects etc, turning all sorts of intuitions and long held assumptions upside down. I found it energizing, in the same way that I have found some blind tests energizing: To experience what seemed to be solid sonic impressions melt away under blind tests didn't feel disappointing as much as an "aha" experience - it meant I was learning something right then and there!

Most people like the feeling of learning something new. But that can feel more threatening if it's directed at our intuitions or deeply held beliefs. In which case clinging to the belief can take priority. If more people perhaps were educated in a scientific way of investigation or thinking, I'd think more would be open to seeing tests of their own experience, or deeply held beliefs not so much as a "threat" but as an "aha" moment, a learning moment. Then blind testing would be seen as revealing, informative, not a threat.

Though, I still think people's inclinations will vary on that. Just as some people just find engineering and science more attractive than others.

Since nothing convinces more than an experience, it makes sense to make an experience easier to have.

I took that approach years ago on the AVS forum (I'm a home theater nut too). Before HDMI became the standard, video cables were often "reviewed" in the subjective press and seen by many videophiles as influencing the picture, just like audio cables purportedly did for a sound system. So you had lots of the usual suspects, Nordost etc, selling expensive S-Video, Component video cables with claims made for picture quality improvements.

I got hold of a range of cables, from cheap, to medium priced, to professional, to Nordost, and set up a sort of syllogistic blind test, posting very carefully taken photos of images on my plasma TV using the different cables. If "obvious differences" in contrast, color, sharpness etc were produced, they should be detectable in photos of images using the different cables. I put the test up, hid the identity of the cables, set up a poll for people to select the best image and asked for comments. Some claimed to see no difference, others were sure one cable produced better image quality, with variations along that spectrum. In the end, to no surprise, the guesses were random with respect to the cables. (The images looked all the same). It was well received by the many participants because I think it was easy and fun. (And some were grateful to find no discernible differences, saving them from the worry they needed to spend more money on cables).
 

Galliardist

Major Contributor
Joined
Jun 26, 2021
Messages
2,558
Likes
3,273
Location
Sydney. NSW, Australia
Really interested by this. While I am pretty certain I can't hear better than CD, I'd like to understand what level of hi-res is needed for full transparency - if the tests determined this.

48/24 good enough or 96/24 needed. Or even higher?
If the theory is right, then the answer lies with the implementation, does it not? After all, by the standard theory. even @amirm isn't hearing 30kHz or -150dB, or anything outside the envelope of what 16/44.1 can theoretically capture and play back We'd need a major feat of research to overthrow the existing theory, and nobody has yet proved that we can perceive 21kHz reliably.




Or have they?
 
OP
Blumlein 88

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,524
Likes
37,057
If the theory is right, then the answer lies with the implementation, does it not? After all, by the standard theory. even @amirm isn't hearing 30kHz or -150dB, or anything outside the envelope of what 16/44.1 can theoretically capture and play back We'd need a major feat of research to overthrow the existing theory, and nobody has yet proved that we can perceive 21kHz reliably.

Or have they?
Yes. Something like 2% of adults under 30 can hear above 20 khz in some fashion. I think about 23 khz is the highest known thus far. Those frequencies are heard as test tones and thresholds are at 100 db SPL and above. I can find some of the info perhaps, but it is legit. This is why J_J wanted a 60 khz sampling rate. Enough for response to 25 khz and room for a good filter.

Found this which is newer than testing I had read in the past. Also shows some response to 28 khz.
 
Last edited:

pma

Major Contributor
Joined
Feb 23, 2019
Messages
4,591
Likes
10,728
Location
Prague
Is not the necessary BW rather about audible differences in transients and pure sine tones. Cut the music signal that has many fast transients somewhere between 16kHz - 20kHz and you may get trained to catch the difference, though you would not hear the individual sine of the cut off frequency.
 

kemmler3D

Major Contributor
Forum Donor
Joined
Aug 25, 2022
Messages
3,022
Likes
5,635
Location
San Francisco
Is not the necessary BW rather about audible differences in transients and pure sine tones. Cut the music signal that has many fast transients somewhere between 16kHz - 20kHz and you may get trained to catch the difference, though you would not hear the individual sine of the cut off frequency.
I think that's why the recording in question is one of keys jangling.
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,414
Likes
12,296
Location
UK/Cheshire
If the theory is right, then the answer lies with the implementation, does it not? After all, by the standard theory. even @amirm isn't hearing 30kHz or -150dB, or anything outside the envelope of what 16/44.1 can theoretically capture and play back We'd need a major feat of research to overthrow the existing theory, and nobody has yet proved that we can perceive 21kHz reliably.




Or have they?
More than 21kHz - no according to Amir's reply to my post above.

More than 16 bits, yes according to the same information.
 

lashto

Major Contributor
Forum Donor
Joined
Mar 8, 2019
Messages
1,045
Likes
535
...
What is a more effective way to engage people who don’t understand things about what can and cannot be heard without chiming in over and over “hey, do an ABX test or it didn’t happen”?
I used to think that the everything-sounds-different 'audiophiles' were the worst and most annoying crowd ever.
After seeing so many everything-sounds-the-same 'objectivists' around here, I am starting to change my mind.

They both seem to think that they are very different (opposite even) but those two crowds sound quite the same to me. Same as entrenched in their beliefs, same as aggressive, same as easily annoyed by any trace of hint/evidence to the contrary .. and same as annoying for ~everyone.

The audio forums would be so much better and enjoyable without both groups..
 

Sokel

Master Contributor
Joined
Sep 8, 2021
Messages
5,844
Likes
5,787
I have one good example of the way some people think.
We have a dac that does 110db SINAD at 1Khz at 0db,ok?That's transparent as some people say here.
Is it still transparent under any condition if it does 95 SINAD at 4Khz or 90 at 10Khz?
Not uncommon at all.

Now combine that with lower level,real world conditions,etc.
So,can we safely presume as a law that one won't hear differences in an everyday system comparing 2 devices each one with it's flaws under real world conditions?
I measure my stuff all the time and I consistently get the same results which vary A LOT under different conditions.
I think we have to cool a little with the "transparent" thing,real world conditions (even mistakes) can do a lot of things.
It's not the test bench with the state of the art measuring gear.

When someone states that differences are night and day the first thing to ask is his gear,how it's hooked up and his normal listening conditions,especially if more than 2 volume controls are in play.
 

lashto

Major Contributor
Forum Donor
Joined
Mar 8, 2019
Messages
1,045
Likes
535
...
I think we have to cool a little with the "transparent" thing,real world conditions (even mistakes) can do a lot of things.
It's not the test bench with the state of the art measuring gear.
...

IMO, "the transparent thing" needs quite a lot of cooling.
Yes most devices are not audible (for everyone) and yes most people should not be concerned about the audibility of modern DACs/AMPs.
But at the same time, "transparent" is a very big word .. at least for the current state of audio tech
 

BDWoody

Chief Cat Herder
Moderator
Forum Donor
Joined
Jan 9, 2019
Messages
6,948
Likes
22,627
Location
Mid-Atlantic, USA. (Maryland)
It can feel destabilizing to start to doubt the basis of your everyday experience, especially when dearly beloved beliefs have been built on that subjective "experience."

Not to mention, often a big mountain of cash went in pursuit of those dearly held beliefs.

The idea that they could have been SO taken advantage of just isn't one of the options they will allow themselves to consider. While there are certainly plenty of people with a high enough net worth that $100k cables is as much of a relative hit to them as a pack of gum is to me, there are also plenty who make very real sacrifices to get that $800 USB cable on their way up the golden audiophool ladder.

It's that second group that I hope we can try to look after a bit better when they stop by. Expressing doubt and bewilderment doesn't make people trolls. We are sometimes a bit quick to start the facepalms when a link to an appropriate video or thread might actually help them understand what we are about.
 

lashto

Major Contributor
Forum Donor
Joined
Mar 8, 2019
Messages
1,045
Likes
535
... We are sometimes a bit quick to start the facepalms when a link to an appropriate video or thread might actually help them understand what we are about.
but only just a teeny tiny bit :)

How about a bit of thank-you-first attitude about all tests and all people doing them ?!
Blind tests are a complex, giant pain. IMO, anyone who tries a ABX/DBT deserves a medal. No matter the results, no matter how right/wrong they did it. (we may also learn a lot from the 'wrong' tests)

How about a bit more 'fairness' !?
Upvotes/thanks only come if the results match the "collective ASR beliefs". For unexpected results, there is an immediate uproar of "did you triple-check the voulme" or such. Why not start with a "thank you for your efforts"!? And I do not see choirs of "did you triple-check your switch" for the sounds-same results.
For unexpected results people are super quick to point stuff like "but but but .. it's only one test/person and should not be easily generalized". None of that for the expected results, although the comment is same as valid.

Yes, it's 100% fair and 100% right to scrutinize the unexpected results more. But still, everyone deserve praise and encouragement for their efforts.

Otherwise, why would anyone go through ABX torture when there seem to be only 2 outcomes:
  1. get expected results and praise/upvotes. But such results are ~useless at this point.
  2. get unexpected results and enjoy a few pounds of 'online spit' with zero thanks.
 
Last edited:

pma

Major Contributor
Joined
Feb 23, 2019
Messages
4,591
Likes
10,728
Location
Prague
Otherwise, why would anyone go through ABX torture when there seem to be only 2 outcomes:
And even more, why to prepare ABX test files for the forum?? It is time consuming, takes a lot of effort to make it properly with the test setup, level matching, time aligning of the files and what is the outcome? 2 - 5 members will take part in the tests and even less would post their abx protocol. So what one gets from his effort? It is 100x easier to write and publish pointless posts.
 
Top Bottom