• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

What to do about the ABX test?

GaryH

Addicted to Fun and Learning
Joined
May 12, 2021
Messages
957
Likes
1,063
Unless one has bat hearing, any difference would have to be in the audible frequency range.
This is undeniable (for rational minds).
Something in the playback system accounts for the difference that may not point to the inherent superiority of HD-Audio.
Yep, I've pointed out the invalidity of these tests before.
 
Last edited:
OP
Blumlein 88

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
17,186
Likes
29,949
If you go to 19minutes in the post 16 video he explains this.
Okay, this wasn't jangling keys as someone else mentioned in this case. It was a music file in Archimago's hi-res listening test. And yes, he did use IEM's and found levels fading under -70 dbFS and pumped up the volume. I do think that is cheating. Amir even says without doing this he could hear no difference. I don't see this as any different than dumping a file in a sound editor and looking for differences. It has zero to do with any normal listening. Amir doesn't consider 16 bits transparent due to this, while I'd say for normal listening it is. One could say 24 bit encoding is transparent to levels beyond the ability of any hardware and 16 bits isn't, but if 16 bits gets enough humans can never tell then what have we gained?

Just after this he shows hearing Ethan's generational test. I don't know what the difference is, but Amir said except for one track with a tale accidentally left in he couldn't hear the 8th generation files I have posted. Of course I didn't leave any dead space in files to prevent someone doing the find quiet portions and amp up the noise to hear differences thing.

What this shows is the use of ABX listening test with technical knowledge of how things work one can discern extremely small differences using ears only. Other methods of blind testing work beyond just the ABX method.
 

fpitas

Major Contributor
Forum Donor
Joined
Jul 7, 2022
Messages
3,240
Likes
3,876
Location
Northern Virginia, USA
The
true. But one can just use the "ignore" button.
The true havoc-masters can and should be managed by admins.

Much easier said then done, I know.
My ignore list is actually empty, I find that function kind of rude. And I might just be too curious for it..
And around here the mods are pretty efficient. The real bad news sorts disappear quickly enough.
 

dlaloum

Major Contributor
Joined
Oct 4, 2021
Messages
1,839
Likes
1,278
I've had this debate a couple of times. It goes like this:"the amplifiers only sound different with (insert name of B&W or Magico printer here) so obviously my speakers are more revealing"
Yep - hence we should all have speakers with impedance dips to 1 ohm, as they are more revealing...

Having said that - those 1ohm stats, are superb speakers, and my current Gallo's which I am very fond of, go down to 1.6ohm due to their CDT tweeter... it is not unusual for certain types of "exotic" drivers... - which can indeed be very revealing... so many of these types of speaker are indeed very revealing... but that isn't the reason the amps sound different on them!
 

JSmith

Major Contributor
Joined
Feb 8, 2021
Messages
3,507
Likes
8,149
Location
Algol Perseus
And around here the mods are pretty efficient.
ASR is very much a breath of fresh air compared to many other audio forums and I think most of us prefer things kept that way.

An example... a forum I would often frequent many years ago had a resident "golden ears" that members would actually send their amps (often expensive too) to for a listening "test", after which time the member would post a few pages of impressions and comments about the amp for the member who sent it and the forum in general. They were only listened to... with comparison comments, yet they were not compared AB let alone blind. In fact the amps he would "compare" the current amp to would not even be in his possession anymore. To me it was a completely useless exercise... and strangely many of the "tested" amps would end up in the FS section not long at high prices for 2nd hand after, with a referral to the "test" post as evidence of it being a fine amp. In fact many members actually relied on this list to make purchasing decisions... and the forum then had sponsors. To make it worse, the person was actually very pleasant and really believed in what they were doing after years of raking amps this way, thus due to this "experience" felt he was in a position to conduct such single "tests".

Now if anyone asked are any of these amps AB compared at least, or any kind of actual testing done on them, or pointing out the folly of same... their posts would be removed and a note sent asking the member to refrain from posting such things as it was against the spirit of the forum, regardless of seniority or join date and regardless of time consuming contributions to the forum itself. ASR is basically the bizarro world opposite of forums like that... a decent place where those attitudes are in the minority based on factual information and real published data.


JSmith
 

lashto

Addicted to Fun and Learning
Forum Donor
Joined
Mar 8, 2019
Messages
873
Likes
455
ASR is very much a breath of fresh air compared to many other audio forums and I think most of us prefer things kept that way.

An example... a forum I would often frequent many years ago had a resident "golden ears" that members would actually send their amps (often expensive too) to for a listening "test", after which time the member would post a few pages of impressions and comments about the amp for the member who sent it and the forum in general. They were only listened to... with comparison comments, yet they were not compared AB let alone blind. In fact the amps he would "compare" the current amp to would not even be in his possession anymore. To me it was a completely useless exercise... and strangely many of the "tested" amps would end up in the FS section not long at high prices for 2nd hand after, with a referral to the "test" post as evidence of it being a fine amp. In fact many members actually relied on this list to make purchasing decisions... and the forum then had sponsors. To make it worse, the person was actually very pleasant and really believed in what they were doing after years of raking amps this way, thus due to this "experience" felt he was in a position to conduct such single "tests".

Now if anyone asked are any of these amps AB compared at least, or any kind of actual testing done on them, or pointing out the folly of same... their posts would be removed and a note sent asking the member to refrain from posting such things as it was against the spirit of the forum, regardless of seniority or join date and regardless of time consuming contributions to the forum itself. ASR is basically the bizarro world opposite of forums like that... a decent place where those attitudes are in the minority based on factual information and real published data.


JSmith
most audio forums are just like you described, especially oldschool ones like audioasylum. Biggest/best 2 examples would be headfi & audiophilestyle, where it's forbidden to even mention tests/DBTs in the 'normal' threads.
And then you have a few in the ~middle like sbaf and headphones.com (or at least they try/pretend to be). And quite a lot of 'cacophonies' like reddit/discord/facebook groups.

The only forum I know to be quite close to ASR is hydrogenaudio. They are even more vocal/strict about things like "DBT or it did not happen". And it's ~deserted nowadays.
They do not have someone like Amir to feed an inhuman amount of info/tests. And those strict, hyper-skepticism policies drove people away. When you only accept DBTed posts, you only have one per month...
 
Last edited:

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,817
Likes
2,421
Location
Massachusetts
Yep, I've pointed out the invalidity of these tests before.
I was referring to the jangling keys discussions that have occurred across many threads as an example where an HD Audio file was distinguishable from the CD quality version. Agreed, raising the volume is the video does not prove that HD Audio makes sense. I think we already knew there was a noise floor.

Reconstruction filters implementation may have an audible effect. I suppose a SBT could be performed for that though.
I see no reason for any implementation that does not have a linear fast filter, as this technically the most correct Implementation.

- Rich
 
Last edited:

krabapple

Major Contributor
Joined
Apr 15, 2016
Messages
2,187
Likes
2,431
Yep, I've pointed out the invalidity of these tests before.
Okay, this wasn't jangling keys as someone else mentioned in this case. It was a music file in Archimago's hi-res listening test. And yes, he did use IEM's and found levels fading under -70 dbFS and pumped up the volume. I do think that is cheating. Amir even says without doing this he could hear no difference. I don't see this as any different than dumping a file in a sound editor and looking for differences. It has zero to do with any normal listening. Amir doesn't consider 16 bits transparent due to this, while I'd say for normal listening it is. One could say 24 bit encoding is transparent to levels beyond the ability of any hardware and 16 bits isn't, but if 16 bits gets enough humans can never tell then what have we gained?

Just after this he shows hearing Ethan's generational test. I don't know what the difference is, but Amir said except for one track with a tale accidentally left in he couldn't hear the 8th generation files I have posted. Of course I didn't leave any dead space in files to prevent someone doing the find quiet portions and amp up the noise to hear differences thing.

What this shows is the use of ABX listening test with technical knowledge of how things work one can discern extremely small differences using ears only. Other methods of blind testing work beyond just the ABX method.


Indeed, and that's pretty much the objection I've posted every time he's posted such 'see, there *is* an audible difference' exercises. And we're talking all the way back to Hydrogenaudio days where Amir and Arny Kruger went at it frequently.

Such forensic 'listening' is 'valid' in the sense that SINAD differences with no likely bearing on audibility are 'valid'. Which is to say, pretty to know...but totally irrelevant to the sorts of 'veils were lifted' and 'of course A sounded better than B' claims routinely presented by audiophiles re: lossy vs lossless audio and hi res vs CD.

Amir is of a very different opinion.
 
Last edited:

krabapple

Major Contributor
Joined
Apr 15, 2016
Messages
2,187
Likes
2,431
The only forum I know to be quite close to ASR is hydrogenaudio. They are even more vocal/strict about things like "DBT or it did not happen". And it's ~deserted nowadays.
They do not have someone like Amir to feed an inhuman amount of info/tests. And those strict, hyper-skepticism policies drove people away. When you only accept DBTed posts, you only have one per month...

You exaggerate. Go there now and tell me how many new posts went up today.

What *is* true is that the majority of posts these days are foobar 2000 related. Nothing wrong with that, as it's a free and highly configurable audio player, and popular among a certain demographic. HA hosts the f2k discussion forum.

HA started out as a forum devoted mainly to improving lossy codecs, not a general audio forum. Inviting sighted anecdotes would have been useless for that purpose.

And of course, plenty of posts there, even before it became so F2K centric, didn't 'require' DBTs -- because they weren't about a poster's claims of audio difference. A lot of posts were and are about how to make stuff work. Or how stuff worked. Including human hearing as well as gear as well as software.

I don't think this forum benefits from the endless rehashes of physically and psychoacoustically dubious claims any more that HA would have.
 
Last edited:

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,817
Likes
2,421
Location
Massachusetts
Um, right. Now explain how any of that contradicts or negates what I wrote.

OK, I found the context. The ABX is a methodology. Listen to A, listen to B, then listen to X that is A Or B.
If strictly adhered to, that requires 3 per test.

From what I understand it, Harman tests are blinded comparisons between speaker A and speaker B.
The listener selects tracks and switches as they like, having been trained they give their impressions of A/B.
There is no X where a speaker is presented that may be A or B.
This is a different methodology but also a blind test.
ABX in this type of test adds no value.

I have done single blinded tests at home, there is A and B and no X in the methodology.
In each listening session A or B may be either component and the user selects how long to listen and when to switch.
At the end, provides a preference if any and a description of the differences.

- Rich
 
Last edited:
OP
Blumlein 88

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
17,186
Likes
29,949
OK, I found the context. The ABX is a methodology. Listen to A, listen to B, then listen to X that is A Or B.
If strictly adhered to, that requires 3 per test.

From what I understand it, Harman tests are blinded comparisons between speaker A and speaker B.
The listener selects tracks and switches as they like, having been trained they give their impressions of A/B.
There is no X where a speaker is presented that may be A or B.
This is a different methodology but also a blind test.
ABX in this type of test adds no value.

I have done single blinded tests at home, there is A and B and no X in the methodology.
In each listening session A or B may be either component and the user selects how long to listen and when to switch.
At the end, provides a preference if any and a description of the differences.

- Rich
Should we discuss 2AFC, triangle and duo-trio testing next? :)
 

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,817
Likes
2,421
Location
Massachusetts

tonycollinet

Major Contributor
Joined
Sep 4, 2021
Messages
3,672
Likes
5,705
Location
UK/Cheshire
OK, I found the context. The ABX is a methodology. Listen to A, listen to B, then listen to X that is A Or B.
If strictly adhered to, that requires 3 per test.

From what I understand it, Harman tests are blinded comparisons between speaker A and speaker B.
The listener selects tracks and switches as they like, having been trained they give their impressions of A/B.
There is no X where a speaker is presented that may be A or B.
This is a different methodology but also a blind test.
ABX in this type of test adds no value.

I have done single blinded tests at home, there is A and B and no X in the methodology.
In each listening session A or B may be either component and the user selects how long to listen and when to switch.
At the end, provides a preference if any and a description of the differences.

- Rich
The point of ABX is to detect if there even is an audible difference to be preferred, or described. It is not intended to determine preference.

If you can't decide reliably if x is A when it is, or B when it is, then it demonstrates that you are unable to detect the difference between A and B - any difference if it exists is below the level of audibility for the listener. Or conversely it demonstrates that you can.

The listener can still decide how long to listen to a/b/x, and can switch back and forwards between them as much as, and for as long as they like.

Once you know that there is an audible difference, you can go on to test for preference or description of the difference.
 

krabapple

Major Contributor
Joined
Apr 15, 2016
Messages
2,187
Likes
2,431
OK, I found the context. The ABX is a methodology. Listen to A, listen to B, then listen to X that is A Or B.
If strictly adhered to, that requires 3 per test.

You quoted Wikipedia's correct statement:
"An ABX test is a method of comparing two choices of sensory stimuli to identify detectable differences between them. A subject is presented with two known samples (sample A, the first reference, and sample B, the second reference) followed by one unknown sample X that is randomly selected from either A or B."

That was in response to me writing (emphasis added here):
It doesn't really add a third. The X is either A or B.

I.e....clearly I know that an X is involved.
ABX 'requires' 3 'samples', but there are only 2 potentially distinct stimuli.

From what I understand it, Harman tests are blinded comparisons between speaker A and speaker B.
The listener selects tracks and switches as they like, having been trained they give their impressions of A/B.
There is no X where a speaker is presented that may be A or B.
This is a different methodology but also a blind test.
ABX in this type of test adds no value.

Uh huh. Which is quite aligned with what I wrote: Harman's blind speaker tests don't employ an ABX protocol. ABX is inappropriate for tests of preference.

I have done single blinded tests at home, there is A and B and no X in the methodology.
In each listening session A or B may be either component and the user selects how long to listen and when to switch.
At the end, provides a preference if any and a description of the differences.

- Rich

Uh huh.

I have no idea what you think you are disagreeing with, in any of this. Wikipedia does not disagree with me; you don't seem to, either.

Let's stop there.
 

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,817
Likes
2,421
Location
Massachusetts
The point of ABX is to detect if there even is an audible difference to be preferred, or described. It is not intended to determine preference.

If you can't decide reliably if x is A when it is, or B when it is, then it demonstrates that you are unable to detect the difference between A and B - any difference if it exists is below the level of audibility for the listener. Or conversely it demonstrates that you can.

The listener can still decide how long to listen to a/b/x, and can switch back and forwards between them as much as, and for as long as they like.

Once you know that there is an audible difference, you can go on to test for preference or description of the difference.

Understood. I am addressing this from the original post
Seems increased commentary in recent weeks about ABX tests. Much of it stemming from people who come to ASR to set us straight about trusting our ears. I do agree with some who have said that calls for ABX or it didn’t happen have become almost like a club to beat people over the head with, and nearly cultish in how some new posters have the call rain down upon them. Not that I haven’t been guilty of it myself.

Some comments by @restorer-john have caused me to think about this situation. We stand little chance of convincing, or engaging in meaningful discussion with people with this approach. Like restorer-john I think there is a lot more talk of it than participation in or use of ABX listening tests among most posters. For most audiophiles it is impractical for most situations.

Some who don’t like ABX tests complain they are stressful. Only if you feel challenged by it or think you’ll suffer loss of face. After you have done it a couple or three times it isn’t stressful. It is major league TEDIOUS and BORING. Most of us do them with Foobar ABX or similar software. That isn’t very useful for amps and not at all for speakers.

So what is a next best alternative? What is a friendlier way to get the point across? How do regular ASR members pick their gear?

Blind tests are the best most discriminating method. I find I can detect with 100% reliability some very small differences when using two segments of 5 seconds or less and rapid switching. OTOH, some of those I score 50/50 if segments are 15 or 30 seconds long. I have found anything I only hear using the very short segments which both can fit inside my Echoic memory are so small they have zero relevance to normal music listening. So on one hand if you cannot hear something using short rapid switching listening tests it is a pretty sure bet you cannot hear it. On the other if the difference isn’t large enough to hear with 30 second segments it isn’t big enough to matter for music listening.

I believe the #1 thing to emphasize with any comparative listening is you must match levels precisely. Set a comfortable listening level and measure voltage of test tones at speaker terminals so each component matches within 1%. You cannot do any useful listening comparisons without this step. This one thing even in sighted listening can cause people to experience the disappearance or large reduction in differences they thought they were hearing.

The #2 thing to make clear is that fairly small deviations in frequency response are audible. So checking that might eliminate any need to go further for differences you hear. There are some simple ways to test this.

So what other things can we do or that some of you do that is useful? What is a more effective way to engage people who don’t understand things about what can and cannot be heard without chiming in over and over “hey, do an ABX test or it didn’t happen”

This thread discussed the validity of components evaluation using fast switching when level matched.
If the listener is unable to find a meaningful difference our mis-identifies the source when unblinded, that is the result.

I have used the MinniDSP dual output device to compare amps because there is a single source with DSP level matching to .1 dB.
This works well for A/B comparison that is not ABX.

There is a value to such tests to select gear, IMO, of course.

- Rich
 
Top Bottom