The ABX test of ABX tests

[Talisman]

Hello, I have always thought, even before starting to read about ASR, that the only way to choose my audio correctly is to blindly test the various components without being influenced by preconceptions.
Recently, however, I have also had some doubts, if there really exist nuances that we cannot grasp in instant ABX but that could have long-term influence in listening? I'm not asking because I'm against science, but on the contrary, because I like to check every statement.

I therefore thought of a possible ABX test of the ABX tests, let's take it for granted that if we cannot distinguish two songs performed with some difference in ABX obviously we must consider them equivalently for us, and therefore adding various modifications, each of them not differentiable in ABX, the result should always be the same.

Let's start from the listening file, take a good file in flac and compare in ABX with gradually worse lossy files (mp3 320, 256, 125, 96 etc ...) and find out which is the first file in which we can distinguish the difference and let's go to the next higher step (if we can see the difference in the mp3 at 256, we will take the mp3 320 as a good file for the test)
We pass to the next level, the transmission of the file to a dac, I used or as a reference the cable and then we test the bluetooth ldac 990, then aptx HD, then apts / aac then sbc, also in this case we identify the level at which we can hear the difference and we go to the next step, if we hear difference with aptx hd then we will take LDAC as protocol for the test.
Let's go to the Dac, we will use a definitely transparent dac and then we will gradually try less performing dacs until we get to the poorest and worst measured converters from a few euros that are found on amazon, also here the same procedure, we will use the poorest among the unrecognizable ones.
Finally we go to the amp, starting with something that is certainly excellent (perhaps a hypex module) and gradually descending among the many small amps that many of us surely have at home, arriving at the worst implementation of TDA 7498 or similar.
At this point we create a listening chain that has everything at the highest level (flac file, cable connection, transparent dac, top amplifier) and on the other side the chain with all the poorest elements but that I cannot identify in abx (mp3 320kbs, ldac, 45 euro nobsound dac, breeze amp with tpa3116d2 .... For example) and we send everything to the same speakers used for the other single tests.
At that point we proceed to the real test, accumulating all these differences, are the two listening chains still indistinguishable? If yes, then we can be reasonably sure our abx tests are worth it, but what if we could feel the difference? We should take into consideration the possibility that there are nuances that we cannot immediately or individually grasp but which add up to make an audible difference. At that point what should we do
Do you think it makes sense? Could you try?

 

[SIY]

Hello, I have always thought, even before starting to read about ASR, that the only way to choose my audio correctly is to blindly test the various components without being influenced by preconceptions.
Recently, however, I have also had some doubts, if there really exist nuances that we cannot grasp in instant ABX but that could have long-term influence in listening? I'm not asking because I'm against science, but on the contrary, because I like to check every statement.

I therefore thought of a possible ABX test of the ABX tests, let's take it for granted that if we cannot distinguish two songs performed with some difference in ABX obviously we must consider them equivalently for us, and therefore adding various modifications, each of them not differentiable in ABX, the result should always be the same.

Let's start from the listening file, take a good file in flac and compare in ABX with gradually worse lossy files (mp3 320, 256, 125, 96 etc ...) and find out which is the first file in which we can distinguish the difference and let's go to the next higher step (if we can see the difference in the mp3 at 256, we will take the mp3 320 as a good file for the test)
We pass to the next level, the transmission of the file to a dac, I used or as a reference the cable and then we test the bluetooth ldac 990, then aptx HD, then apts / aac then sbc, also in this case we identify the level at which we can hear the difference and we go to the next step, if we hear difference with aptx hd then we will take LDAC as protocol for the test.
Let's go to the Dac, we will use a definitely transparent dac and then we will gradually try less performing dacs until we get to the poorest and worst measured converters from a few euros that are found on amazon, also here the same procedure, we will use the poorest among the unrecognizable ones.
Finally we go to the amp, starting with something that is certainly excellent (perhaps a hypex module) and gradually descending among the many small amps that many of us surely have at home, arriving at the worst implementation of TDA 7498 or similar.
At this point we create a listening chain that has everything at the highest level (flac file, cable connection, transparent dac, top amplifier) and on the other side the chain with all the poorest elements but that I cannot identify in abx (mp3 320kbs, ldac, 45 euro nobsound dac, breeze amp with tpa3116d2 .... For example) and we send everything to the same speakers used for the other single tests.
At that point we proceed to the real test, accumulating all these differences, are the two listening chains still indistinguishable? If yes, then we can be reasonably sure our abx tests are worth it, but what if we could feel the difference? We should take into consideration the possibility that there are nuances that we cannot immediately or individually grasp but which add up to make an audible difference. At that point what should we do
Do you think it makes sense? Could you try?

A little searching will turn up strong evidence that rapid switching gives better detection sensitivity that long-term with slower changeovers. And several people have already done the "compare an entire chain of electronics with front-to-back cheap to front-to-back high end."

General advice: Pick a hypothesis, state it clearly and unambiguously, then run an experiment to attempt to falsify it before you decide what the follow-up is. It's often hard, sometimes impossible, to get people to do this. But it's the key to doing good and useful experiments. See this for some examples.

 

[DVDdoug]

t we cannot grasp in instant ABX but that could have long-term influence in listening?

There's no time limit for ABX. In general you can listen as long you want and you can switch between A, B, and X until you make a decision, or until you just give-up and guess, or until you give-up and stop the test.

But, it has been shown that you're generally more likely to hear a difference if you can switch quickly. After a day, or a week, it becomes harder. It can be worthwhile to listen long enough to hear a difference or a defect. For example, certain sounds are hard to compress and you may have to listen for awhile before you hear a passage where the compression artifact shows-up.

And you could argue that the longer-term less-sensitive listening test is more useful. If you have two amplifiers and you can't tell if you're listening to A or B the next day it doesn't make sense to buy the more expensive one...

I'm not asking because I'm against science, but on the contrary, because I like to check every statement.

ABX tests ARE scientific (if done properly). And in fact, MP3 (or other lossy compression) artifacts are often time-related* and they don't necessarily show-up in traditional noise, frequency response, and distortion measurements. Sometimes people look at the spectrum of an MP3 and see a loss of high frequencies but with good-quality (high-bitrate) MP3s, the loss of highs isn't normally the kind of artifact that you hear. Other lossy formats may retain the high frequencies but still have compression artifacts.

* There is an MP3 artifact people call "pre-echo" and as far as I know it doesn't go-away at higher bitrates.

 

[amirm]

Recently, however, I have also had some doubts, if there really exist nuances that we cannot grasp in instant ABX but that could have long-term influence in listening?

As noted, this has been studied. See: https://www.audiosciencereview.com/...ity-and-reliability-of-abx-blind-testing.186/

 

[Talisman]

I actually expressed myself badly, rather than "long-term listening" I meant to focus attention on any tiny differences that are not perceptible in a single ABX test (if they actually exist) but that eventually added together can lead to an audible difference even in ABX.

ABX tests ARE scientific (if done properly).

Certainly it is, this is absolutely not in question, I was simply wondering how sensitive it was as a test with respect to our auditory perception. If I don't recognize a 320kbs mp3 file from a flac is that all there is to it or can there be nuances that can jump out? (if as in the example of the proposed test they are added together), but I repeat, for the avoidance of doubt, I DO NOT HAVE A THEORY ABOUT IT, and that is why I would like to perform the test (but I do not have sufficient equipment and preparation for such a complex ABX test)

 

[Keened]

A little searching will turn up strong evidence that rapid switching gives better detection sensitivity that long-term with slower changeovers. And several people have already done the "compare an entire chain of electronics with front-to-back cheap to front-to-back high end."

General advice: Pick a hypothesis, state it clearly and unambiguously, then run an experiment to attempt to falsify it before you decide what the follow-up is. It's often hard, sometimes impossible, to get people to do this. But it's the key to doing good and useful experiments. See this for some examples.

I wish Wavelet had ABX modes built in: a timed ABX swap test (10s, 10s, etc) and a continual generational one where it tweeks random things and you just say whether it sounds better or worse and iterates over that.

I think that would let people really understand why ABX tests works, it would give them inductive proof that ABX generates psycho-accoustically better results and then they could use that sense proven curve to measure for objective results. If it is possible to generate better results through ABX, and you can see that ABX gives you a more approximate conformance to the reference signal, then it's a very small step to say that you should probably be using ABX to do course correction towards your target.

It can't tell you if something is 'correct' or not, it isn't deductive. But it has shown that it gets you closer to whatever correct appears to be (what is perceived to be correct ('transparency'/auditory hallucination/detail retrieval/emotional resonance/whatever) and what we are referencing as correct (the source signal). We have a partially bounded region now, even if we don't have a full picture. Like how entire genres are made in mind with cannibus use to distort our perception of time or were made specifically to be sung while plastered. So you could get the physical sound reproduction right but the psycho-accoustics would always be off.

So we haven't triangulated/platonically defined perfection or said that any other standard is inherently wrong; and this method can't absolutely disprove all other claims, but it can say whether or not claims are significant within these boundaries. If you are interested in pursuing this target (i.e. being an audiophile) then you must accept it or submit your counter method for testing. Your method may in fact work better for you, but if we accept that target is at least partially outside ourselves then it must be accessible to others as well.


------------

An ABX Expo where you had speakers on some kind of multi-height adjustable lazy susan so you could rotate the speakers into the exact same position each time already corrected for height would be cool. The issue is getting the power to the speakers but inductive sliprings are within budget if you're at the point where you are willing to create a fake wall to lift and drop behind the speakers each time.

Leave it for a year and do a tournament style shootout.

 

[Keened]

I DO NOT HAVE A THEORY ABOUT IT, and that is why I would like to perform the test (but I do not have sufficient equipment and preparation for such a complex ABX test)

I know, a little bluetooth triggered 2x2x2 source, line level, and speaker level matrix would be amazing for that.

 

[Newman]

Hello, I have always thought, even before starting to read about ASR, that the only way to choose my audio correctly is to blindly test the various components without being influenced by preconceptions.

??? The only way that would make sense would be if you intend to do all your listening to recorded music in a scenario where you don’t know the gear being used.

Which would be a bizarre scenario. It would entail something like someone else buying all your playback gear, their choice not yours, not tell you what they bought, put it all in a black box with a disc tray at the top and wires (hidden in tubes: don’t want the wires to be recognisable) out the bottom, going to speakers behind acoustically transparent screens that you are committed to never peep behind.

And as much as I do think that scenario has a kind of fundamentalist purity to it, it would take a special kind of dedicated fundamentalist to implement it. In reality, it just isn’t what we do.

In any other, more normal scenario, choosing blind, then listening at home sighted, is kind of self-defeating. It is a case of cutting off your nose to spite your face. Because we actually do listen at home in sighted listening conditions, our ‘preconceptions’ as you call them, along with other non-sonic perceptual mechanisms, are in play, and if they ‘don‘t like’ what they ‘see’, your perceptions of the sound will suffer accordingly.

Those of us, including me, who are interested in pursuing the most-preferred sound waves alone, are gently dismayed by this truth, but it’s there and it plays out in all our homes. We can each reach our own accomodation with it. Some will willingly cut off their nose. Some will choose by sighted listening but only from a sub-list of gear that meets performance criteria that correspond with being preferred for sound waves alone (from blind testing) at reasonable cost. And some will sucker-punch themselves and their bank accounts in a purely hedonistic pursuit dominated by non-sonic influences, and if that means their (our) easily-influenced minds tell them/us that it’s “oooh, so much more organic and real” coming from Mikey F’s cartridge of the month through a $60,000 phono preamp that is advertised in this month’s edition, $4,000 power cord, and legacy legendary Altec fullrange driver in a Voigt pipe, then I understand this is rational and good luck to them and I sure hope their family’s happiness wasn’t cremated on the altar.

As long as the latter group don’t run around the audio forums insisting that the sound waves from such gear are the source of their ecstatic pleasure and vastly audibly superior to the sound waves from equal-or-better-measuring gear at much less cost, or insisting that digititis is real because they can hear it, and if we object, we are told we can’t hear it because we lack the refined hearing that they have (along with Mikey F).

For those in the middle category (choosing gear via subjective listening from a sub-list of objectively well-performing gear), and I seem to find myself there, the sub-list is pretty huge because it includes all well-performing electronics (and the sensible option to choose from reliable brands with good support and reasonable prices is well-represented on the sub-list), many speakers at all price ranges that have smooth, flat, extended frequency response and well-controlled directivity, and a clear understanding that the sound waves will be preferred from multi-channel, multi-sub setups that are intelligently measured and equalised in-situ. If we choose from that sub-list based on our personal sighted listening with non-sonic influences in play, accepting that it isn’t the sound waves causing the preferences but also knowing that the sub-list is preventing us from falling into Alice’s rabbit hole, then that is also a rational way to deal with the conundrum that the reality of sighted listening at home presents.

cheers

 

[magicscreen]

Do you think it makes sense?

No. People usually cannot make a difference between 320k mp3 vs FLAC.
It is a scientific fact that there is difference. But you cannot prove it in blind test.
So any blind testing is an invalid, failed, wrong method.

"being influenced by preconceptions"

Why is that a problem? If you buy a golden colored amplifier and you hear better sound quality because of that, be happy, and enjoy the music.
Do not listen to these "have a bad hearing" people here.

 

[Talisman]

"No. People usually cannot make a difference between 320k mp3 vs FLAC.
It is a scientific fact that there is difference. But you cannot prove it in blind test.
So any blind testing is an invalid, failed, wrong method."

I don't think I understand your point of view, forgive me, it's probably my fault and my bad English translated automatically.
I don't know if normally people can blindly hear the difference between mp3 320kbs and Flac, I'm interested in knowing if I CAN DO IT.
And the next step will be: "What is my minimum quality threshold at which I can feel / not feel the difference?
And the next question will still be: Between the two steps of can / cannot hear the difference is there a nuance that added to other nuances can make an audible difference in the blind?

Having said that I am a complete objectivity, my objectivity also makes me consider myself to be a human being and as such subject to perception bias, but I can use it in my favor, between two instruments that I can rationally sound the same to my ears, I can choose the one which I like best because I know this will affect my perception of quality and pleasantness on an unconscious level, (but it definitely won't make me buy a very expensive and useless boutique dac that will only empty my wallet)

 

[BDWoody]

No. People usually cannot make a difference between 320k mp3 vs FLAC.
It is a scientific fact that there is difference. But you cannot prove it in blind test.
So any blind testing is an invalid, failed, wrong method

I'm not following you here...

If one can't tell the difference in a controlled test, that isn't supposed to mean there is no difference, just that it isn't audible to the person taking the test.

What's invalid, failed and wrong about that?

If you buy a golden colored amplifier and you hear better sound quality because of that, be happy, and enjoy the music.

Do you believe there is any merit at all to measuring performance, or trying to understand how we process sound?

 

[xaviescacs]

just that it isn't audible to the person taking the test

Please correct me if I'm wrong.

Wouldn't it be more precise to say "consciously audible" (or something of the sort)? We perceive a lot of things that our brain processes, that we aren't consciously aware of, yet we "hear" them. Hence, I would say the majority of tests are really about someone's ability to distinguish certain aspects of sound, rather than absolute human hearing.

To me, an inaudible level of distortion is one that no human has been able to hear. If one human prove to be able to hear it, then it's audible. Then, we can test individual subject's ability to hear/identify it, but then we are testing each subject's ability and hearing capabilities combined.

 

[DonH56]

What I have done in the primordial past was to repeat AB/ABX testing after long-term listening tests. The idea is (was) that listeners identify certain sonic attributes they attribute to the components after a "long" (hours to months) time of listening that they claim are not heard in their initial listening tests. I invited attendees (myself including) to note what musical passages showed the differences and then we used those in blind tests. They were AB at first, did not have an ABX setup initially (*).

Now for the part that is just a foggy old memory of testing done in the early 1980's. The results when folk focused on the passages that clearly showed differences between two components were mixed IIRC. Some still disappeared under blind testing but some remained. Of the ones that remained, most were the same as the initial testing -- IOW, a difference that showed in in the initial testing, was still there after extending listening at home. Again IIRC, the couple of "real" differences we found after long-term listening (**) were the result of power amplifiers interacting with certain speakers, where certain material would bring out differences not identified in the initial tests due to changes in source material.

The one clear take-away I recall is that using pink noise was reliable and predictive of differences for listening short or long in term. That is, using pink noise in initial testing was enough to clearly and reliably identify any differences, whereas random musical selections (this was before HT) was more hit or miss. Seems obvious now, but at the time surprised some folk and caused me to always include pink noise in any round of AB/ABX tests after that.

FWIWFM - Don

(*) I started with an AB system I designed; the store I worked later bought an ABX text box. My AB box had a little "noise" circuit so when I pushed a button it randomly selected A or B. It had LEDs to show if A or B was selected with a switch so I could turn them off when I was running the test. Logic kept track of how many tests (up to 25 IIRC) and whether A or B was selected for each, again using rows of LEDs (didn't have fancy displays) that were turned off during the test. At the end, I'd flip the switch, and two rows of LEDs would show the AB switch positions during the test (top row A, bottom row B). I had to manually record that and the individual responses from the listeners (written on notepads) to correlate the results. It was a bit of a pain... I had a program written for a VIC-20 (anybody remember those?) but graduated before it saw any use.

(**) Note that in this context, long-term listening meant components people had generally bought and taken home to use in their own systems for a while, comparing them to their previous (or just other) components. They brought them back into the store, or I took the AB/ABX test rig (and my pink noise generator, a little battery-powered unit) to their house (more common), and repeated the testing after they had lived with the components a while. Also note the test, like any AB/ABX test, was to identify differences and not preferences among components.

 

[levimax]

I would recommend spending some time doing ABX tests of different files of the same music to get used to ABX testing and then try a piece of hardware. I think you will find all of of your "doubts" will disappear and you will gain a lot of confidence and understanding of what you can hear or can't hear and what is important. Reading about ABX testing is not the same as doing it yourself, I found it to be well worth the effort.

 

[Talisman]

I would recommend spending some time doing ABX tests of different files of the same music to get used to ABX testing and then try a piece of hardware. I think you will find all of of your "doubts" will disappear and you will gain a lot of confidence and understanding of what you can hear or can't hear and what is important. Reading about ABX testing is not the same as doing it yourself, I found it to be well worth the effort.

I have tried to do as many ABX tests as possible, precisely because I really like to understand what I can and cannot hear, regardless of any bias. I discovered, for example, that I can not hear the difference between an E50 topping and a fiio taishan, but that I can feel a distortion at - 51db, this information is precious for me to understand where to invest the money to have a real improvement of the audio performances.
From this comes my post, the idea of a test that collects in a single ABX all the boundary results to understand if, added up, they can still give an audible difference in ABX that they do not have as a single component. An ABX of an entire chain, but calibrated on individual perception with on one side the whole top, on the other everything that falls by a hair within the threshold of inaudibility of the single factor

 

[levimax]

The distortion of the highest distortion component in the chain dominates... either speakers or possibly a TT. See the bottom of link to see how distortion adds up http://www.sengpielaudio.com/calculator-thd.htm

 

[xaviescacs]

I can not hear the difference between an E50 topping and a fiio taishan, but that I can feel a distortion at - 51db, this information is precious for me to understand where to invest the money to have a real improvement of the audio performances.

This is a bit off-topic again, but that was my point actually. If you can't differentiate them, are you going to trust your ears and training and buy the fiio?

As said before many times, you can't prove you can't hear the differences. All you can do is try to hear them, again and again, and by doing so, you will be restricting the possibilities of a positive result. Which translated means that may not be able differentiate them right now but you may be able to do so at some point due to more training, experience, better designed test, etc. Furthermore, if that eventually happens, it would mean you were hearing those differences all the time, because they are audible.

This is where amir's approach with SINAD and transparency kicks in. If SINAD is low enough, it's transparent and we don't need to trust our ears nor our hearing and test design capabilities.

Bottom line, as i see it, if you buy the E50 instead of the taishan, there will be a "real improvement" of the audio performance, because it can be measured, and the differences aren't that small. The implications of a negative test result on your part are pretty narrow and don't discard by any means that the differences between them aren't audible.

 

[Talisman]

Bottom line, as i see it, if you buy the E50 instead of the taishan, there will be a "real improvement" of the audio performance, because it can be measured, and the differences aren't that small. The implications of a negative test result on your part are pretty narrow and don't discard by any means that the differences between them aren't audible.

But if I can't perceive a difference or distortion at - 60db, why should there be a difference to me between a - 90/100 and a - 125 sinad dac?
Once I reach what may be, beyond a reasonable doubt, my perception limit, the rest of my money can go for functionality or aesthetics.
The concept of my test is precisely not to stop at the door of my abilities, but to reach what are beyond any doubt my limits of perception.
For example, I have a Cambridge audio cxa61, recently tested in its cxa81 form with bad results, well, I didn't hear any problems in my ears, because clearly the limits of the amplifier, although not state of the art, are still higher than my ability to perceive differences. However after seeing the review I tried to put the volume to maximum with no signal coming in and I actually noticed a slight rustle, but it never bothered me, nor had I ever noticed.

I want to make a different speech, and perhaps poorly digested on this site, but which if you think about it carefully is extremely rational.
If I have 1000 euros available for an audio setup, should I buy a Topping E50 because it is "state of the art" or should I save and buy a cheaper dac but I DEFINITELY can't hear a difference with?
Or better still, if between two dacs in which I cannot, beyond all doubt, perceive a difference, it makes sense to buy one just because it has a little better specifications instead, for example, of a set of additional features, of an additional digital input. or even just simply an aesthetic that I prefer?
Once the audible limit has been exceeded, what is the added value of a more performing instrument from an acoustic point of view for me?
Instead I could have a lot of added value from different functionalities, from a better structured remote control, or from an aesthetic that I like the most. I think this is extremely rational as a speech

 

[JSmith]

It's amusing how some people say "I trust my ears"... yet when it comes to blind ABX, suddenly it's the test that is wrong and the ol' trustworthy ears go out with the bathwater? If one trusts their ears, then that should apply in a blind ABX test too.


JSmith

 

[Talisman]

It's amusing how some people say "I trust my ears"... yet when it comes to blind ABX, suddenly it's the test that is wrong and the ol' trustworthy ears go out with the bathwater? If one trusts their ears, then that should apply in a blind ABX test too.


JSmith

Are you referring to something I said?
I have never thought of having golden ears, indeed I candidly admit to having evident perceptual limits, this is precisely the point.
I absolutely agree with the blind ABX test, what I'm saying is different, it concerns any subtle differences between two steps that blind can be / cannot be heard and the possibility that adding them together they are heard, but always in ABX.
I feel like I can't explain what I mean, probably because I don't speak in my native language