Relevance of Blind Testing

[polmuaddib]

Recently, I watched a video of John Atkinson's RMAF 2018 presentation "50 years being an audiphile" and he told one interesting story among other things. He said that he attended DBT of amplifiers in the 70s, if i remember correctly, where they basically concluded that all amps sound the same. But after a while, having a regular Quad amp, he found that he didn't enjoy listening to music anymore. So, he got himself a new system, reignited the passion for music listening again and deduced that DBT has some flaws.
That story got me thinking on what happens after blind testing. I am not really questioning DBT, because i know it is an essential scientific tool.
But, let's say you have a sighted test of two amps (could be preamps, dacs... doesn't matter) and you like the sound of A amp better then B amp. Now you do the DBT of those same two amps, and most likely you can't hear the difference. They sound the same. Now when you do the sighted test again, will you like the same A amp sound better or not? Will you now be illuminated (for the lack of more appropriate word) and never have bias again when sighted testing? Or you will have bias, but not for those two particular amps?

 

[andreasmaaan]

In a nutshell: participation in a blind test won't necessarily eradicate psychological factors beyond the blind test itself (not even with regard to the specific components used in the blind test).

 

[polmuaddib]

Thanks. So if you look at it from a subjectivist view, when choosing new hifi component, it is important to do a sighted test. Because, in the end, you are gonna look at it and listen to it and attribute all those qualities that the brain assigns to the sound.
Objectively two devices can sound the same, even measure the same, but you do hear a difference when you know what component you are listening and this difference can't be measured. But it is there all the same. It is in our heads, but it is real, right?

 

[solderdude]

Thanks. So if you look at it from a subjectivist view, when choosing new hifi component, it is important to do a sighted test. Because, in the end, you are gonna look at it and listen to it and attribute all those qualities that the brain assigns to the sound.
Objectively two devices can sound the same, even measure the same, but you do hear a difference when you know what component you are listening and this difference can't be measured. But it is there all the same. It is in our heads, but it is real, right?

I would agree but not that differences are real.
Only that the mind makes it (seem) 'real' for the owner.

When one get's more 'enjoyment' from a system then from another (even when secretly the actual insides are the same) then I see no reason not to go for the system that makes you enjoy it more.
Enjoyment is not just audio. It is the package.
When the owner thinks it is worth the price I am all for it. Even when it is full placebo.

That said.. I do like to know myself whether or not I am fooling myself and don't like to fool others nor myself and this is where measurements and nulling comes in handy.
The brain cannot be trusted but we have one and is all we can use. Enjoyment is not just sounds it's the total package.

 

[Frank Dernie]

Thanks. So if you look at it from a subjectivist view, when choosing new hifi component, it is important to do a sighted test. Because, in the end, you are gonna look at it and listen to it and attribute all those qualities that the brain assigns to the sound.
Objectively two devices can sound the same, even measure the same, but you do hear a difference when you know what component you are listening and this difference can't be measured. But it is there all the same. It is in our heads, but it is real, right?

Indeed.
The placebo effect is very real, yes, and powerful.
A sugar pill can actually cure several illnesses if the subject believes it will work, so it is entirely likely that a piece of hifi will sound better to somebody who expects it to do so.

 

[polmuaddib]

Exactly.
But bias is not in our control. For instance, I like the esthetics of power cables and i would very much like that they have some impact on my sound. But they don't. I tried some (not all, not extremely expensive) and I could never hear a difference when I focus on specific audio band one at a time.
No cables do it for me, unfortunately. Maybe, because I never believed in cables.

 

[Frank Dernie]

Exactly.
But bias is not in our control. For instance, I like the esthetics of power cables and i would very much like that they have some impact on my sound. But they don't. I tried some (not all, not extremely expensive) and I could never hear a difference when I focus on specific audio band one at a time.
No cables do it for me, unfortunately. Maybe, because I never believed in cables.

Precisely. If you don't believe it can work it won't but if you do it may.

 

[charleski]

Any experimental design contains multiple components. Double-blind is only one of those components, but people tend to get stuck on that and ignore other elements that might be important.

One example is the amount of time given to each presentation. If one amp has a bad FR anomaly it may be trivial to distinguish it from a better amp after listening for a second. But most of the flowery language used by the typical audio reviewer is referring to high-order cognitive effects that are processed many layers up the chain and take time to evolve. This can be shown even with a simple contour identification test, in which tasks that require switching attention between different auditory objects result in over a doubling of response time. If (let’s say) there were indeed some currently-unexplained phenomenon that resulted in different amps evoking different high-order cognitive effects, then it may take a considerable length of time for that response to evolve to a level of statistical significance. Furthermore, since the source of most changes in these effects probably lies in internal fluctuations in mood, it might be necessary to accumulate a large number of presentations spaced out over weeks or months in order to average out these fluctuations.

Subjectivists who fail a double-blind test and are flailing around looking for an explanation are barking up the wrong tree by trying to criticise the double-blind element, which is absolutely necessary. But it’s possible that other elements of the experimental design might be failing to provide sufficient power to expose the (extremely subtle) effects they claim to hear. As scientists we have to keep an open mind, which is why I’m posing the possibility. It’s not very likely, though, and I haven’t seen any evidence to suggest what these elements might be (the CIT referred to above still had response times on the order of seconds).

 

[TSB]

Also, the placebo effect doesn't go away just because you know about it.

 

[andreasmaaan]

Any experimental design contains multiple components. Double-blind is only one of those components, but people tend to get stuck on that and ignore other elements that might be important.

One example is the amount of time given to each presentation. If one amp has a bad FR anomaly it may be trivial to distinguish it from a better amp after listening for a second. But most of the flowery language used by the typical audio reviewer is referring to high-order cognitive effects that are processed many layers up the chain and take time to evolve. This can be shown even with a simple contour identification test, in which tasks that require switching attention between different auditory objects result in over a doubling of response time. If (let’s say) there were indeed some currently-unexplained phenomenon that resulted in different amps evoking different high-order cognitive effects, then it may take a considerable length of time for that response to evolve to a level of statistical significance. Furthermore, since the source of most changes in these effects probably lies in internal fluctuations in mood, it might be necessary to accumulate a large number of presentations spaced out over weeks or months in order to average out these fluctuations.

Subjectivists who fail a double-blind test and are flailing around looking for an explanation are barking up the wrong tree by trying to criticise the double-blind element, which is absolutely necessary. But it’s possible that other elements of the experimental design might be failing to provide sufficient power to expose the (extremely subtle) effects they claim to hear. As scientists we have to keep an open mind, which is why I’m posing the possibility. It’s not very likely, though, and I haven’t seen any evidence to suggest what these elements might be (the CIT referred to above still had response times on the order of seconds).

This is why things like e.g. training and fast switching times are important. If they’re to have the best chance of success, subjects should be given a chance to (try to) learn how to identify differences under sighted conditions prior to testing, and should be able to switch at will and near-instantaneously between stimuli during the test.

 

[polmuaddib]

This is interesting about placebo effect not going away. So, even after I know that two amps sound the same, I will still have sound improvement (in my head) if i invest in the more expensive amp (usually the case and i just chose that criteria as an example)?
Other example is for DACs. We have lot of DACs measured here and everyone here knows that, past certain SINAD, they are transparent and sound the same. But if someone values high SINAD in a device, they will hear (in their head, of course) the difference between 121 SINAD and 105 SINAD as an improvement in SQ?
Or there might be more to this?
Do all of us have different audiophile belief systems in our subconscious?

 

[andreasmaaan]

This is interesting about placebo effect not going away. So, even after I know that two amps sound the same, I will still have sound improvement (in my head) if i invest in the more expensive amp (usually the case and i just chose that criteria as an example)?
Other example is for DACs. We have lot of DACs measured here and everyone here knows that, past certain SINAD, they are transparent and sound the same. But if someone values high SINAD in a device, they will hear (in their head, of course) the difference between 121 SINAD and 105 SINAD as an improvement in SQ?
Or there might be more to this?
Do all of us have different audiophile belief systems in our subconscious?

Yes, and other psychological factors that may not be so obvious may also come into play. For example, if the first time we hear a component we are having a great day in terms of hearing (well-rested, no wax in ears, good mood, etc.), this may colour our perception of this component into the future (and vice-versa, of course).

 

[solderdude]

An anecdotal story.
Some decades ago, in my still largely subjective years, I built a pre-amp with a switch on the back.
It contained 2 similar circuits but one had a cheap generic opamp (TL072) which wasn't properly decoupled and had a bunch of 'poor sound quality' capacitors in its signal path.
The other circuit was a (back then) well received OP27 opamp with proper decoupling and no capacitors in the audio path.
I used it to demonstrate to visitors that there were differences in sound. Of course I told them what circuit was in there.
It was clear that the better opamp circuit sounded 'better'.

One day I forgot to switch it back to 'HQ' and left it on the poor quality position.
For about half a year I listened to it 'knowing' it was in the good sound position.
Sounded marvelous to me all the time.
One day when someone came by to listen I wanted to demonstrate the 'sound quality of poor non biased electrolytic caps and crappy opamps' only to find out it was in crap position.

That was the day I started to investigate my hearing abilities, built null testers and AB boxes and started testing blind and level matched and started testing other people (them not knowing), built a 'tube amp' that secretly was a crappy 4 transistor design etc.

The brain is a funny thing.

 

[jae]

People fail to realise how much a cup of coffee can magically change their sound system

 

[charleski]

This is why things like e.g. training and fast switching times are important. If they’re to have the best chance of success, subjects should be given a chance to (try to) learn how to identify differences under sighted conditions prior to testing, and should be able to switch at will and near-instantaneously between stimuli during the test.

It really depends on what sort of difference you’re testing for. If you want to test discrimination of low-level sensory features then rapid-switching and relatively brief presentations are the way to go as our memory of such raw sensory experience decays rapidly. But if you’re interested in higher-order effects then it might be more powerful to use far longer presentations, as memory of such mental states can be encoded semantically and thus persist far longer. When designing such experiments it’s always important to have a very clear idea of precisely what you’re trying to test.

 

[BDWoody]

People fail to realise how much a cup of coffee can magically change their sound system

I guess it depends on where you pour it...

I don't think people realize how much almost everything impacts the perception of sounds.

 

[pkane]

Yes, and other psychological factors that may not be so obvious may also come into play. For example, if the first time we hear a component we are having a great day in terms of hearing (well-rested, no wax in ears, good mood, etc.), this may colour our perception of this component into the future (and vice-versa, of course).

The actual act of doing testing already colors our ability to hear differences. By paying attention to specific aspects of sound when evaluating some new piece of equipment, we focus and hear things differently than the last time we heard the same track, and this has little to do with expectation bias... except maybe that we are expecting to hear something different. Our ability to focus on specific elements of sound works against us in this case: we hear more soundstage, more detail, more microdynamics, etc., precisely because we focus our attention on these, trying hard to hear those aspects.

I don't know how many times I've introduced a change in my system and heard major, obvious differences on initial testing only to later not be able to detect these differences at all. Also, don't know how many times I introduced a system change and heard major difference only to find out later that I forgot to press the right button to make the switch between the two devices, and was still listening to the same device both times.

 

[BaaM]

I do not deny the psychoacoustic biases to which we are all subject, but I do not think that blind testing is systematically revealing. For example, I have failed quite a few blind tests on whiskeys, which are undoubtedly very different from each other. However, it's an experiment to be done, it's a lot of fun, but I don't draw any definite conclusions from it.
I had read an interesting paper that was shared here on slow listening if you're interested: https://www.audiosciencereview.com/forum/index.php?attachments/aes20547-pdf.42956/

Most people will reply that blind testing does not impose a time limit, but in fact, I have never found audio blind tests that extend over several weeks or even months, probably for practical reasons.

Regarding measurements, I have to quote Jonathan Novick: "not all that matters can be measured, not all that can be measured, matters".

 

[Speedskater]

John Atkinson has a vested interest in blind tests not working.

 

[andreasmaaan]

I don't know how many times I've introduced a change in my system and heard major, obvious differences on initial testing only to later not be able to detect these differences at all. Also, don't know how many times I introduced a system change and heard major difference only to find out later that I forgot to press the right button to make the switch between the two devices, and was still listening to the same device both times.

Have had exactly this experience many times when mixing/mastering music and adjusting a parameter that later turned out not to be in the signal chain of the track I thought I was working on.

It really depends on what sort of difference you’re testing for. If you want to test discrimination of low-level sensory features then rapid-switching and relatively brief presentations are the way to go as our memory of such raw sensory experience decays rapidly. But if you’re interested in higher-order effects then it might be more powerful to use far longer presentations, as memory of such mental states can be encoded semantically and thus persist far longer. When designing such experiments it’s always important to have a very clear idea of precisely what you’re trying to test.

I agree, but that's why I said that the switching time should be rapid, but not necessarily the presentation time (which should be left up to the subject).

Having said that, I'm not aware of any evidence that audiophile-length presentation times (days, weeks or months) have ever been shown to allow for better discrimination. Has anyone come across any evidence of this?