Blind test - objectivists with tin hearing?

[sergeauckland]

But if the person have no idea whether A and B are identical and have no expectation of them being or not, then he/she is forced to actually listen and be (relatively) objective.

Anyone participating in a blind test will know at least what the object of the test is, For example, to choose whether is A better than B or are A and B same/different. It's also pretty likely they will be shown before the actual tests what A and B are, so they know what to listen for. Indeed before participating in any test, I would insist on knowing the procedure and methodology as I wouldn't waste my time on a flawed test. Furthermore, in an AB test, it's often deliberately done first sighted, so participants know what they're listening to, and can make their choice sighted, to be compared with their choice when the test is done blind.

None of this invalidates a blind test, it depends on the purpose of the test. Giving a totally blind test without even knowing what it is that's being tested isn't, I think, terribly useful or possibly even ethical except as a test of human psychology.

S

 

[Killingbeans]

Anyone participating in a blind test will know at least what the object of the test is, For example, to choose whether is A better than B or are A and B same/different.

I genuinely don't understand why that is necessary? If you can't detect someting without knowing what to look for, then why is that thing even worth spending time on?

It's also pretty likely they will be shown before the actual tests what A and B are, so they know what to listen for.

Then what's the point of the test in the first place?

Indeed before participating in any test, I would insist on knowing the procedure and methodology as I wouldn't waste my time on a flawed test.

That just makes you a bad test subject (no offence). Any test supervisor with respect for him/herself should refuse to give you that data.

Furthermore, in an AB test, it's often deliberately done first sighted, so participants know what they're listening to, and can make their choice sighted, to be compared with their choice when the test is done blind.

Again, why on earth would you do that? Deliberately contaminate the test subjects with heaps of expectations... I don't get it.

Giving a totally blind test without even knowing what it is that's being tested isn't, I think, terribly useful or possibly even ethical except as a test of human psychology.

Well, I see it the completely other way around. A totally blind test is the only way to get hard useful data. The other variants are fun and give insight to different parts of the human psyche, but they each introduce their own variables that make them impossible to converge into an overall conclusion.

 

[Blumlein 88]

I took part in several blind tests as a poor college student for spending cash. Always perceptual testing of one of your senses. Sometimes you were told what the testing was about. Sometimes not. Sometimes you could tell it wasn't what you were told. With proper controls any will give useful results. So the discussion about this is not really pertinent except for those who may have the wrong idea.

The big difference vs forum dbt talk? You never had skin in the game, no thought that failing the test was challenging your manhood, no idea that one result was better than another. When somebody says I can hear x and someone else says no you can't then it is a confrontational situation.

The other difference is no one told me my results. I asked sometimes and the only answer was your results were fine.

 

[SIY]

It's important to design a test to fit the specific question being asked. And equally important to define in advance what the criteria are- you can't go in post hoc and say, "Well, we got a null result for X, but unexpectedly, we see Y." You have to design a different test for Y. Example: let's say you want to have a group test of 100 people with 5 trials to distinguish between A and B, with the idea of seeing if overall differentiation exceeds 50% by a significant amount. If you get 50% +/- whatever deviation you deemed in advance as statistically insignificant, that's a null result. It is absolutely dishonest to go back, grab the two people who scored the highest, add their results together, and proclaim that the difference could be heard. If you want to say that their results are significant, you need to run tests specifically on them, and more than 5 trials to achieve any reasonable significance. After all, even if choices are completely random, you expect several people to score 5/5 and several people to score 0/0.

A common tactic of the non-rationalist objectors is that there has to be one and only one way for a sensory test to be structured, especially invoking universal requirements beyond double blind (I call this the Procrustean Fallacy). It's especially useful for then proclaiming that DBTs are difficult and expensive, so people need not do them to assert claims. Another common tactic is loosely related to my first point- if the experiment is designed to measure X, it is dishonest to object to it on the grounds that it doesn't measure Y.

I often ask non-rationalists, "What special insight do you get into sonics by using your eyes?" That is not a popular question. "Trust you ears" means "Trust your ears." No peeking. If you have to peek (or as the dishonest euphemism goes, "open listening"), you are doing the opposite of trusting your ears.

 

[mansr]

One way of mitigating bias against any perceptible difference is to include a third case, C, already known to be distinguishable from A or B, though not too easily. Instead of a plain ABX, you randomise each round as ABX, ACX, or BCX. Afterwards, the success rates of the three variants are compared. If AB is as easily distinguishable as AC or BC, the scores should be similar.

 

[SIY]

One way of mitigating bias against any perceptible difference is to include a third case, C, already known to be distinguishable from A or B, though not too easily. Instead of a plain ABX, you randomise each round as ABX, ACX, or BCX. Afterwards, the success rates of the three variants are compared. If AB is as easily distinguishable as AC or BC, the scores should be similar.

That will depend on the what specific hypothesis is being tested. Sometimes that's appropriate, sometimes not.

 

[Blumlein 88]

I've thought up down perceptual testing might be more convincing. You vary some aspect like noise. If you choose the noisy version noise is decreased. You continue until your success rate is random and you know your limit.

They problem with many audiophile ideas is they are untestable in this format. And even those that are won't be accepted. Frequency response is very amenable to this type test. But if it shows you can't hear when 14 khz is your limit with music you dismiss the test as wrong because you know nothing less than 192 khz sampling will do.

 

[SIY]

I've thought up down perceptual testing might be more convincing. You vary some aspect like noise. If you choose the noisy version noise is decreased. You continue until your success rate is random and you know your limit.

I outlined a variation of that in my Linear Audio article; I daisy chained op-amps and kept reducing the number to determine what my limits were to hear the difference.

 

[Blumlein 88]

I outlined a variation of that in my Linear Audio article; I daisy chained op-amps and kept reducing the number to determine what my limits were to hear the difference.

Yes I remember that article and have pointed others toward it. Good article and good approach.

 

[HammerSandwich]

What do you think of this?

I think that if the tester had changed nothing from Test A to Test B, the objectivists' results would have been exactly the same. And the subjectivists' would have, too.

 

[DDF]

Yes I remember that article and have pointed others toward it. Good article and good approach.

And to help make these difference tests more statistically valid, see DMOS methods in ITUT p.913.

 

[GGroch]

Going back to the OP's original story. If true, this is the strongest proof I have ever seen that sighted testing of audio is totally invalid.

Think about it.

The three groups of participants were given the same information, the tests are of different power cords, but it influenced them in different ways.

1. Objectivists - Since they do not believe power cords impact sound...this preconception would influence them to not hear differences, and in fact , in this case it apparently prevented them from noticing strong differences that exist.

2. Subjectivists - Since they expect to hear differences, they would be likely to hear differences, and would concentrate to hear them. Differences existed, and they heard them.

3. Neither: Since they theoretically have no opinion on the impact of power cords...they would also be open to hearing differences. When large differences...like out of phase speakers exist, they will hear them.

Out of phase speakers are incredibly simple for even the most untrained listeners to hear. We would expect the vast majority of groups 2 and 3 to notice them. If the Objectivists did not, ...it is because the information provided them before the test had a huge impact on their subjective interpretation of what they heard.

SO....if you are conducting critical listening tests to identify minute differences in the sound of speaker cables:

1. Do not invite Ethan Winer to be part of your group.... He may have great hearing but the topic would be prejudicial to him...and would impact his responses.

2. Absolutely make the tests blind. We now have proof that providing any information to listeners before a test concerning the technology that is being tested will destroy its validity. This is true even when the differences are as significant as a signal being out of phase.

 

[flipflop]

We now have proof that providing any information to listeners before a test concerning the technology that is being tested will destroy its validity.

No, we don't. For all we know, the group labelled 'objectivists' could in fact have been incredibly poor at discerning audible differences despite trying their hardest.

 

[tmtomh]

"I had a long conversation during the show with Thorsten Loesch of Abbington Musical Research and IFI. He told me a fascinating story about confirmation bias. That’s when you are so sure of something that even strong evidence to the contrary will not persuade you to change your mind.

Thorsten put together a blind ABX testing where he told participants it was a comparison of two power cables. But when he went behind the curtains, ostensibly to change the power cable, what he actually did was switch the speaker cables on one channel, so the system was playing out of phase. Thorsten had three different types of audiophiles take his test: subjectivists, objectivists, and those who were neither. The subjectivists and neutral listeners heard the effects of the system being thrown out of phase. The objectivists heard no differences. It was a robust test with clearly correlated results.

And how noticeable is having one speaker’s channel out of phase with the other? Ten years ago, at CES I entered a room with an “All Digital System” that had all the DACs and electronics in the loudspeakers. I listened for about ten seconds, then I turned to the gentleman who was giving the presentation and told him, “One of your channels is out of phase with the other.” He told me that was impossible since all the connections were hard-wired. I thanked him and left. A day later he caught me in the hall and explained that indeed one channel had been mis-wired out of phase.

The fact that the objectivists in Thorsten’s test were the ones who were so set in their opinions that it blinded them to the aural facts in front of their ears is a delicious irony. Why? Because those audiophiles who embrace ABX testing with the most fervor are those who believe most strongly in effects of expectation bias, which is why sighted testing is, in their eyes, flawed. Thorsten’s test indicates a strong tendency for objectivists to listen with closed ears whether the test is blind or sighted, which isn’t very objective, is it? "

www.theabsolutesound.com/articles/rmaf-2018-digital/

What do you think of this?

Here's what I think of it:

1. Whether the story is true or apocryphal, it illustrates an important point: Everyone is potentially subject to confirmation bias.
2. It is ironic that objectivists' opinions blinded them to the aural facts - but it is equally ironic that a subjectivist would use blind ABX testing to try to further their point of view - and it's quite telling that you seem unaware of that irony.
3. Objectivists, generally speaking, are the ones who believe in confirmation bias. So the story not only reinforces the value of ABX testing, but it shows - even more strongly - the central role of confirmation bias, which is something objectivists routinely emphasize and something subjectivists routinely dismiss, downplay, or ignore.

I should clarify that I'm not a big believer in simplistic objectivist vs subjectivist labels and am using them as a convenience here, in order to respond to your post.

 

[GGroch]

No, we don't. For all we know, the group labelled 'objectivists' could in fact have been incredibly poor at discerning audible differences despite trying their hardest.

Except that is not what the author of the Absolute Sound article said. "The fact that the objectivists in Thorsten’s test were the ones who were so set in their opinions that it blinded them to the aural facts in front of their ears is a delicious irony." So, blinded by their preconceptions...not their impacted ear wax.

J.J. Johnson tells a very similar story concerning electronics engineers (mostly, "we believe in measurement?") and members of an audiophile society (mostly, "we believe our ears?")......Edit. and I see this story was already posted on this thread HERE.

The point is we ALL hear largely what we want or expect to hear...what fits into our world view. That is because we are humans. It is not an insult.

Scientists are not exempt from that, they are humans too...which is one reason they insist on true blind tests.

 

[Old Listener]

Speaking of confirmation bias.

Subjectivists can dismiss dozens of well run double blind tests that don't support their pre-existing views. Tell them one story of a test result that agrees with their views and they can't wait to accept it as proof. Details of the test don't matter; agreement with their views is the measure of its validity.

 

[Sir Sanders Zingmore]

Speaking of confirmation bias.

Subjectivists can dismiss dozens of well run double blind tests that don't support their pre-existing views. Tell them one story of a test result that agrees with their views and they can't wait to accept it as proof. Details of the test don't matter; agreement with their views is the measure of its validity.

And the irony is that it doesn’t agree with their views

 

[Sal1950]

Subjectives, they love to tell their "stories"
Always told totally without any supporting evidence.
No names, places, or things.
Hey yo Mr Thorsten Loesch, I've got a story for you,

Twas the night before Christmas
And all thru the house
Not a creature was stirring
----- ------------- ------------ ----

 

[Sal1950]

At 3:20 to 4:39

Why would someone name their kid after a bagel? Poppy Crum.

I don't know but I always thought she looked pretty hot in that video.
Maybe just add a garter belt, some nylons, heels, ---, ---,
You guys know where I'm going here.

 

[jsrtheta]

Anecdote, so not evidence of anything: First "stereo" I ever owned was an ElectroVoice FM tuner/amplifier with "matching" speakers and a Garrard turntable. Got it home, wired it up, and immediately knew something was wrong. Sure enough, I'd wired it out-of-phase.

This ain't complicated.