Limitations of blind testing procedures

[Jakob1863]

Jakob,

Will you put into "bullet points" some of these limitations you believe, example

- people become stressed when tested blind but are not stressed when tested and can see the complete stereo system
- etc

tomelex, i´m sorry but i don´t understand what you´re asking for.

 

[tomelex]

OK,

What are some of the limitations of blind testing? In your opinion right now.

They are:

- This type of test is too complicated for a hobby audiophile to conduct

- etc

 

[Thomas savage]

"The capacitor case" was our first try in "blind tests"......

Iv No problem with your 'case' being used as a example relevant to the to the topic of this thread but Iv been round long enough to know how that can snow ball and create a avalanche of pointless back and forth. Just want to head that off before it starts

As long as it pertinent to the topic of the thread your ok

( you have done nothing wrong, feel free to start a thread on your conclusions about audible differences between components with the same claimed value )

 

[RayDunzl]

Yes, I have heard people slagging off the Japanese amps from the 80s. I presume that something was found later that didn't showed up in the distortion measurements? Or is it a case of audiophiles just pronouncing them no good?

I have a Pioneer SX780. Does that count?

Maybe I should take a look at it.

 

[Cosmik]

I have a Pioneer SX780. Does that count?

Maybe I should take a look at it.

The service manual is online. They give THD and IMD specs at 23W and 1W output:

Presumably a resistive load. Does it fall apart into a real speaker?

 

[RayDunzl]

Presumably a resistive load. Does it fall apart into a real speaker?

It's paired with a set of Infinity P-363 right now, firing into the foot of Mother's (R.I.P.) bed.

I'll drag it all out here into the labmotory one day soonish and see what it launches into the room at reasonable levels.

 

[Jakob1863]

OK,

What are some of the limitations of blind testing? In your opinion right now.

They are:

- This type of test is too complicated for a hobby audiophile to conduct

- etc

Ah, thanks for the clarification.
First of all, as said before, the "blinding" is just one property among others, therefore the narrowing on this virtue of a controlled listening experiment/test does not help.

One of the main problems lies in the fact that you have to create an artifical setting that diverges from the "normal listening" situation, althoug we are actually interested in the listeners ability under those "normal listening" conditions. That´s the reason why we tried the "hidden test" approach with the preamplifiers.

Usually we are interested in tests with correct results. But quite often in these listening tests a situation exists where the experimenter uses a difference of unknown audibility together with a detector (i.e. listener) of unknown detection capability in an experimental setup of unkown impact on the detector.

But a test is a good one only if it is valid, objective and reliable. In the example above the validity is not ensured; if the experimenter does not care about it, then his objectivity is questionable and reliability is most likely not given either.

Some countermeasures do usually help:

-) cleary defined question that the test should help to examine
-) use of the best suitable test protocol
-) training of detectors (i.e. listeners) under the specific test conditions
-) use of positive controls (a positive control is a difference that must be detected under test conditions) and negative controls as well
-) doing a statistical power analysis to choose an appropiate sample size (means number of listeners and/or number of trials)

 

[March Audio]

Sometimes you´re obviously right. I think we can agree that in every group of people there is proportion acting in a strange way.
But if we are talking about people working on a professional level in the audio field we have to notice that there exists a wide spread in personal beliefs about the audibility of "audiophile effects".

I haven often seen that people with a technical background, but without deeper knowledge wrt hearing sense, assert categorical statement that a listener couldn´t have heard something, because it would be impossible which is strange behavior itself. Questioning other peoples perception (due to doubts concerning may be not sufficiently controlled conditions) is one thing, but stating impossibility is another.
Usually discussions went wild afterwards.......



Imo they think (or have learned) that the "otherbelievers" are more interested in "proving audiphiles wrong" than in finding the truth about "strange effects". I have in german forums often seen, that audiophiles accepted that controlled listening is a good thing, tried it (sometimes with some advice from me), got positive results but had to notice that nobody was interested in those.



Which was nearly exactly the argument from Dan Shanefield (afiar he called it "not useful") that i´ve mentioned a couple of posts before; it sounds reasonable but is it true?



We seem to agree that the difference isn´t really insignificant - although it´s debateable if it qualifies for the "night and day" label- , yet remained undetected by a lot of people, so provides imo contradictionary evidence to your (and Shanefields) argument.

Of course you´re right, not everyone gets distracted to the same degree and obviously a lot of people noticed the "gorilla", but it is a nice example for the importance of statistical power. The "gorilla" and the scoring task were presented in different conditions, the highest score for fail was in the case where people had to count the passes from the players in white dresses, it was ~49,x %.
So if you run an experiment with 100 participants, you have to conclude that the "gorilla" is indeed invisible, even in the case where the "failure rate" was low at ~30.x % you have to run an experiment with at least around 90 participants to reject the nullhypothesis.



Could you give some more details about the testing (details about blinding, controls, criteria for success and so on) and the conversation that led to the test?



I´ve experienced the same, as said before, even in our first test ever. Me and my colleague listening to capacitors with different dielectrica (and did describe the sonic differences in a qualitative way exactly the same) but he failed while i got a correct result. Quite suprising as i was sure that he could hear the difference (which he really did, as we later could confirm) but failed nevertheless.

Do you remember that i wrote about Dave Moultons description that in his blind tests people at first even had problems to detect a 6 dB difference in level?



Yes, it isn´t "rocket science" but sensory testing is a quite complex field and we´ve named a plethora of bias effects and a lot of these are still at work in a controlled listening test (yes even including the blind conditions), that´s the reason why training under and accomodation to the specific task and test conditions is of such importance.


Might be wrong but might also be true.



I agree.
But isn´t the "nonbeliever´s" unwillingness to accept (and to overcome) the difficulites/problems of (blind) sensory testing, which means to reject the "science of testing", concerning in the same way?

As I stated previously I think you will find most technical people view audiophile claims with caution rather than out right impossibility. However, I have seen some proposterous claims.

I think the problem I have with your post is it seems to be saying "anything is possible" without actually providing anything of substance to demonstrate that it actually is.

When individuals are claiming they hear things which are un measureable, then it is perfectly reasonable to doubt the claim. When people claim they hear things that are allegedly caused by mechanisms which are not technically feasible to those with knowledge of said technical discipline, then it is perfectly reasonable to doubt the claim.

If when tested under controlled conditions, the alleged difference disapears then its perfectly reasonable to doubt the claim. What I dont think is reasonable is to give much creedance to the idea that testing is always inneffective because its possible that some individuals are incapable of being tested without going into aural melt down,

I have been involved with a formal blind test at Tag Mclaren some years ago. It wasnt stressful in any way. I didnt feel my aural abilities were negatively affected in any way. I have also been involved with formal picture quality testing at the BBC. Needles to say I didnt go blind during that test either.

I think you are going to have to come up with a stronger argument than "Its possible".

Btw the gorrilla vid still isnt an appropriate comparison , well maybe if you flash pictures of Margot Robbie as a distraction during a test you might have a point

 

[March Audio]

Ah, thanks for the clarification.
First of all, as said before, the "blinding" is just one property among others, therefore the narrowing on this virtue of a controlled listening experiment/test does not help.

One of the main problems lies in the fact that you have to create an artifical setting that diverges from the "normal listening" situation, althoug we are actually interested in the listeners ability under those "normal listening" conditions. That´s the reason why we tried the "hidden test" approach with the preamplifiers.

)

Why is that automatically a problem? Your inference is that will automatically degrade someones ability to hear? Why is this?

So because someone cant see the equipment, this is abnormal and will definitely negatively affect them?

 

[March Audio]

Usually we are interested in tests with correct results. But quite often in these listening tests a situation exists where the experimenter uses a difference of unknown audibility together with a detector (i.e. listener) of unknown detection capability in an experimental setup of unkown impact on the detector.

But a test is a good one only if it is valid, objective and reliable. In the example above the validity is not ensured; if the experimenter does not care about it, then his objectivity is questionable and reliability is most likely not given either.

Some countermeasures do usually help:

-) cleary defined question that the test should help to examine
-) use of the best suitable test protocol
-) training of detectors (i.e. listeners) under the specific test conditions
-) use of positive controls (a positive control is a difference that must be detected under test conditions) and negative controls as well
-) doing a statistical power analysis to choose an appropiate sample size (means number of listeners and/or number of trials)

Dont have any problem with this, with those controls tbe alleged affect of the environment should be seen, ie if the individual has gone into aural melt down and cant hear the known audible differences

 

[Cosmik]

When individuals are claiming they hear things which are un measureable, then it is perfectly reasonable to doubt the claim.

Quite so. It raises a philosophical point:

For something to be un-measurable and yet audible means that it is beyond the understanding of audio designers and, in fact, science. As such, no hypothesis can be put forward to be tested.

The phenomenon cannot be summoned up or banished 'at will' because its cause is beyond understanding. So even if such a phenomenon existed, it would be to all intents and purposes a phantom. No one knows when it will appear or disappear. Temperature, humidity, the sea tides, audiophile gaseous emissions, passing vehicles, etc. could trigger it or drive it away. No one knows. There's no point testing for it, because you can't guarantee it will even stick around for the duration of your listening test. Even running the results through some of your magic statistical formulae can't change that fact.

What does this resemble? Why, it is no different from testing for paranormal activity. Get some people in a room and ask them 'scientifically' to indicate when they feel the room getting colder (while the thermometer stays the same); ask them to push a button if they feel 'a presence'. Write it all down very carefully and scientifically. Feed the results into some statistical formulae. 99% of the time, the tests show nothing. But on that hundredth test you may get lucky. "Statistical significance!". Publish the paper in the Journal of Supernatural Phenomena.

It is laughable and most certainly not science.

 

[Jakob1863]

Tales of mystery and imagination.....

In fact it´s mostly the other way round; there is nearly always a measurable difference, but some people claim (means put forth the hypothesis) that no difference can be perceived.

"For something to be un-measurable and yet audible means that it is beyond the understanding of audio designers and, in fact, science. As such, no hypothesis can be put forward to be tested."

Which is a remarkable assertion because :
H0: p = 0.5
H1: p > 0.5

is obviously a set of testable hypothesises.

Btw, beside the audio field, literature covering the history of scientific progress is usually telling that chance plays often an important role......

 

[Jakob1863]

As I stated previously I think you will find most technical people view audiophile claims with caution rather than out right impossibility. However, I have seen some proposterous claims.

Presumably we have to agree to disagree on that point.

I think the problem I have with your post is it seems to be saying "anything is possible" without actually providing anything of substance to demonstrate that it actually is.

Hm, that´s imo a contradictio in ratio; you demand a controlled (including blinding) listening test although you don´t know if the sighted listening tests had it wrong but just because you assume that it _might_ have been wrong, don´t you?

The same takes place in designing a useful test, you will not wait until something did happen you want to avoid that it happens. One of the golden rules of testing (surely i´ve cited that already) is:
"Block out what you can and randomize what you can´t block out"

That simply means that you try to block out any confounder, which is already known due to scientific efforts to be possibly harmful, and that you try to randomize any effect from confounders that you can´t block out.

When individuals are claiming they hear things which are un measureable, then it is perfectly reasonable to doubt the claim. When people claim they hear things that are allegedly caused by mechanisms which are not technically feasible to those with knowledge of said technical discipline, then it is perfectly reasonable to doubt the claim.

No problem with (reasonable) doubting.....

If when tested under controlled conditions, the alleged difference disapears then its perfectly reasonable to doubt the claim.

If the experimenter decided (often due to unknowing) to neglect what science already knows about sensory testing, things get a bit more complicated. Of course one still states that the nullhypothesis couldn´t be rejected, but he/she doesn´t know why.
To _believe_ isn´t sufficient otherwise there is no reason to do any test work at all.

What I dont think is reasonable is to give much creedance to the idea that testing is always inneffective because its possible that some individuals are incapable of being tested without going into aural melt down,

I have been involved with a formal blind test at Tag Mclaren some years ago. It wasnt stressful in any way. I didnt feel my aural abilities were negatively affected in any way. I have also been involved with formal picture quality testing at the BBC. Needles to say I didnt go blind during that test either.

Which is fine, but unfortunately doesn´t mean so much wrt other people (see the next paragraph). I am still interested in more details, as the mere tale of " involved in yyyy test" isn´t that descriptive.

I think you are going to have to come up with a stronger argument than "Its possible".

Hm, i provided two examples of well known bias effects (Rosenthal and Hawthorne) that might have an impact, provided an example from my own experience where my colleague demonstrated his problems with listening under test conditions and cited Dave Moulton with his observation that listeners (not experienced in "blind" tests) often had problems to detect even a 6dB level difference at the beginning.
If that doesn´t mean anything i can´t help. But, as said before, what you _believe_ isn´t that important, we are talking about putting beliefs to test - that´s what you basically demand from "audiophiles" - and you should simply do not less yourself.

Btw the gorrilla vid still isnt an appropriate comparison , well maybe if you flash pictures of Margot Robbie as a distraction during a test you might have a point

Well, your´s and Shanefield´s argument was that an undetected difference couldn´t be of relevance. The "gorilla experiment" and the experiments on inattentional deafness provide some counterevidence. In both experiments differences remained undetected but these differences were obviously not irrelevant.
But you seem to believe that a similar distraction can´t occur in "ordinary" "blind" tests but at that point we have been before.....

 

[March Audio]

Presumably we have to agree to disagree on that point.



Hm, that´s imo a contradictio in ratio; you demand a controlled (including blinding) listening test although you don´t know if the sighted listening tests had it wrong but just because you assume that it _might_ have been wrong, don´t you?

.

Disagree we might, but I have only seen outright denial of Audiophile claims when said claim is an outright contradiction of known science.

Its not a contradiction, sighted tests are flawed - period. This is proven science. The risks and effects of bias are well documented, proven and frankly quite obvious.

 

[March Audio]

[

Hm, i provided two examples of well known bias effects (Rosenthal and Hawthorne) that might have an impact, provided an example from my own experience where my colleague demonstrated his problems with listening under test conditions and cited Dave Moulton with his observation that listeners (not experienced in "blind" tests) often had problems to detect even a 6dB level difference at the beginning.
If that doesn´t mean anything i can´t help. But, as said before, what you _believe_ isn´t that important, we are talking about putting beliefs to test - that´s what you basically demand from "audiophiles" - and you should simply do not less yourself.

You still havent explained why, just because the subject cant see the equipment, they will have their aural capabilities negatively affected.

...at the beginning.... dont misunderstand me, Im not contradicting the idea that familiarisation and training may be required in a test to increase sensitivity, or that controls are required. However that still doesnt explain why audiophiles go deaf when in comfortable and familiar environments with familiar equipment and music when being tested.

Mine are less beliefs, and more real practical experience.

Also if these things are so difficult to hear, why are some (namely audiophiles) ascribing so much significance to them?

 

[March Audio]

P

Well, your´s and Shanefield´s argument was that an undetected difference couldn´t be of relevance. The "gorilla experiment" and the experiments on inattentional deafness provide some counterevidence. In both experiments differences remained undetected but these differences were obviously not irrelevant.
But you seem to believe that a similar distraction can´t occur in "ordinary" "blind" tests but at that point we have been before.....

Im afraid it doesnt. the Gorilla is an utterly contrived situation designed to prove a point, it couldnt be less similar to listening to some music excerpts. As I already mentioned any two individuals level of in-attention is not going to be the same, so the likelyhood of all subjects missing the same things is pretty small. Mind you, as I said, if you flashed pictures of Margot Robbie in front of me I could well become distracted.

 

[Blumlein 88]

Im afraid it doesnt. the Gorilla is an utterly contrived situation designed to prove a point, it couldnt be less similar to listening to some music excerpts. As I already mentioned any one individuals level of in-attention is not going to be the same, so the likely of of all subjects missing the same things is pretty unlikely. Mind you, as I said if you flashed pictures of Margot Robbie in front of me I could well become distracted.

Can we do the Margot Robbie blind test? I know you're thinking what good is Margot Robbie if you are blinded. Well bear with me.

I suggest we have Margot Robbie speak live or be a sound of her speaking thru a speaker. All behind a curtain so we can't see anything. See how well we do making choices if its live or memorex.

Next we drop the curtain so we can see Margot Robbie (I am sure she can lip synch as an actor). Have speak and either be her or the speaker. See if we do as well or better or worse than the blinded situation.

Then have cocktails and have Ms Robbie sit in and figure out under which conditions we have excelled. Does someone here have contacts with Ms Robbie to ask?

I mean why argue about such thought experiments. This is Audio Science we need to conduct the experiment for real. All about science you know.

 

[RayDunzl]

I have to admit that the first time I saw the gorilla I didn't see the gorilla.

---

http://www.aranmichaelmanagement.com/actor/margot-robbie/

Or, more to the point:

http://www.nopactalent.com/speaker/margot-robbie.php

.

 

[Cosmik]

You still havent explained why, just because the subject cant see the equipment, they will have their aural capabilities negatively affected.

...at the beginning.... dont misunderstand me, Im not contradicting the idea that familiarisation and training may be required in a test to increase sensitivity, or that controls are required. However that still doesnt explain why audiophiles go deaf when in comfortable and familiar environments with familiar equipment and music when being tested.

Mine are less beliefs, and more real practical experience.

Also if these things are so difficult to hear, why are some (namely audiophiles) ascribing so much significance to them?

The truth is that you will never know. Or maybe you can set up a Truman Show experiment where you observe an audiophile listening to music without his knowledge. Gradually you go from glorious high res down to the depravity of 16 bit CD and observe how he begins to become agitated, eventually cutting the session short in disgust. By keeping a record of these experiments you assemble enough data to feed through some statistical formulae. Yay! Statistical significance!

 

[March Audio]

Im not sure why you say that. I have yet to see any evidence that God exists and whole load of evidence that points to the fact that it doesnt. So, as far as science is concerned (and me) it doesnt exist. Millions of people believing in such things doesnt change this. Saying that it is still a possibility is of no value.

Audiophiles believing they can hear things, for which there is no evidence, is no different a situation.

Sure, lets keep investigating, thats the scientific thing to do, but lets not pander to the dogma. Lets not pander to the God in the gaps BS.

Mind you, I think your suggestion of a covert audiophile mental health study would be very interesting