Limitations of blind testing procedures

[Cosmik]

All I know is that every single time I put ardent audiophiles under the most unobtrusive of controls, they totally lose their ability to discern differences they claim to find under sighted conditions.

Even if their claims of stress due to the listening conditions are true (which personally I dont buy into), all it shows is that the differences they suddenly cant hear are totally trivial. Do you really buy into the idea that they go significantly deaf because they are being tested?

And this is what it comes down to. Rigorous test conditions, statistical shenanigans that show the result is 99.99% significant blah blah. But at the end of the day, whether it is worth a damn comes down to "Do you really buy into the idea..?" because no one can prove whether people do lose their critical abilities as soon as they are put under pressure. This is limitation of blind testing procedures #1.

There are thousands more. Because we are dealing with complex biological and psychological systems, no one can really say whether 10 minutes of car noise ruins your critical faculties for 2 hours after that - because no one has tested for it. Or whether listening to music you like swamps your critical faculties, or if music you don't like prevents you from engaging sufficiently with it. Or if repeated listening to one piece of music kills all ability to 'hear' it. etc. etc.

Thank goodness that the people who design useful, reasonably-priced amplifiers and DACs and so on simply design them to spec as straightforward electronic circuits rather than in response to the listening tests that people bizarrely think have contributed anything to the world of audio! Amplifiers and DACs are still flat to 20 kHz and aim to generate as close to zero noise and distortion as possible. Listening tests have made no difference to that, nor how it is achieved!

 

[Thomas savage]

And this is what it comes down to. Rigorous test conditions, statistical shenanigans that show the result is 99.99% significant blah blah. But at the end of the day, whether it is worth a damn comes down to "Do you really buy into the idea..?" because no one can prove whether people do lose their critical abilities as soon as they are put under pressure. This is limitation of blind testing procedures #1.

There are thousands more. Because we are dealing with complex biological and psychological systems, no one can really say whether 10 minutes of car noise ruins your critical faculties for 2 hours after that - because no one has tested for it. Or whether listening to music you like swamps your critical faculties, or if music you don't like prevents you from engaging sufficiently with it. Or if repeated listening to one piece of music kills all ability to 'hear' it. etc. etc.

Thank goodness that the people who design useful, reasonably-priced amplifiers and DACs and so on simply design them to spec as straightforward electronic circuits rather than in response to the listening tests that people bizarrely think have contributed anything to the world of audio! Amplifiers and DACs are still flat to 20 kHz and aim to generate as close to zero noise and distortion as possible. Listening tests have made no difference to that, nor how it is achieved!

That's precisely my thoughts.

 

[Fitzcaraldo215]

And this is what it comes down to. Rigorous test conditions, statistical shenanigans that show the result is 99.99% significant blah blah. But at the end of the day, whether it is worth a damn comes down to "Do you really buy into the idea..?" because no one can prove whether people do lose their critical abilities as soon as they are put under pressure. This is limitation of blind testing procedures #1.

There are thousands more. Because we are dealing with complex biological and psychological systems, no one can really say whether 10 minutes of car noise ruins your critical faculties for 2 hours after that - because no one has tested for it. Or whether listening to music you like swamps your critical faculties, or if music you don't like prevents you from engaging sufficiently with it. Or if repeated listening to one piece of music kills all ability to 'hear' it. etc. etc.

Thank goodness that the people who design useful, reasonably-priced amplifiers and DACs and so on simply design them to spec as straightforward electronic circuits rather than in response to the listening tests that people bizarrely think have contributed anything to the world of audio! Amplifiers and DACs are still flat to 20 kHz and aim to generate as close to zero noise and distortion as possible. Listening tests have made no difference to that, nor how it is achieved!

But, was it not via controlled listening tests that the 20k limit you are assuming was, in fact established, as opposed to say 5 kHz or 50 kHz? Was 20k just pulled out of thin air?

 

[Thomas savage]

But, was it not via controlled listening tests that the 20k limit you are assuming was, in fact established, as opposed to say 5 kHz or 50 kHz? Was 20k just pulled out of thin air?

No, just bog standard hearing exams.. as far as I'm aware anyhow.

 

[Fitzcaraldo215]

No, just bog standard hearing exams.. as far as I'm aware anyhow.

So, yes, it was by scientific testing of some sort on a lot of human subjects, though perhaps not DBT's.

 

[Jakob1863]

So, yes, it was by scientific testing of some sort on a lot of human subjects, though perhaps not DBT's.

I think i´ve linked already in an earlier post; it was Harry F.Olson "who saved the high fidelity" as he did a quite large scale controlled listening experiment where people were listening to original music (purely acoustical instruments) in a full bandwidth conditon and to a acoustically filtered (down to roughly 5kHz cut off) version. Around 75% of the listeners preferred the full bandwidth condition.

/1/Harry F. Olson, Frequency Range Preference for Speech and Music, The Journal of the Acoustical Society of America 19, 549 (1947)

I think we can find in the last couple of posts some examples that illustrate why i called it just another sort of belief system.

-) "Nonbelievers" can do sighted listening tests all day long, in a miracle way they are never influenced by bias
-) A difference not perceived in a "dbt" can´t be of relevance
-) listening tests (and statistical shenanignas) are evil; a music lover only needs to buy (?every year?) the newest and best equipment to get the best sould quality

Edit: Sorry, "sighted listening" instead of "unsighted"
Edit2: Sorry, corrected the Olson reference, it is "Frequency Range Preference" instead of "Frequency Preference"
Edit3: Sorry, it were only 69% in Olson´s study who preferred the full bandwidth sound

Should i worry if the edit list gets longer than the actual post........

 

[Thomas savage]

I think i´ve linked already in an earlier post; it was Harry F.Olson "who saved the high fidelity" as he did a quite large scale controlled listening experiment where people were listening to original music (purely acoustical instruments) in a full bandwidth conditon and to a acoustically filtered (down to roughly 5kHz cut off) version. Around 75% of the listeners preferred the full bandwidth condition.

/1/Harry F. Olson, Frequency Preference for Speech and Music, The Journal of the Acoustical Society of America 19, 549 (1947)

I think we can find in the last couple of posts some examples that illustrate why i called it just another sort of belief system.

-) "Nonbelievers" can do unsighted listening tests all day long, in a miracle way they are never influenced by bias
-) A difference not perceived in a "dbt" can´t be of relevance
-) listening tests (and statistical shenanignas) are evil; a music lover only needs to buy (?every year?) the newest and best equipment to get the best sould quality

You did amirs head in so bad he's had to take a road trip ( I know he's really checked into argumentative awareness rehab)

 

[Thomas savage]

So, yes, it was by scientific testing of some sort on a lot of human subjects, though perhaps not DBT's.

Of course, but nothing to do with hifi or aimed at audio reproduction . It would be humans sitting down with headphones on while test tones are played and the subject raises a hand or presses a button when they detect sound, a bog standard hearing test these days.

how that information then applied to audio reproduction, what's worth capturing and reproducing etc is another matter entirely.

Nothing in cosmiks post claimed the discovery of the frequency range of our hearing was unnecessary or bogus just beyond a simple test tone frequency test it's a crapshoot . Too many un-understood variables that could impact the results.. I agree with him, just make the gear objectively as 'perfect' as technology will allow.

Anyway as I don't believe in listening tests for design of audio reproduction equipment, blind or other wise I will check out of this thread ..

 

[Fitzcaraldo215]

I find it interesting that among many audio objectivists, who are otherwise comfortable with science and measurements, etc., that empirical measurements of auditory response by human subjects are somehow invalid, taboo, meaningless, fake science, bullshit, or what have you. I guess Floyd Toole was wasting his time, as were a lot of other guys.

Personally, I think that improvements in audio via purely "hard science" are all good and worthwhile, but that paradigm has been moving at a snail's pace in making sound better. I think the place to look for improvements to audio that can have much more impact is psychoacoustics and a better understanding of what is or is not important in how we humans perceive sound and audio. But, doing so requires a shift of gears from insisting solely on traditional hard science into accepting the perceptual and behavioral sciences and much more of statistical proofs based on experiments with human subjects.

 

[Cosmik]

I find it interesting that among many audio objectivists, who are otherwise comfortable with science and measurements, etc., that empirical measurements of auditory response by human subjects are somehow invalid, taboo, meaningless, fake science, bullshit, or what have you.

I would remove the word "otherwise".

I guess Floyd Toole was wasting his time, as were a lot of other guys.

The impression I get is that his listening tests were aimed at confirming that dispersion and the off-axis sound matters - rather than making a brand new discovery. The fact that existing speakers were badly designed doesn't mean that listening tests were needed to prove it.

A few threads back I linked to a 1970s paper from KEF that decribed the thinking behind the model 105; it was designed in full knowledge of why the off-axis sound should be considered as well as the on-axis.

Personally, I think that improvements in audio via purely "hard science" are all good and worthwhile, but that paradigm has been moving at a snail's pace in making sound better.

If stereo is the 'format', then the hard science people have simply been aiming for perfection, originally with LPs, then CD, now active speakers, DSP correction and active dispersion control. These are all developments aimed towards an ideal system 'on paper' i.e. measured objectively.

I think the place to look for improvements to audio that can have much more impact is psychoacoustics and a better understanding of what is or is not important in how we humans perceive sound and audio. But, doing so requires a shift of gears from insisting solely on traditional hard science into accepting the perceptual and behavioral sciences and much more of statistical proofs based on experiments with human subjects.

For better or worse, I think it will take quite a shift to get away from people listening in stereo only, and I don't think there is much scope for improving a system that is already close to perfection. If there are psychoacoustic things to be done, these will be provided by the people on the recording side - as they already are.

People did used to have a psychoacoustic enhancer button labelled 'Loudness', but no audiophile worth their salt would use it even though it might be kind of useful sometimes. The real audiophile would always listen at the 'correct' volume anyway . But I think this is a useful example of why the playback system cannot really be improved: given just the mixed-down recording, what can the playback system do with it? Even a 'scientific' 'enhancement' such as attempting to fiddle with the Fletcher Munsons can only be imperfect guesswork.

Listening tests might show that a particular listener likes the sound of flutes and the key of C-Major. Should the personalised audio system, therefore attempt to make everything sound a bit more flute-like? (not a problem if that's what we really think). Should it attempt to re-tune the recording to the listener's preferred style?

Statistics may (purport to) show that American female listeners prefer more bass than, say, a UK male listener. If so, should the system have a menu to set the country of origin and gender of the listener? I don't think so.

Ultimately, anything but a 'straight' playback system can be seen as absurd.

 

[Blumlein 88]

So it seems that it is quite often not about objectivism but about defense (or promotion) of just another belief systems.

I think i´ve linked already in an earlier post; it was Harry F.Olson "who saved the high fidelity" as he did a quite large scale controlled listening experiment where people were listening to original music (purely acoustical instruments) in a full bandwidth conditon and to a acoustically filtered (down to roughly 5kHz cut off) version. Around 75% of the listeners preferred the full bandwidth condition.

/1/Harry F. Olson, Frequency Preference for Speech and Music, The Journal of the Acoustical Society of America 19, 549 (1947)

Some interesting details in that Olson test of bandwidth. Some testing indicated people preferred restricted bandwidth. In some portion of Olson's testing people aged 14-20 years still preferred restricted bandwidth. This was thought to be because of their familiarity listening to restricted range phonographs and radio of the time. They also restricted the low end at 100 hz. They found sound was more balanced if chopping the high end was paired with chopping the low end. Getting the upper and lower limits to multiply to 500K or so was the target. A practice I had heard of before without knowing where the idea came from. Yet when all types of people across ages were used yes preference for 30hz-17khz bandwidth. They had musicians playing real time practiced to keep levels in a comfortable range. During the test, the filter would be put in and then taken out every 15 seconds.

 

[March Audio]

And this is what it comes down to. Rigorous test conditions, statistical shenanigans that show the result is 99.99% significant blah blah. But at the end of the day, whether it is worth a damn comes down to "Do you really buy into the idea..?" because no one can prove whether people do lose their critical abilities as soon as they are put under pressure. This is limitation of blind testing procedures #1.

There are thousands more. Because we are dealing with complex biological and psychological systems, no one can really say whether 10 minutes of car noise ruins your critical faculties for 2 hours after that - because no one has tested for it. Or whether listening to music you like swamps your critical faculties, or if music you don't like prevents you from engaging sufficiently with it. Or if repeated listening to one piece of music kills all ability to 'hear' it. etc. etc.

Thank goodness that the people who design useful, reasonably-priced amplifiers and DACs and so on simply design them to spec as straightforward electronic circuits rather than in response to the listening tests that people bizarrely think have contributed anything to the world of audio! Amplifiers and DACs are still flat to 20 kHz and aim to generate as close to zero noise and distortion as possible. Listening tests have made no difference to that, nor how it is achieved!

I dont think its that simple. Whilst its possible there may be some individuals that go into aural meltdown because they are being tested, equally there will be many that dont. So with a statistical test this should become apparent. Its not a case of "not buying into" and thats that, there are reasons behind it which are testable. If they poor audiophile luvvies are that stressed then they are going to miss subtleties that ordinary unstressed subjects will hear. Why is it that only audiophiles suffer this affliction where they are "super hearers" one minute and not the next? I think it far more likely that they just were "ordinary hearers" all along.

Actually the effects of loud noise on hearing have been explored. Temporary hearing shift is well documented. You should get told not to be around anything noisy 12 hours prior to a hearing test for this reason. Also anyone who has been to a gig has experienced the more extreme end of it.

You refer to boredom/dislike of music affecting results. Well again, whilst some individuals will react that way, equally many wont, so statistically that should be visible.

However I agree with your last sentence, I have not heard a piece of kit that measures well, sound crap.

 

[Jakob1863]

I think someone that accepts the reality of sighted listening, ie massive potential for bias, has less invested in the outcome of an un-sighted test. They understand their aural and psychological limitations in a way the typical audiophile doesn't. They dont make up excuses, such as "im too stressed to tell the difference cos I'm being tested" or "its an unfamiliar environment and I lose the ability to hear". In simple terms, I dont find that the typical blind advocate has anything to prove, they are just realists - unlike many audiophiles Im acquainted with.

I´m not sure about that. Quite often people having a technical background seem to have a strong belief (i.e. usually "can´t be" ) about a claim wrt a sonic difference first and trying to find a some arguments to back up their position. "DBTs" are often used just as a knock out argument but they forget that they have entered a field (cognitive psychology/psychophysics) in which they have little to no experience. And most are not that happy to learn what they have missed, see for example the discussion in the stereophile letters section, mentioned above. A reaction i´ve experienced myself in a lot of forum and real life discussions and that i´ve seen that hitting others as well.
As said in my post above, if the goal is just to find the truth, this sort of reaction isn´t justifiable......

Does that opinion of mine prove blind advocates are unbiased, no not really. All I know is that every single time I put ardent audiophiles under the most unobtrusive of controls, they totally lose their ability to discern differences they claim to find under sighted conditions.

Even if their claims of stress due to the listening conditions are true (which personally I dont buy into).......

I wouldn´t buy it either always (means once and forever), but science already knows that peoples abilities might be different if tested and if knowing about, even if people know that the experimenter think they will fail, it might have an impact. (Hawthorne effect and Rosenthal effect)

....., all it shows is that the differences they suddenly cant hear are totally trivial. Do you really buy into the idea that they go significantly deaf because they are being tested?

No, not going significantly deaf, but perhaps being influenced by artifical/unsual conditions. I´ve seen those effects when conducting controlled tests with listeners and most people i know, who did/do the same, noticed similar problems. Dave Moulton for example wrote that people even had difficulties to notice a 6dB level difference at first in listening tests.

Distraction might lead to quite severe perception alterations, therefore i pointed to the inattentional blindness experiments from Simons (for example):

Depending on the specific conditions, 30% - 50% failed in this experiments.

People then worked out quite fanciful arguments why that doesn´t mean anything and even were questioning that similar problems could happen in auditory tasks.

I later pointed to the inattentional deafness experiments where for example in a piece from "Also sprach Zarathustra" a electro guitar was introduced.

So i can´t totally reject the "stress argument" but would argue that for good reasons in the ITU-R BS.1116 recommendation a sustained training and accomodation time is included and the use of positive/negative controls.
Usage of postive/negative controls also strongly recommended by old retired jj "Do you have to use controls? Only in case you want to know if your test is good"

 

[Purité Audio]

I don't think that was a real gorilla.
Keith

 

[Jakob1863]

Some interesting details in that Olson test of bandwidth. Some testing indicated people preferred restricted bandwidth. In some portion of Olson's testing people aged 14-20 years still preferred restricted bandwidth. This was thought to be because of their familiarity listening to restricted range phonographs and radio of the time. They also restricted the low end at 100 hz. They found sound was more balanced if chopping the high end was paired with chopping the low end. Getting the upper and lower limits to multiply to 500K or so was the target. A practice I had heard of before without knowing where the idea came from. Yet when all types of people across ages were used yes preference for 30hz-17khz bandwidth. They had musicians playing real time practiced to keep levels in a comfortable range. During the test, the filter would be put in and then taken out every 15 seconds.

I might be mistaken but i think Olson wrote that they (in some runs) restricted the bass to ~100 in the filtered version but did not find a difference to the filtered version without bass restriction.
I was surprised too to learn that the "5ook thing" was introduced such a long time ago .....

 

[Jakob1863]

I don't think that was a real gorilla.
Keith

Pah, that´s just because your monitor isn´t revealing.....

 

[March Audio]

I´m not sure about that. Quite often people having a technical background seem to have a strong belief (i.e. usually "can´t be" ) about a claim wrt a sonic difference first and trying to find a some arguments to back up their position. "DBTs" are often used just as a knock out argument but they forget that they have entered a field (cognitive psychology/psychophysics) in which they have little to no experience. And most are not that happy to learn what they have missed, see for example the discussion in the stereophile letters section, mentioned above. A reaction i´ve experienced myself in a lot of forum and real life discussions and that i´ve seen that hitting others as well.
As said in my post above, if the goal is just to find the truth, this sort of reaction isn´t justifiable......



I wouldn´t buy it either always (means once and forever), but science already knows that peoples abilities might be different if tested and if knowing about, even if people know that the experimenter think they will fail, it might have an impact. (Hawthorne effect and Rosenthal effect)



No, not going significantly deaf, but perhaps being influenced by artifical/unsual conditions. I´ve seen those effects when conducting controlled tests with listeners and most people i know, who did/do the same, noticed similar problems. Dave Moulton for example wrote that people even had difficulties to notice a 6dB level difference at first in listening tests.

Distraction might lead to quite severe perception alterations, therefore i pointed to the inattentional blindness experiments from Simons (for example):

Depending on the specific conditions, 30% - 50% failed in this experiments.

People then worked out quite fanciful arguments why that doesn´t mean anything and even were questioning that similar problems could happen in auditory tasks.

I later pointed to the inattentional deafness experiments where for example in a piece from "Also sprach Zarathustra" a electro guitar was introduced.

So i can´t totally reject the "stress argument" but would argue that for good reasons in the ITU-R BS.1116 recommendation a sustained training and accomodation time is included and the use of positive/negative controls.
Usage of postive/negative controls also strongly recommended by old retired jj "Do you have to use controls? Only in case you want to know if your test is good"

No I dont think thst is the case. Usually people with a technical background have a strong justification for doubting the claims of the audiophile. These doubts will be based on actual knowledge, education and experience of a relevant technical field. I have frequently seen, when an audiophile is presented with the technical and scientific explanation as to why something they beleive is not the case, a reaction of " well science doesnt know everything". Their fallable personal experience trumps all.

I think most of us, apart from the typical audiophile, can grasp the cognetive psycologlogy of sighted listening and the problems it presents. However I would love someone to explain the reasons why audiophiles seem to object to blind listening and why that would impair their ability to hear differences, which is often their position. The " I need to be exposed to it for months to get a real sense ofvthe difference, I need to be in a Zen like state otherwise I am too stressed to hear the difference" really just points to one thing; if there really is a difference then it is utterly insignificant if it is that difficult to hear. You know, a sense of perspective.

I do think the distraction video is quite irrelevant to the point. Yes people can get distracted, but in a controlled test there would be a spread of people. Not everyone is going to mentally wander off missing the same aural characteristic.

So, when I have tested people its been informal and blind in famiiar settings with familar people and equipment. Im not trying to make any scientifically scrutiniseable claim with this, its been for fun. Yet when they cant see what kit is being used they simply dont hear the differences they report when sighted. They arent stressed, they arent distracted, just listening.

This isnt rocket science and is well documented bias effect, yet you try and find an audiophile on an internet forum that is willing to take that bias out of the equation.

"I heard it therefore it is"

For me the issue is the tendancy for the audiophile community to reject science that is concerning. The tendancy to accept any old marketing pseudo science bollocks..........and then ascribe the positive improvement in audio quality they perceive to the new psuedo science item they have introduced.

 

[Reginald Addison]

I think someone that accepts the reality of sighted listening, ie massive potential for bias, has less invested in the outcome of an un-sighted test. They understand their aural and psychological limitations in a way the typical audiophile doesn't. They dont make up excuses, such as "im too stressed to tell the difference cos I'm being tested" or "its an unfamiliar environment and I lose the ability to hear". In simple terms, I dont find that the typical blind advocate has anything to prove, they are just realists - unlike many audiophiles Im acquainted with.

Does that opinion of mine prove blind advocates are unbiased, no not really. All I know is that every single time I put ardent audiophiles under the most unobtrusive of controls, they totally lose their ability to discern differences they claim to find under sighted conditions.

Even if their claims of stress due to the listening conditions are true (which personally I dont buy into), all it shows is that the differences they suddenly cant hear are totally trivial. Do you really buy into the idea that they go significantly deaf because they are being tested?

An honest but naive opinion.

No I dont think thst is the case. Usually people with a technical background have a strong justification for doubting the claims of the audiophile. These doubts will be based on actual knowledge, education and experience of a relevant technical field. I have frequently seen, when an audiophile is presented with the technical and scientific explanation as to why something they beleive is not the case, a reaction of " well science doesnt know everything". Their fallable personal experience trumps all.

I think most of us, apart from the typical audiophile, can grasp the cognetive psycologlogy of sighted listening and the problems it presents. However I would love someone to explain the reasons why audiophiles seem to object to blind listening and why that would impair their ability to hear differences, which is often their position. The " I need to be exposed to it for months to get a real sense ofvthe difference, I need to be in a Zen like state otherwise I am too stressed to hear the difference" really just points to one thing; if there really is a difference then it is utterly insignificant if it is that difficult to hear. You know, a sense of perspective.

I do think the distraction video is quite irrelevant to the point. Yes people can get distracted, but in a controlled test there would be a spread of people. Not everyone is going to mentally wander off missing the same aural characteristic.

So, when I have tested people its been informal and blind in famiiar settings with familar people and equipment. Im not trying to make any scientifically scrutiniseable claim with this, its been for fun. Yet when they cant see what kit is being used they simply dont hear the differences they report when sighted. They arent stressed, they arent distracted, just listening.

This isnt rocket science and is well documented bias effect, yet you try and find an audiophile on an internet forum that is willing to take that bias out of the equation.

"I heard it therefore it is"

For me the issue is the tendancy for the audiophile community to reject science that is concerning. The tendancy to accept any old marketing pseudo science bollocks..........and then ascribe the pisitive improvement in audio quality they perceive to the new psudo science item they have introduced.

 

[March Audio]

You are going to need to explain that statement

 

[Cosmik]

I would love someone to explain the reasons why audiophiles seem to object to blind listening and why that would impair their ability to hear differences, which is often their position.

Because everyone must surely realise that there are some days when their hearing is 'on fire' and they seem to hear every detail, and other days when it all seems rather flat - it might as well be two different systems. It is perfectly reasonable to suggest that a listening test is likely to put one in the 'flat' frame of mind that we all experience. Science cannot begin to demonstrate that it doesn't, or whether that flat frame of mind inhibits hearing ability, therefore any claims based on listening tests are not scientific.

These listening tests are not objective measurements; they are basically asking people how they feel about something. Humans are not transducers; they are open to all kinds of influences that mean that their answers are not objective.

At the end of the day, actual audio systems owe nothing to these tests, nor the ravings of subjectivists - which is why they work so well. If we had followed the meanderings of middle aged audiophile listening test preferences, we would probably have audio systems that make all music fed into them more 'jazz-like' with "warm tone". Thank goodness this has got nothing at all to do with actual audio systems.