Limitations of blind testing procedures

[tomelex]

I don´t get it why that is a limititation of the "DBT" . Do you mean it in a sense that a person has to realize first that he is prone to bias effects before even thinking about taking a controlled listening test?



I would often question the assertion that most of the people demanding "DBTs" are really objectivists and are therefore unwilling to realize the limitations of controlled experiments of the usual kind.
And that has a long tradition. As an example, Les Leventhal wrote an article /1/ that was published in the JAES in 1986 about the risk of committing beta-errors (which means to _not_ reject the null hypothesis although it is wrong) if running the usual 16 trial ABX. The response letter of Shanefield, Nousaine and Clarke /2/ didn´t express exactly appreciation to get the chance to improve the test schemes but instead reluctance. Dan Shanefield wrote imo the most professional answer part in asserting that, while Leventhal was correct, a difference wouldn´t be of importance if not detected under the test conditions. I was wondering at that time when reading the exchange about the slight hostility (only slight, because rules in a Journal like the AES prohibit any heated exchange) especially from Nousaine and Clark because it didn´t really fit the assumption that the goal was to find the truth.

The discussion went further in the letter pages of stereophile where the rules weren´t as strict. /3/
A couple of years later, knowing a bit more about the basics of experimental work, i realized that Leventhal wasn´t introducing a new revolutionary idea but did only report something known for ages (i.e. the concept of statistical power analysis) and therefore i was even more surprised by the reactions.

So it seems that it is quite often not about objectivism but about defense (or promotion) of just another belief systems.

/1/ Les Leventhal, Type 1 and Type 2 Errors in the Statistical Analysis of Listening Tests, JAES Volume 34 Issue 6 pp. 437-453; June 1986
/2/ Daniel Shanefield, David Clark, Thomas Nousaine, Les Leventhal; Comments on "Type 1 and Type 2 Errors in the Statistical Analysis of Listening Tests" and Author´s Reply, JAES Volume 35 Issue 7/8 pp. 567-572; July 1987
/3/ https://www.stereophile.com/content/highs-lows-double-blind-testing-page-2


Edit: added references

While I agree there a limitations to blind testing, they are far less than sighted testing, and its a fact that big talking audiophiles always have a reason they can not do a blind test. The reason is they know their hearing is variable, and they really deep down do not want to lose their status among their friends.

After all, you and I can sit down and listen to a song on your stereo, and I can say, do you hear " blah blah blah" and you have a choice right now, yes I can, or say no and admit your hearing is not as good as mine. I win if you say no, I am the audiophile golden ear, you are just the unwashed wannabe listeners who spend huge amounts of money on a system that you can not even hear what I can hear on it!

And that fear of losing status, also comes when audiophile pay huge money for gear, they can not listen to it blind (for a few seconds, or days or months or years...it does not matter) and find they can not tell the difference between the Hafler amplifier from 30 years ago and the newest Mark Levinson from today.

Bottom line, limitations or not, if they are not willing to submit to any type of non sighted test, they are just another person exercising their jaw muscles. Claiming there are limitations, whether mathematical or not, does not invalidate blind testing as a superior concept to sighted listening when others are wanting to see if they are really a qualified "golden ear" or just another wannabe.

Of course, the fallacy to all this is, even if you hear a difference, which rendition is accurate to the record or cd or digital file, which is the reference, and the only reference when talking about audio playback. So, you are stuck with which one you prefer, and these audiophiles don't want to find out they prefer the Hafler over the Mark Levinson. During the sighted test they would prefer the Mark Levinson. Someone off the street, who is not aquainted with audiophile brands would have atleast not have the name recognition bias ( one of main limitations of sighted testing). And, if they chose the thirty year old amp, they would just be like...yeah, it sounds better to me. No loss of eqo, prestige, golden ear status, etc. All of you reading this just know that the Levinson has to sound better than the Hafler..........don't you....?

 

[Thomas savage]

Because everyone must surely realise that there are some days when their hearing is 'on fire' and they seem to hear every detail, and other days when it all seems rather flat - it might as well be two different systems. It is perfectly reasonable to suggest that a listening test is likely to put one in the 'flat' frame of mind that we all experience. Science cannot begin to demonstrate that it doesn't, or whether that flat frame of mind inhibits hearing ability, therefore any claims based on listening tests are not scientific.

These listening tests are not objective measurements; they are basically asking people how they feel about something. Humans are not transducers; they are open to all kinds of influences that mean that their answers are not objective.

At the end of the day, actual audio systems owe nothing to these tests, nor the ravings of subjectivists - which is why they work so well. If we had followed the meanderings of middle aged audiophile listening test preferences, we would probably have audio systems that make all music fed into them more 'jazz-like' with "warm tone". Thank goodness this has got nothing at all to do with actual audio systems.

I think we all agree humans in this context are a 'faulty' measurement device, blind or sighted but it's just most here seem to think we have mitigated those 'faults' by the use of certain control criteria in the testing procedures.

That might be true, 'might'. Now if there were no other choice and we had to rely on listening tests I would accept that 'might' just as I accept the hundred other mechanisms we use to mitigate the human condition in areas like law , the financial sector, politics..., they all use a kind of filtering to protect the integrity of the system over our own inherent fallibilitys.

Now in every society I know all those areas above have not reached a 'ideal' a 'perfection' , they all operate as ' the best we can do' ' all we have, given there's no better alternative ' etc ..

In audio there is a human proof alternative, its math, physics, electrical engineering these are known systems and can provide measurements we know objectively are accurate.

So in my head this is why I don't see the need to 'risk it' there clearly is a alternative that cuts any chance of us messing with the results out..

 

[March Audio]

Because everyone must surely realise that there are some days when their hearing is 'on fire' and they seem to hear every detail, and other days when it all seems rather flat - it might as well be two different systems. It is perfectly reasonable to suggest that a listening test is likely to put one in the 'flat' frame of mind that we all experience. Science cannot begin to demonstrate that it doesn't, or whether that flat frame of mind inhibits hearing ability, therefore any claims based on listening tests are not scientific.

These listening tests are not objective measurements; they are basically asking people how they feel about something. Humans are not transducers; they are open to all kinds of influences that mean that their answers are not objective.

At the end of the day, actual audio systems owe nothing to these tests, nor the ravings of subjectivists - which is why they work so well. If we had followed the meanderings of middle aged audiophile listening test preferences, we would probably have audio systems that make all music fed into them more 'jazz-like' with "warm tone". Thank goodness this has got nothing at all to do with actual audio systems.

Yeah, but its funny that the audiophiles hearing always seems to fall flat the moment they are put under the most unobtrusive controls.......every time in my experience

Sorry, but a properly controlled listening test is objective. They are designed to remove, as far as reasonably practicable, how someone "feels"....you know... feel about things like the brand, price aesthetics etc.......

I'm not sure I understand your connection to the audio system

 

[Purité Audio]

I find it odd how subjectivists cling so tightly to dogma, there has been forty years of indoctrination that audio reproduction relies on constant upgrading that the ultimate replay is just one component change away.
A good pair of loudspeakers in a benign room that is it.
Keith

 

[Thomas savage]

I find it odd how subjectivists cling so tightly to dogma, there has been forty years of indoctrination that audio reproduction relies on constant upgrading that the ultimate replay is just one component change away.
A good pair of loudspeakers in a benign room that is it.
Keith

A set of beliefs are established, folk then defend those beliefs often at the expense of rational thinking. Nothing exclusive to subjectivist audiophiles in that, you will also do this in areas of your life on a daily basis.

 

[Purité Audio]

Ask of everything thing what is it unto itself!
Keith

 

[March Audio]

I find it odd how subjectivists cling so tightly to dogma, there has been forty years of indoctrination that audio reproduction relies on constant upgrading that the ultimate replay is just one component change away.
A good pair of loudspeakers in a benign room that is it.
Keith

The problem with this sort of reality is that the hobby is finished. For example I have spent very little on equipment in the past 15 years because I bought my Tag McLaren kit with room eq which educated me about room acoustics.

As you say decent speakers in a benign room with maybe a dash of room eq and problem really is solved.

However that is not necessarily what people are looking for.

 

[March Audio]

A set of beliefs are established, folk then defend those beliefs often at the expense of rational thinking. Nothing exclusive to subjectivist audiophiles in that, you will also do this in areas of your life on a daily basis.

This is of course true, that behaviour is not in any way exclusive to audiophiles.

 

[Frank Dernie]

An honest but naive opinion.

BE718 makes much more sense to me than most. He is a knowledgeable and experienced man not a gullible fellow easily conned. IMHO of course.

 

[Reginald Addison]

I find many devices wanting without the benefit of comparison.

 

[Purité Audio]

Fremer has wonderful ears,

'

• 
  But this battleground I will step into: Swapping out power cords produced major differences in the sound. No wonder Brinkmann tuned his own power cord to supply with the Nyquist. Unfortunately, the second sample of the Nyquist didn’t include Brinkmann’s cord. Instead, I compared Dynamic Design’s Neutron 16 power cord, specifically designed for digital audio ($7500), with the digital version of Shunyata Research’s ZiTron Sigma ($2138). While the Shunyata’s slightly warm sound complements solidstate DACs like the Simaudio Moon Evolution 650D and 780D DACs, the Dynamic Design’s more open, crystalline sound proved the ideal match for the Nyquist. Is it worth spending $7500 on an active, shielded power cord for use with an $18,000 DAC—or any DAC, for that matter? That’s for you to answer.'

Heart warming isn't it ,I am sure there will more than a few saps buying one of those.
Keith

• 
  

 

[Reginald Addison]

"Fremer has great ears..."

Not really
He is an old geezer.

 

[Fitzcaraldo215]

An honest but naive opinion.

So, that's it? Dismissive with an "I know better than you" air, yet absolutely no specifics about what it is you object to and why? Great. You do all subjectivists proud.

 

[Reginald Addison]

So, that's it? Dismissive with an "I know better than you" air, yet absolutely no specifics about what it is you object to and why? Great. You do all subjectivists proud.

No way for you to know. That was not supposed to be posted.
It does not appear on my cue. A couple of other posts appeared the same way.

 

[amirm]

"Fremer has great ears..."

Not really
He is an old geezer.

What are the rest of us?

 

[Reginald Addison]

What are the rest of us?

It's only OK to call someone a geezer if I'm one.
You guys can speak for yourself.

 

[RayDunzl]

http://www.dailymail.co.uk/news/article-1147686/Pictured-The-man-worlds-longest-ear-hair.html

 

[tomelex]

It's only OK to call someone a geezer if I'm one.
You guys can speak for yourself.

Fair enough. Many of us sure are not spring chickens around here!

 

[Kal Rubinson]

http://www.dailymail.co.uk/news/article-1147686/Pictured-The-man-worlds-longest-ear-hair.html

HRTF = Hair related transfer function

 

[Sal1950]

No way for you to know. That was not supposed to be posted.
It does not appear on my cue. A couple of other posts appeared the same way.

I'll give ya a hint, If you post something, then quickly change your mind and go back and edit the post, it's too late. The forums software is very quick in sending out notifications to those subscribed to that thread and they read what you originally posted in their emails.
Think before you post.