Limitations of blind testing procedures

[Sal1950]

Fremer has wonderful ears,

'

• 
  But this battleground I will step into: Swapping out power cords produced major differences in the sound. No wonder Brinkmann tuned his own power cord to supply with the Nyquist. Unfortunately, the second sample of the Nyquist didn’t include Brinkmann’s cord. Instead, I compared Dynamic Design’s Neutron 16 power cord, specifically designed for digital audio ($7500), with the digital version of Shunyata Research’s ZiTron Sigma ($2138). While the Shunyata’s slightly warm sound complements solidstate DACs like the Simaudio Moon Evolution 650D and 780D DACs, the Dynamic Design’s more open, crystalline sound proved the ideal match for the Nyquist. Is it worth spending $7500 on an active, shielded power cord for use with an $18,000 DAC—or any DAC, for that matter? That’s for you to answer.'

Heart warming isn't it ,I am sure there will more than a few saps buying one of those.
Keith

Look out Keith, Uncle Mikey will be threatening with suites by Entertainment Networks lawyers again.

 

[Reginald Addison]

I'll give ya a hint, If you post something, then quickly change your mind and go back and edit the post, it's too late. The forums software is very quick in sending out notifications to those subscribed to that thread and they read what you originally posted in their emails.
Think before you post.

I never completed the thought or hit post.
Thanks for the advice.

 

[Sal1950]

I never completed the thought or hit post.
Thanks for the advice.

Oh oh, must be that bit of spyware Trojan code I wrote for the site. Borrowed some magic dust from Synergistic Research the can even read your thoughts and report them to ASR.
Be careful what you think.

 

[John Max]

Nice info

 

[Don Hills]

Fremer has wonderful ears,

... Is it worth spending $7500 on an active, shielded power cord for use with an $18,000 DAC—or any DAC, for that matter? That’s for you to answer.

I see what he did there...

 

[Reginald Addison]

Oh oh, must be that bit of spyware Trojan code I wrote for the site. Borrowed some magic dust from Synergistic Research the can even read your thoughts and report them to ASR.
Be careful what you think.

 

[Reginald Addison]

Those who know me know I am not shy when it cones tI speaking ny mind. I stand by what I say. There is no need to retreat from such a harmless statement.
It is incomplete because I abandoned the thought and never intended to post it.
I am sure there is an explanation for how it got posted. I just have no interest in pursuing it.

 

[Thomas savage]

Those who know me know I am not shy when it cones tI speaking ny mind. I stand by what I say. There is no need to retreat from such a harmless statement.
It is incomplete because I abandoned the thought and never intended to post it.
I am sure there is an explanation for how it got posted. I just have no interest in pursuing it.

You must of pressed the 'post reply' button.. Iv often written stuff then thought better of it ( many times) even quitting out of the thread and coming back to see it still in the bottom box but NOT published.

 

[Reginald Addison]

You must of pressed the 'post reply' button.. Iv often written stuff then thought better of it ( many times) even quitting out of the thread and coming back to see it still in the bottom box but NOT published.

Hair trigger?

 

[Thomas savage]

Hair trigger?

Self sabotage my guess

 

[Purité Audio]

Premature ejaculation.
Keith

 

[Thomas savage]

Premature ejaculation.
Keith

Obviously Keith NEVER thinks better of it and just hits 'post reply'

Back to audio

 

[Jakob1863]

That's precisely my thoughts.

That´s a bit surprising....

But imo we´re discussing two (at least) very different topics, one would be which way audio development industrywise is driven and the other which way a listener could/should evaluate audio gear.

As said before if a consumer only decides by numbers chances are high that he/she sometimes buys some real devices that will not work sufficiently together; remember the japanese number race in the 1970s - 1980s? Really impressive amplifiers with remarkable low THD+N/IMD numbers, sufficient power output capabilities and low hum. But with some of these good sound quality wasn´t achievable. Of course i/we weren´t able to tests every possible combination of loudspeakers and amplifiers so that remark is restricted to some loudspeakers although those covered a fairly broad range as two and three ways were included and Quads (later other ESLs) as well.

 

[Thomas savage]

If one wants to guess at ones own personal preference that's ok, listen and have a daydream about what it all means and why it sounds like it does.. your doing no harm and attaching no universal truth to your musings so I'm ok with that and that played a large roll in my choices of what to buy. ( I did lead myself up the garden path though)

In the larger context , establishing critical criteria for the development of accurate audio reproduction technology I would leave anecdotal evidence no matter what controls have been applied well alone. We have better options imo.

once the play back system is as accurate as we can get it my suggestion is we sit down and listen to what the recording was made to sound like, after that if you want to deviate with phycoacoustic filtering , EQ etc that's up to you..

There is a thread somewhere I started to address matching pre amp to the amp and amp to the speakers etc .. certain members thought it was a non issue but there you go.

Best leave humans out at every possible opportunity

I'd add there so much disagreement in this thread within the 'pro' blind test guys it's served to reinforce my view. You are all arguing about how best to circumnavigate the faulty nature of the human to give valid results .. I just say let's just dump the idea .

 

[Cosmik]

...remember the japanese number race in the 1970s - 1980s? Really impressive amplifiers with remarkable low THD+N/IMD numbers, sufficient power output capabilities and low hum. But with some of these good sound quality wasn´t achievable.

Yes, I have heard people slagging off the Japanese amps from the 80s. I presume that something was found later that didn't showed up in the distortion measurements? Or is it a case of audiophiles just pronouncing them no good?

Two things are at play here:
1. People believing that an amplifier is something special, not just a standard electronic circuit.
2. People believing, therefore, that a cheap amplifier can't sound good - and this influencing what they hear.

The actual fact is that an amplifier is just an electronic circuit made from components that cost in the pennies range, and as long as it is well designed, a cheap one is perfectly adequate - and sounds good. If it's all on an integrated circuit it probably works even better.

 

[Jakob1863]

No I dont think thst is the case. Usually people with a technical background have a strong justification for doubting the claims of the audiophile. These doubts will be based on actual knowledge, education and experience of a relevant technical field. I have frequently seen, when an audiophile is presented with the technical and scientific explanation as to why something they beleive is not the case, a reaction of " well science doesnt know everything". Their fallable personal experience trumps all.

Sometimes you´re obviously right. I think we can agree that in every group of people there is proportion acting in a strange way.
But if we are talking about people working on a professional level in the audio field we have to notice that there exists a wide spread in personal beliefs about the audibility of "audiophile effects".

I haven often seen that people with a technical background, but without deeper knowledge wrt hearing sense, assert categorical statement that a listener couldn´t have heard something, because it would be impossible which is strange behavior itself. Questioning other peoples perception (due to doubts concerning may be not sufficiently controlled conditions) is one thing, but stating impossibility is another.
Usually discussions went wild afterwards.......

I think most of us, apart from the typical audiophile, can grasp the cognetive psycologlogy of sighted listening and the problems it presents. However I would love someone to explain the reasons why audiophiles seem to object to blind listening and why that would impair their ability to hear differences, which is often their position.

Imo they think (or have learned) that the "otherbelievers" are more interested in "proving audiphiles wrong" than in finding the truth about "strange effects". I have in german forums often seen, that audiophiles accepted that controlled listening is a good thing, tried it (sometimes with some advice from me), got positive results but had to notice that nobody was interested in those.

The " I need to be exposed to it for months to get a real sense ofvthe difference, I need to be in a Zen like state otherwise I am too stressed to hear the difference" really just points to one thing; if there really is a difference then it is utterly insignificant if it is that difficult to hear. You know, a sense of perspective.

Which was nearly exactly the argument from Dan Shanefield (afiar he called it "not useful") that i´ve mentioned a couple of posts before; it sounds reasonable but is it true?

I do think the distraction video is quite irrelevant to the point. Yes people can get distracted, but in a controlled test there would be a spread of people. Not everyone is going to mentally wander off missing the same aural characteristic.

We seem to agree that the difference isn´t really insignificant - although it´s debateable if it qualifies for the "night and day" label- , yet remained undetected by a lot of people, so provides imo contradictionary evidence to your (and Shanefields) argument.

Of course you´re right, not everyone gets distracted to the same degree and obviously a lot of people noticed the "gorilla", but it is a nice example for the importance of statistical power. The "gorilla" and the scoring task were presented in different conditions, the highest score for fail was in the case where people had to count the passes from the players in white dresses, it was ~49,x %.
So if you run an experiment with 100 participants, you have to conclude that the "gorilla" is indeed invisible, even in the case where the "failure rate" was low at ~30.x % you have to run an experiment with at least around 90 participants to reject the nullhypothesis.

So, when I have tested people its been informal and blind in famiiar settings with familar people and equipment.

Could you give some more details about the testing (details about blinding, controls, criteria for success and so on) and the conversation that led to the test?

Im not trying to make any scientifically scrutiniseable claim with this, its been for fun. Yet when they cant see what kit is being used they simply dont hear the differences they report when sighted. They arent stressed, they arent distracted, just listening.

I´ve experienced the same, as said before, even in our first test ever. Me and my colleague listening to capacitors with different dielectrica (and did describe the sonic differences in a qualitative way exactly the same) but he failed while i got a correct result. Quite suprising as i was sure that he could hear the difference (which he really did, as we later could confirm) but failed nevertheless.

Do you remember that i wrote about Dave Moultons description that in his blind tests people at first even had problems to detect a 6 dB difference in level?

This isnt rocket science and is well documented bias effect, yet you try and find an audiophile on an internet forum that is willing to take that bias out of the equation.

Yes, it isn´t "rocket science" but sensory testing is a quite complex field and we´ve named a plethora of bias effects and a lot of these are still at work in a controlled listening test (yes even including the blind conditions), that´s the reason why training under and accomodation to the specific task and test conditions is of such importance.

"I heard it therefore it is"

Might be wrong but might also be true.

For me the issue is the tendancy for the audiophile community to reject science that is concerning.

I agree.
But isn´t the "nonbeliever´s" unwillingness to accept (and to overcome) the difficulites/problems of (blind) sensory testing, which means to reject the "science of testing", concerning in the same way?

 

[Jakob1863]

Yes, I have heard people slagging off the Japanese amps from the 80s. I presume that something was found later that didn't showed up in the distortion measurements?

For the sake of formality, does it matter if it _later_ showed up?
I understand your argument in the way, that development would always hunt for the best possible (it must be measurementwise, because you argued that listening tests weren´t useful) and therefore the consumer would always get the best if buying the device offering the best specificitions; of course in short, but did i got it wrong?

<snip>
The actual fact is that an amplifier is just an electronic circuit made from components that cost in the pennies range, and as long as it is well designed, a cheap one is perfectly adequate - and sounds good. If it's all on an integrated circuit it probably works even better.

The "pennies range" isn´t exactly true, expecially if one aimes for the best possible.....

 

[Thomas savage]

Let's keep this to the topic of blind testing etc and not get drawn into arguments about the audible differences between capacitors and alike..

 

[tomelex]

Jakob,

Will you put into "bullet points" some of these limitations you believe, example

- people become stressed when tested blind but are not stressed when tested and can see the complete stereo system
- etc

 

[Jakob1863]

Let's keep this to the topic of blind testing etc and not get drawn into arguments about the audible differences between capacitors and alike..

"The capacitor case" was our first try in "blind tests"......