Paper called "What Do Blind Evaluations Reveal?" about selection of submissions for journals etc.

[MaxwellsEq]

This is an interesting paper. Although actually off-topic for audio, it's on-topic for how our biases affect the decisions we make.

Haruka Uchida of the University of Chicago compared academic paper submissions, both blinded and non-blinded. Non-blinded there was a positive selection bias for submissions by more senior male contributors. Blinded, the scoring differences between senior and student contributions was less. However, interestingly, good submissions still came to the top, so blinding did NOT reduce the quality of the best papers which were chosen. This refutes the argument by those who consider that blinded testing hides higher quality audio components; allegedly bringing everything to a mediocre common quality.

https://haruka-uchida.github.io/websitefiles/uchida_haruka_blinding.pdf

 

[DVDdoug]

Well, you can't easily reject a paper from a known & respected professor or scientist.

This refutes the argument by those who consider that blinded testing hides higher quality audio components; allegedly bringing everything to a mediocre common quality.

"Audiophiles'" often make all kinds of excuses for "failing" blind listening tests.

Personally, I can't accept that making a test blind can ever make it less reliable. And we know that sighted listening tests are often unreliable. That's been demonstrated when an A-B switch doesn't do anything, or when "night and day" differences magically go-away when the test is done blind.

Not everything has to be done blind. Some things are truly obvious... If you are hearing a buzz from your speakers or getting no sound out of the left channel I'm not going to suggest a blind test. Amir does sighted testing as part of his routine reviews but he backs-up his impressions & perceptions with measurements.

 

[JanL]

This refutes the argument by those who consider that blinded testing hides higher quality audio components; allegedly bringing everything to a mediocre common quality.

https://haruka-uchida.github.io/websitefiles/uchida_haruka_blinding.pdf

It does no such thing. The test was for classifying papers, not testing the sense of hearing.
That being said, someone should devise a test to actually test the sensitivity to absolute audio quality sighted versus unsighted. If people track objectively superior quality better when sighted than unsighted then there would be some confirmation to the argument that unsighted guessing blocks something in the audio perception.

 

[MaxwellsEq]

It does no such thing. The test was for classifying papers, not testing the sense of hearing.
That being said, someone should devise a test to actually test the sensitivity to absolute audio quality sighted versus unsighted. If people track objectively superior quality better when sighted than unsighted then there would be some confirmation to the argument that unsighted guessing blocks something in the audio perception.

I recommend that you read Dr Floyd Toole's book. This is one of the things that his research demonstrated.

 

[JanL]

I recommend that you read Dr Floyd Toole's book. This is one of the things that his research demonstrated.

I am familiar with dr. Toole's findings. He showed that blind impressions are superior to impressions where the subjects know what product or brand they are listening to. My suggestion is a little different.
Take the sighted tests where the reproduction quality objectively tracks preferences and run them again unsighted. See if the preferences stay the same or become more random. That would be a way to verify the claims in question.

 

[MaxwellsEq]

I am familiar with dr. Toole's findings. He showed that blind impressions are superior to impressions where the subjects know what product or brand they are listening to. My suggestion is a little different.
Take the sighted tests where the reproduction quality objectively tracks preferences and run them again unsighted. See if the preferences stay the same or become more random. That would be a way to verify the claims in question.

Presumably you have not read the book.

 

[Head_Unit]

Presumably you have not read the book.

Book Shmook, I can see what sounds better! It's obvious!

Blind evaluations are time consuming and difficult, I guess that's why they seldom happen.

 

[sqrl]

It does no such thing. The test was for classifying papers, not testing the sense of hearing.
That being said, someone should devise a test to actually test the sensitivity to absolute audio quality sighted versus unsighted. If people track objectively superior quality better when sighted than unsighted then there would be some confirmation to the argument that unsighted guessing blocks something in the audio perception.

That would be audiovisual perception by definition.

 

[jsrtheta]

So many "Giant Killers", so little time.

 

[Timcognito]

As a engineer in medical devices who's products were subjected to double trials blind where the evaluators were not given any demographic data and could not be or have contact with the clinicians who placed or used device until the data was crunched by independent statisticians, one key element is and was patient selection. In most blind listening experiments bias come in in many ways not only looking at the speakers but not having rigorous leveling selection of the listening evaluators, all meeting certain hearing levels and listening training, also rigorous predefined unified qualities that they are rating with. Without such standards those listening tests become consumer preference surveys.

 

[DAP chap]

This is an interesting paper. Although actually off-topic for audio, it's on-topic for how our biases affect the decisions we make.

Haruka Uchida of the University of Chicago compared academic paper submissions, both blinded and non-blinded. Non-blinded there was a positive selection bias for submissions by more senior male contributors. Blinded, the scoring differences between senior and student contributions was less. However, interestingly, good submissions still came to the top, so blinding did NOT reduce the quality of the best papers which were chosen. This refutes the argument by those who consider that blinded testing hides higher quality audio components; allegedly bringing everything to a mediocre common quality.

https://haruka-uchida.github.io/websitefiles/uchida_haruka_blinding.pdf

Ooops... Is there a way to delete a like /thumbs-up?? I appreciated the link only.

 

[solderdude]

Removing as much 'bias' (knowledge) and applying statistical relevance is an important aspect for scientific proof.
In most cases this is very difficult and tiring (which is an issue) to do and thus should be done with rigor and using subjects that know what to look for (trained).

People evaluating differences at home are not doing it for, nor in, a scientifically 'sound' method.
They just want to know that what they bought/trial 'sounds better' to them and the 'result' they get is real for them (because they heard it).

 

[voodooless]

However, interestingly, good submissions still came to the top, so blinding did NOT reduce the quality of the best papers which were chosen. This refutes the argument by those who consider that blinded testing hides higher quality audio components; allegedly bringing everything to a mediocre common quality.

How does that follow? The domain it a totally different one. You can’t just assume that you can extrapolate this conclusion.

 

[TK_IIVI]

How does that follow? The domain it a totally different one. You can’t just assume that you can extrapolate this conclusion.

Does it really matter that the domain is different though? Wouldn't that (that it matters) imply that bias functions differently for people judging the quality of audio components than "papers"?

If bias functions differently, in what way and why?

(I'm not saying I agree with the premise what you're responding to, I'm just trying to understand the reason why you think it's wrong)

 

[voodooless]

If bias functions differently, in what way and why?

Bias functions similarly for sure. But it’s not the same biases.

Plus it’s demonstrably incorrect: plenty of super transparent devices cannot be distinguished from poorer performing devices in a blind test. So yes, blinding hides some of the engineering excellence, at least in electronics.

For speakers, this may be less so, and I think we have plenty of research proving this (Floyd Toole).

 

[JaMmy]

A statistician writes…
Bias is about measurement, regardless of whether that measurement is objective or subjective or what it is that is being measured. Blinding controls for one source of bias: knowledge of what a measurement would imply for the result of a trial. There is no difference in the logic of this process whether it is applied to academic papers, loudspeakers, DACs, medical interventions or for that matter prayer, UFOs or telepathy.