• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

The frailty of Sighted Listening Tests

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
Since it created controversies in a couple of speakers review I would like to know what's your thought about blind speaker testing and about the reviewer's experience.

Here's what Sean Olive wrote a couple of years ago :

https://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html?m=1

About biaises @pkane did a great job here :

https://audiophilestyle.com/forums/topic/59351-does-bias-affect-audio-test-results/

Double blind tests are hard and expensive to put in place, but I think there's couple of simple things that can help to acheived a more neutral/scientific result. Science as whole is not doing so great in the public opinion, the Covid-19 is another exemple of what happened when what science is, and how it(should) work, is not understood. (I'm not saying that I understand/know everything too). Hobbies and passions are great ways to get introduced To what science is. Basics principles can then he used for other subjects.

I already have commented too much, and I hope ou won't be seen as something hostile.

P.S if you're a "grammar nazi" let me know in private message, I'll correct my post.
 

aarons915

Addicted to Fun and Learning
Forum Donor
Joined
Oct 20, 2019
Messages
685
Likes
1,140
Location
Chicago, IL
Importantly from Olive's article:

"Finally, the tests found that experienced and inexperienced listeners (both male and female) tended to prefer the same loudspeakers, which has been confirmed in a more recent, larger study. The experienced listeners were simply more consistent in their responses. As it turned out, the experienced listeners were no more or no less immune to the effects of visual biases than inexperienced listeners."

So a "trained" listener simply comes to their conclusions faster and are more consistent but doesn't appear to affect their biases. Perhaps there is different training bias that this article doesn't discuss, if anyone has a link or information on that it would be helpful.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
Importantly from Olive's article:

"Finally, the tests found that experienced and inexperienced listeners (both male and female) tended to prefer the same loudspeakers, which has been confirmed in a more recent, larger study. The experienced listeners were simply more consistent in their responses. As it turned out, the experienced listeners were no more or no less immune to the effects of visual biases than inexperienced listeners."

So a "trained" listener simply comes to their conclusions faster and are more consistent but doesn't appear to affect their biases. Perhaps there is different training bias that this article doesn't discuss, if anyone has a link or information on that it would be helpful.

Yes it also means that untrained listeners have the same "authority" about preferences.

Let's not forget that those preference things are generalisation and are very usefull for companies that want to make profit. I guess it should not prevent people to try different ways of enjoying the audio experience.
 

SoundAndMotion

Active Member
Joined
Mar 23, 2016
Messages
144
Likes
111
Location
Germany
It is a valuable and important article, based mostly on one of his AES articles (also worth reading).

For those inclined to read the whole thing, it is time well spent. For those who will only read the title and quickly skim the text, it is useful to read what he later wrote about the title:
In retrospect, my choice of the word "Dishonest" in the title was perhaps too strong and sensational. Sighted evaluations have their use(I use them sometimes myself), and doing one doesn't necessarily imply willful disregard for the truth. However, people who do sighted listening tests should be aware of their limitations and potential biases, as demonstrated in this article. Unfortunately, many people in our industry routinely report results from sighted listening evaluations without regard to or acknowledgment of these biases or limitations. Some even go as far to argue that sighted tests are more accurate and less biased than blind tests. Call that whatever term you feel is most appropriate: unprofessional, lack of journalistic integrity,___
My opinion as stated in the article is that the true sound quality of a component can only be reliably measured via a blind test. Anything less than that may be a willful or unwillful distortion of the truth.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
I changed the title , the word ' dishonesty ' was not appropriate.

I just did a copy and paste of Olive's text title. Someone even shared his explanations about it. He could have changed the title if he wanted.

But what's surprising me the most, is that you changed it whitout asking me. I guess you don't want to hurt people, but your intervention is a little bit strange.
 
Last edited:

Robbo99999

Master Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
6,874
Likes
6,672
Location
UK
I think it's ok that sighted listening tests are done on this site, the measurements are the most important part and what makes this site unique from others, so the listening test is just a little bit of icing on the cake. I would generally be swayed a fair amount by the listening tests on this site though, in terms of if I was considering buying a speaker that was under test here.....i.e. I probably wouldn't buy a speaker that measured ok/quite well if it got a bad listening test review, I'd probably just buy a speaker that happened to have good measurements as well as one that did well in the listening test. So even thought the measurements are the most important part of the reviews, I would still be swayed by the listening test, but no way would I buy a poor measuring speaker that got a positive listening test review.
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
I just did a copy and paste of Olive's text title. Someone even shared his explanations about it. Je could have changed the title if he wanted.

But what's surprising me the most, is that you changed it whitout asking me. I guess you don't want to people, but your intervention is a little bit strange.
It's not strange at all, it's misleading and unnecessarily to label subjective listening as dishonest. A million or more people come here a month and likely it would cause a problem as some of them would latch onto that word and get defensive.

Regards of wether olive used that word few would know that . Now if you'd put the title in quotation marks ... You didn't so they appear as your own words and I delt with them in that context. You also dumped a load of off topic posts that boarded on trolling in another thread so I did not feel you warranted further courtesy, beyond that iv shown you already but allowing that.
 

Foxxy

Member
Forum Donor
Joined
Jul 31, 2020
Messages
24
Likes
43
Location
Austria
Yeah, I think I need about what, 12 years to burn through most of my biases? And that's only because I constantly challenged my hearing and went out and often was suprised my bias didn't hold up with a particular speaker. And honestly, thank god for headphones and studio monitors. Those get rid of so many horrible sound signatures you are exposed to with home HiFi.

Also, giving scores is a horrible idea. That's saying, "Which loudspeaker do you like more!" when I would like to have a multi-axis diagram. If you want you can then make a score out of that but it would be a dishonest value. Scores suck.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
It's not strange at all, it's misleading and unnecessarily to label subjective listening as dishonest. A million or more people come here a month and likely it would cause a problem as some of them would latch onto that word and get defensive.

Regards of wether olive used that word few would know that . Now if you'd put the title in quotation marks ... You didn't so they appear as your own words and I delt with them in that context. You also dumped a load of off topic posts that boarded on trolling in another thread so I did not feel you warranted further courtesy, beyond that iv shown you already but allowing that.

You are clearly upset and I didn't troll.

I created this thread to be able to hear what people have to say about that whitout being under mod pressure to stay on topic. I decided to keep Olive's word because dishonesty can be about yourself. We lie to ourselves pretty often, and I'm sure Olive would agree with that, and it may be one of the reason he chose it.

It brings us to what science allows us to do. It shows that whoever we are we all have the same cognitiv limitations. And ways to help with that is with the scientific method and critical thinking. Bertrand Russell and others will add moral qualities to that like humility, courage, intelligence, etc.

Now it would be nice to have your input about the subject, your frustrations about me personnaly are off topic.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
Speakers are the hardest to not be subjective about. Most of us will never have a chance at any sort of blind audition or comparison of speakers. So other factors will surely enter into it unless you wait and purchase only from measured results.

I sort of agree with Foxxy about scoring. In retrospect I enjoyed every speaker I've ever owned (except one pair which was a trade). I've enjoyed many others my friends owned, and felt a few of theirs were poor choices. I'd hate to score them on a sheet. Yet such is what is needed if we were to get into blind testing. I believe it would work, and results would be surprising, but it isn't practical at all.

The speaker which most impressed me just from hearing it was a pair of Quad ESL63's. An acquaintance purchased some and invited me over to hear them. I left saying, "I've got to get a pair of those". Within a month I had a pair. Used them for 12 years happily. It wasn't just being a panel electrostat either as I owned Acoustats at the time.

There are the Magnepan's of which I've owned a few. I'm reminded of something another fellow said about them. 2nd happiest day of my life was when I purchased these Maggie 3.3R's. Happiest day of my life is when I sold them. They seem good, but something about them rub me the wrong way over time.

All of which explains why I consider Toole's breaking the circle of confusion to be so important. I think it really is the initial step only toward transparent speakers that sound even handed and good anywhere you use them. But previously there wasn't a basis for making progress except for fits and starts and accident. Evolution is very slow compared to directed design. I feel like prior designs had some good, some bad, and sometimes they hit the right spot to be a success and sometimes were a miss.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
It is a valuable and important article, based mostly on one of his AES articles (also worth reading).
Yeh, I wonder if OP has read the research paper or is just going by the blog post. Of what you quoted, let me emphasize this: "Sighted evaluations have their use(I use them sometimes myself), and doing one doesn't necessarily imply willful disregard for the truth. "

They have their place! He uses them himself! Isn't this what I have been saying all along? That we use trained listeners in sighted evaluations because they are quick and in general they are far more correct than wrong?

And how many of you have been lecturing me when I use them when Sean says "don't necessarily imply wilful disregard for the truth?"

He then says this in your quote: "However, people who do sighted listening tests should be aware of their limitations and potential biases, as demonstrated in this article. Unfortunately, many people in our industry routinely report results from sighted listening evaluations without regard to or acknowledgment of these biases or limitations. "

So knowledge of your limitations can help in assessing the reliability of sighted test. Joe blow that knows nothing about speakers, gets a free one from the company to "review" and posts sponsored links should have your antenna up, way up. Putting me in the same bucket, has my antenna going up, way up :D, on your theory being wrong.

I am well aware of my limitations or I would just give you my subjective listening results and no measurements. That I lead with measurements should tell you that I am not walking into subjective testing blind, pun intended.

So did you post this in defense of OP or against? I see that he gave you a like even though the starting line 100% went against his thesis.

I have a lot more to say about this when I get back from dropping off some review gear. For now, you better know the backstory about this paper and testing. If you don't, then you are not appreciating what it is about.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
In the Harman blind speaker tests, do they allow you to go back and change the ratings you gave to speakers after you hear subsequent speakers?
As a practical matter, no in that the test is randomized so there is no going back to anything specific. I had serious problem with this as the first time I heard all the speakers, I had no frame of reference so more or less gave numbers out of thin air (relative to each other) and regretted doing so as the next round started and I recognized at least one of the speakers again (the Martin Logan).

This is in the context of how the test was presented to us when visiting Harman. It may be different when in their own testing where the user may have control of switching of speakers.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
Yeh, I wonder if OP has read the research paper or is just going by the blog post. Of what you quoted, let me emphasize this: "Sighted evaluations have their use(I use them sometimes myself), and doing one doesn't necessarily imply willful disregard for the truth. "

They have their place! He uses them himself! Isn't this what I have been saying all along? That we use trained listeners in sighted evaluations because they are quick and in general they are far more correct than wrong?

And how many of you have been lecturing me when I use them when Sean says "don't necessarily imply wilful disregard for the truth?"

He then says this in your quote: "However, people who do sighted listening tests should be aware of their limitations and potential biases, as demonstrated in this article. Unfortunately, many people in our industry routinely report results from sighted listening evaluations without regard to or acknowledgment of these biases or limitations. "

So knowledge of your limitations can help in assessing the reliability of sighted test. Joe blow that knows nothing about speakers, gets a free one from the company to "review" and posts sponsored links should have your antenna up, way up. Putting me in the same bucket, has my antenna going up, way up :D, on your theory being wrong.

I am well aware of my limitations or I would just give you my subjective listening results and no measurements. That I lead with measurements should tell you that I am not walking into subjective testing blind, pun intended.

So did you post this in defense of OP or against? I see that he gave you a like even though the starting line 100% went against his thesis.

I have a lot more to say about this when I get back from dropping off some review gear. For now, you better know the backstory about this paper and testing. If you don't, then you are not appreciating what it is about.

For what I've read no one ever said that sighted listening were not usefull.

Now the utility will change with the desired goal(s).
I don't fully understand why you use : "with or against the OP". There's no need to have camps or dogmatic positions.
 
Last edited:

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
snip....
They have their place! He uses them himself! Isn't this what I have been saying all along? That we use trained listeners in sighted evaluations because they are quick and in general they are far more correct than wrong?

And how many of you have been lecturing me when I use them when Sean says "don't necessarily imply wilful disregard for the truth?"
snip....

I have a lot more to say about this when I get back from dropping off some review gear. For now, you better know the backstory about this paper and testing. If you don't, then you are not appreciating what it is about.

That we use trained listeners in sighted evaluations because they are quick and in general they are far more correct than wrong?

Do we really know the statement above is true? There is evidence that trained listeners do better in blind testing. It seems like it would make sense they probably also are more discriminating and reliable in sighted tests, but has that supposition itself ever been tested? I think it is probably true, but wouldn't be too surprised to find with all the biases of seeing a design this might overwhelm much of the benefit of trained listeners vs untrained.

Maybe your more extended reply can speak to that if there is more we haven't heard about.
 

SoundAndMotion

Active Member
Joined
Mar 23, 2016
Messages
144
Likes
111
Location
Germany
....
Putting me in the same bucket, has my antenna going up, way up :D, on your theory being wrong.

[snip]

So did you post this in defense of OP or against? I see that he gave you a like even though the starting line 100% went against his thesis.

[snip] For now, you better know the backstory about this paper and testing. If you don't, then you are not appreciating what it is about.
I'm not sure if all these "you"s and "your"s apply to me, but a least a couple do, so I'll make a couple points.

I was unaware of your tiff that I seem to have stepped right into the middle. I didn't post to support or oppose the OP*, rather to support Olive's (and Toole's) excellent paper (with which I'm familiar) and oppose the title of the blog post.

As for the condescending parts, I'm quite able to read and evaluate research papers, especially these that have strong parallels to my own work. But I am, and will remain, anonymous, so feel free to discard my input.

* If we have to choose teams, I don't care who's on what team: I'll choose shirts over skins, since I'm a bit self-conscious about the few extra pounds the decades have gifted me. ;)
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
I would say... If you want to go to 'court' or submit it for peer review using listening test results it will be best to use statistically valid blind results where the test itself is fully documented and checked for validity. There is no way around that.

When you merely want to 'select' gear and just listen (including your eyes) for yourself, go ahead and do this sighted. Just don't go telling people everywhere you did a 'blind' test and this or that was so much better. It may have been or you may have been 'fooling' yourself without realizing it.

One can test anyway in between these extremes.
Only the blind test above will provide real proof.
The rest is anecdotal or at least suspect/questionable.

The test described above is VERY hard to do correctly. One has to know possible pitfalls, measure signals and verify.
The latter test is casual, even if the tester feels it is a rigid test.
The more one knows how to test properly the more the more accurate the 'test' results will be.
 
Last edited:

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
As someone who is not a reviewer and just a regular guy dishonesty probably is too strong. For commercial reviewers it likely isn't.

Just as a for instance, I first liked ESL panels because they didn't have a box. When I first heard a boxless speaker I suddenly realized all speakers I'd heard before were poor because of the boxiness. Likely there were box speakers for which this wasn't true, but I'd not heard them. So my first ESLs were Acoustat Two's which actually are a pretty poor speaker. But they didn't have box. They had pretty extended if uneven response. I could listen to an acoustic guitar in person and hear the sharp pluck of strings. Hear it recorded on Acoustat speakers and it seemed to barely miss the beginning of the transient pluck, getting maybe 75% of it. Your basic box speaker..........whew.......it was like 80% of the transient pluck was over and gone before the cone began to move. It sounded slow, and undetailed and lacking in clarity. Yet I was able to really like a speaker with one good attribute (subjective speed) which in most other ways was a poor speaker.

When I later heard Quad ESL63's it had enough speed, and a balanced frequency response being a good version of the boxless speaker I wanted. I really loved it.

Now some modern speakers are clean, fast, balanced and include a box you don't hear. For a long time, I'd almost hear a boxiness in a speaker just from seeing a box. Seeing only a thin panel makes it easy to imagine it moves more quickly.
 

hardisj

Major Contributor
Reviewer
Joined
Jul 18, 2019
Messages
2,907
Likes
13,908
Location
North Alabama
* If we have to choose teams, I don't care who's on what team: I'll choose shirts over skins, since I'm a bit self-conscious about the few extra pounds the decades have gifted me. ;)

I will never hear the term "shirts vs skins" and not think about this scene from Along Came Polly

PKxW.gif
 
Top Bottom