• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

The frailty of Sighted Listening Tests

OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
I'm curious why you don't respond to any of the points that Amir has made regarding the paper and blog post. He even helpfully bolded some statements / clarifications / conclusions. Since you cited the paper / blog post, do you agree with his analysis? Or you disagree?

Because I'm not an expert and I shared other people's work. Also I don't anwser because he is cherry picking on precise parts of the study, never adress the conclusion of the study or the what the whole study shows us. Also I tried to anwser, as well as other people, but it always turn as it was an attack on his abilities (strawman). Several points are also avoided.

Like I said above I'll give more weight on scientific papers and scientist. As I know Amir is not a scientist, he's not creating science and he claimed that he's less biaised because he's trained and his experience.

About science the way to debate or argues about studies is to produce science and more studies, personnal opinion are no help.

I proposed an exchange with a scientist, the one who worked on what I shared. I'm sure he's more apt than me to anwser arguments about it.

And finaly I created this thread to have other peoples input. My goal were and are not to start a debate between me and Amir. Sure I'm no expert in the audio field but I can be good at critical thinking. And warning flags are pretty easy to spot. Now I appreciate all the work that have been done so far, I don't fully understand why basic advices like : listen before look at measurments create that much of heated discussion.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
@patate91 What exactly is the point you're trying to make? What is your argument?

My point here is : what people and scientists have to say about sighted vs blind listening. You can also re-read my first post.

*EDIT* For what I've read about cognitiv biaises no training help. Critical thinking, scientific method do, but it is not perfect and always field specific.
 
Last edited:

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
Because I'm not an expert and I shared other people's work. Also I don't anwser because he is cherry picking on precise parts of the study, never adress the conclusion of the study or the what the whole study shows us. Also I tried to anwser, as well as other people, but it always turn as it was an attack on his abilities (strawman). Several points are also avoided.

Like I said above I'll give more weight on scientific papers and scientist. As I know Amir is not a scientist, he's not creating science and he claimed that he's less biaised because he's trained and his experience.

About science the way to debate or argues about studies is to produce science and more studies, personnal opinion are no help.

I proposed an exchange with a scientist, the one who worked on what I shared. I'm sure he's more apt than me to anwser arguments about it.

And finaly I created this thread to have other peoples input. My goal were and are not to start a debate between me and Amir. Sure I'm no expert in the audio field but I can be good at critical thinking. And warning flags are pretty easy to spot. Now I appreciate all the work that have been done so far, I don't fully understand why basic advices like : listen before look at measurments create that much of heated discussion.

Yes, it seems like you're hanging onto the 'scientist' label for dear life.
At some point you do have to agree on something, otherwise you're just trolling / someone with a 'dishonest' agenda.

This was what I wrote in the SVS thread.
Here are a set of 4 simple questions with yes/no answers:
1. Do you accept that a professional odor tester can detect odors in perfumes / fragrances that regular, untrained humans cannot?
2. Do you accept that a professional pantone color expert can detect minor color variations that regular, untrained humans cannot?
3. Do you accept that a professionally trained marathon runner can run for longer distances that regular, untrained humans cannot?
4. And finally, do you accept that a professionally trained listener can detect acoustic variations, AND have reduced biases during sighted listening, that regular, untrained humans cannot?

For anyone with a bone to pick with Amir's subjective opinions, in answering the above 4 questions, I believe we will know on a bias spectrum, where the person stands in his trust level of anything produced by Amir.

So as a simple exercise, can you answer the 4 questions?
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
Yes, it seems like you're hanging onto the 'scientist' label for dear life.
At some point you do have to agree on something, otherwise you're just trolling / someone with a 'dishonest' agenda.

This was what I wrote in the SVS thread.



So as a simple exercise, can you answer the 4 questions?

Sure I'll stick to science and scientist, the forum is called : Audio SCIENCE Review.

1. Yes I accpet it because there works are done with rigorous protocols.

2. Yes, but the ability of spoting color change is out of scope of cognitiv biaises.

3. Yes, but link with cognitiv biaises are inexistant.4.

4.Yes and No, that's not what science reaserches tell us. You can read what Pkane shared to have an introduction.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
I'll take a break it's getting late, english is not my mother tongue and it's taking me a lot of energy. I'll come back tomorrow.

It started good with peoples takes on the subject, I hope more will participate.
 

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
Sure I'll stick to science and scientist, the forum is called : Audio SCIENCE Review.

1. Yes I accpet it because there works are done with rigorous protocols.

2. Yes, but the ability of spoting color change is out of scope of cognitiv biaises.

3. Yes, but link with cognitiv biaises are inexistant.4.

4.Yes and No, that's not what science reaserches tell us. You can read what Pkane shared to have an introduction.

Pkane and Amir are both sharing 'science' stuff, why do you dismiss Amir and revere Pkane? Is Pkane a scientist?
 

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,554
Likes
1,701
Location
California
My point here is : what people and scientists have to say about sighted vs blind listening. You can also re-read my first post.

I've read your first post many times as well as your posts in the SVS thread. I honestly can't figure out what your point is. And it's likely that other people can't either. If you're wondering why some folks have commented that you seem like a troll, this is one of the reasons.

*EDIT* For what I've read about cognitiv biaises no training help. Critical thinking, scientific method do, but it is not perfect and always field specific.

So, is this what you're trying to say? Specifically, that undergoing Harman listener training does not reduce the influence of cognitive biases on listener perception?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,376
Likes
234,497
Location
Seattle Area
Good points and the "experienced listeners" vague definition is the weak part of the study but it is still the basis for Harman conducting their listening tests blind. Logic says that if Harman thought their trained listeners were just as reliable in sighted evaluations, they would have no need to run the tests blind.
Oh no. Even though we routinely used our trained listeners sighted, final outcomes always required blind testing. We would usually reserve this for final stages of development. I know Harman can't release a new speaker without passing the blind test.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,376
Likes
234,497
Location
Seattle Area
Sure I'll stick to science and scientist, the forum is called : Audio SCIENCE Review.
Yes, it is called audio science review. We are not here to invent new science. If you are going to do that after months of research, sure, you perform double blind tests to go with it. We are here to review, i.e. publish and analyze, science and engineering. We are not a research organization.

My job is to pass judgement on a speaker in 24 hours. You want me to do it without listening. I tell you that you are wrong. We would be ridiculed left and right if I did that. And would require putting more weight on single research project than is merited (i.e. the score).

Are we done with debating tactics?
 

GD Fan

Addicted to Fun and Learning
Forum Donor
Joined
Jan 7, 2020
Messages
932
Likes
1,699
Location
NY, NY USA
I'll take a break it's getting late, english is not my mother tongue and it's taking me a lot of energy. I'll come back tomorrow.

It started good with peoples takes on the subject, I hope more will participate.
Good call taking a break. You really aren't scoring any points anyway.

When you asail someone's (well established) bona fides, you might want to bring more to the fight. Like your own bona fides, which you seem to confess are none.
 

GD Fan

Addicted to Fun and Learning
Forum Donor
Joined
Jan 7, 2020
Messages
932
Likes
1,699
Location
NY, NY USA
I'll add that you distinctly come across to me as argumentative and lacking respect for those more learned than yourself. Everyone here owes a debt of gratitude to the founder and yet you persist in an unnecessary line of attack that seems to further the discourse none whatsoever.
 

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
I changed the title , the word ' dishonesty ' was not appropriate.

It's not strange at all, it's misleading and unnecessarily to label subjective listening as dishonest. A million or more people come here a month and likely it would cause a problem as some of them would latch onto that word and get defensive.

Regards of wether olive used that word few would know that . Now if you'd put the title in quotation marks ... You didn't so they appear as your own words and I delt with them in that context. You also dumped a load of off topic posts that boarded on trolling in another thread so I did not feel you warranted further courtesy, beyond that iv shown you already but allowing that.

I don’t like the word “dishonestly,” either. But it’s obvious that it was copied from Olive’s post. It seems like your policing of the title and labeling @patate91’s posts in the SVS thread “trolling” are transparent attempts at sparing Amir from criticism and being contradicted by outside sources.
 
Last edited:

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
Klippel is not science. It is a set of measurements. Those measurements are very difficult to interpret as "buy don't buy" against countless other speakers with similar looking measurements. 1 dB peak at 600 Hz is not the same as 1 dB peak at 1.5 kHz. Yet the score may be identical. We need to bridge that gap so people can purchase speakers without listening to them which is the norm today.

If we had a scoring system where we could all stand behind it so much that if it said speaker a is better than b, that would be the "truth," then sure, I would not need to do listening tests. But we are not there. Scoring system is like a compass that shows you north. It is not a turn by turn navigation system for driving in the city.

Also, when I first started to do measurements, people kept asking me what I recommend. I refused to say so. We had a bunch of debate threads about them. Eventually I got tired of answering those questions in private and in public and added the recommendations. That has proven to be hugely popular and rarely controversial. Today, I cannot, without listening to a speaker, give such recommendations. So as much work and aggravation it has turned out to be, I listen and provide this as a factor in my recommendation.

And no, not all "human beings" are the same. Which one of you has been exposed to nearly 80 speakers in the last 7 months where you could compare and correlate measurements to what you could hear? The answer is none. In other words, I am not situated like any of you. There are many things that apply to you that don't apply to me and vice versa. We rely on informed opinion of experts in real life all the time. Not sure why it is such a big deal to follow the same in audio.

This completely contradicts your copious previous statements about other gear that’s been tested, the use of SINAD and linearity as clear “better than ___” markers.

If we’re now at “measurements aren’t science, they’re just data, and experienced listeners’ views matter,” that’s fine. But you’d better be consistent and understand that there are lots of audio writers, professionals, and even hobbyists who hear more gear than you do.

Oh, blind tests can be just as questionable. Blinding removes some bias, it doesn't automatically make it a truth teller.... I take a trained listener sighted tests over untrained and improper blind testing every day of the week and twice on sunday!

If only this had been uttered earlier on this site, we could’ve avoided lots of debates. Only how that Amir’s own subjective views are being questioned by measurements can it be uttered. What a mess.
 
Last edited:

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
And please don't try to be clever with debating tactics within your comment above. "You go talk to the researcher if you think you know more." I know more than you. That is the key point. Make yourself more knowledgeable so that you can defend yourself instead of using such tactics.

This is the same appeal to authority fallacy nonsense you were pulling in the MQA thread. Your working on WMA, mentioned above, has no bearing on this. (Indeed, if I were considering it, I’d see it as a net negative.)

Moreover, you have no idea whether you actually know more about any given topic than everyone on this site, especially given the number of pseudonymous users.

Where’s the actual evidence that your training has made you able to conduct unbiased sighted tests, while others can’t? Please cite a study or prove it by undergoing bind versus sighted testing of, say, the Revel versus the SVS and a few others.

Otherwise, everyone can provide their own subjective opinions of gear, not just you.
 
Last edited:

beefkabob

Major Contributor
Forum Donor
Joined
Apr 18, 2019
Messages
1,636
Likes
2,074
My experience of listening is that listening critically is difficult. All sorts claim to be able to hear this or that among audio components, like how wires or wooden blocks or expensive capacitors lift veils. There's a ton of bullshit.

People can hear things. The drum mic is clipping. The woofer has a resonance. It gets a lot harder to hear subtle harmonics and pick them out. It's pretty clear when there are large frequency dips and spikes. It's not so clear when those are small. Where the frequency issues lie matters.

Then you get into bias. The box. The brand. The weight. The price.

To truly sort through all this would be an expensive and difficult process, beyond the means of one guy in his garage and listening room.

That said, there is a lot one guy can do. We get measurements. We discuss them alongside research. We get subjective impressions.

I don't think we can say, without a lot more research, exactly how every frequency dip and peak, exactly how each curve, and exactly how distortion and countless other factors all play together. We have a reasonably good sense though. Flatter response, especially trending gently downward, lower distortion, and a wide listening window all make music sound better to more people.
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
I don’t like the word “dishonestly,” either. But it’s obvious that it was copied from Olive’s post. It seems like your policing of the title and labeling @patate91’s posts in the SVS thread “trolling” are transparent attempts at sparing Amir from criticism and being contradicted by outside sources.
' obvious ' to whom?

Yes because I'm famous for sparing Amirm from criticism! I do it all the time don't I.

Anyone else want to talk nonsense and throw stones about lol
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
If anybody cared to read what iv said in relation to amirm's sighted listening evaluations they would see that I too take them onboard with a degree of caution. This is what one dose with information, you place your own sense of value on it.

Where things got a bit challenging here is where Amirm seemed to suggest his sighted listening was impervious to bias, he was in my opinion a bit absolute there and it's created this .

What's being done by Amirm is hugely valuable, this includes his sighted listening that offers extra information and puts the raw data in context. As with anything there's time limits in what can be done but that should not mean we do nothing.

Some of you seem to think unless something is perfect and exposed to extreme scientific rigour is not worth doing at all. To those people I'd say , have at it and do some of this yourselves!

You have measurements, you have some informed subjective investigation of those speaker measurements. You are all free to put a value on those things as you see fit . There nothing much constructive to be gained by playing debating games , endless manipulative games , trying to rope in 3rd parties to bat for you ' against ' Amirm . It's not constructive and it's not something I'm willing to put up with much longer. The wider topic that's being discussed here is welcome and I'd encourage that to continue, just without the debating games that are focused on harassing Amirm and now it seems poor Mr olive .
 

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,554
Likes
1,701
Location
California
Since it created controversies in a couple of speakers review I would like to know what's your thought about blind speaker testing and about the reviewer's experience.

Here's what Sean Olive wrote a couple of years ago :

https://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html?m=1

1596605058182.png
If you're trying to make the argument that sighted listening cannot provide useful information about listener preferences, the blog post that you provided doesn't support your viewpoint.

This experiment exposed subjects to an extreme condition of potential bias (i.e. company loyalty, as Harman employees were asked to "rate" their own Harman products...). Yet the "effect" of the bias was only 1 unit on the preference rating scale! So we could infer that in the case of Amir, who is exposed to substantially less pressure to inflate his ratings, we can expect that the bias effect of sighted vs. blinded listening would influence his preferences by less than 1 unit.

Also, what's interesting is that for Loudspeaker T, the sighted and blinded preference scores were essentially identical. This implies that listener preferences are not always affected by sighted listening.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,376
Likes
234,497
Location
Seattle Area
If only this had been uttered earlier on this site, we could’ve avoided lots of debates. Only how that Amir’s own subjective views are being questioned by measurements can it be uttered. What a mess.
You are very confused but you make the same mistake of overgeneralizing (something people with no knowledge of the topic do).

Take speaker cables. If someone said they sound different, we would all stand on one side of the fence and demand a double blind test. Why? Because measurements would show them to be identical in audible band. So the notion of subjective test resulting in a wildly different outcome requires a controlled test that is blind since the sighted one seems quite invalid.

Now take the identical situation and substitute a speaker for the cable. Now if someone says the two speakers sound different, do you demand a double blind test? Of course not. We know speakers can and do sound different. This is so obvious that it would be the height of silliness to demand a double blind test.

Take in-room measurement. A single microphone measurement can never show you what your two ears and a brain perceive. It simply can't. Whereas electronic measurements do not have that problem whatsoever. We can measure two channels with ease and there is no impact of room or listener location to worry about.

Bottom line, context is everything. When it comes to sound reproduction, we start with imperfection. Measurements and the right ones help a lot but they are nowhere near as conclusive as they are in electronics where we have ruler flat response and vanishingly low distortion amounts.
 
Top Bottom