The frailty of Sighted Listening Tests

[amirm]

Also, what's interesting is that for Loudspeaker T, the sighted and blinded preference scores were essentially identical. This implies that listener preferences are not always affected by sighted listening.

Indeed, that was the single non-harman speaker where listener bias of who they worked for did not apply:

 

[amirm]

Moreover, you have no idea whether you actually know more about any given topic than everyone on this site, especially given the number of pseudonymous users.

I said I know more than the OP because he is quote a research which does not help his cause at all. Seeing how he is just going by the blog post, it is clear he has not read the paper. Have you read the paper?

Where’s the actual evidence that your training has made you able to conduct unbiased sighted tests, while others can’t? Please cite a study or prove it by undergoing bind versus sighted testing of, say, the Revel versus the SVS and a few others.

There is no evidence because what you say about me is not true. I am not unbiased. I have corrected people on that multiple times including this thread. It seems that you all know that you have no case unless you exaggerate my position to the silly extreme of "no bias."

What I have said is that I am a trained listener and I have managed trained listeners. We use them in the industry routinely in sighted tests. Sean Olive as a trained listener also uses sighted evaluations as I noted earlier:

I am fully aware of the limitations of what I volunteer them all the time.

The position taken is that my subjective tests should not even be reported because by definition they are wrong. This is the absurd position we are arguing. Not that my subjective results are guaranteed to be right. If I thought this, I would not measure!

So take a seat and listen and learn the topic before jumping in with anger and emotion with zero contribution to the topic. Heaven knows we already have too many like you spoiling this dish.

 

[amirm]

I don’t like the word “dishonestly,” either. But it’s obvious that it was copied from Olive’s post. It seems like your policing of the title and labeling @patate91’s posts in the SVS thread “trolling” are transparent attempts at sparing Amir from criticism and being contradicted by outside sources.

Sparing me? What planet do you live in? Does it look like I am enjoying any protection from likes of you or OP?

Go after Thomas one more time with such nonsense and I will ban you permanently. His job is difficult enough without dealing idiotic conspiracy theories like this.

 

[solderdude]

I noticed the thread veered of in the direction of questioning Amirs motives/test methods again.
So many questions keep popping up thinking about the goal of this thread (while leaving dr. Olive out of this as he does not seem to be debated/questioned here.)

Since it created controversies in a couple of speakers review I would like to know what's your thought about blind speaker testing and about the reviewer's experience.

Who's opinion are you looking for ?
What is the goal of 'collecting' various opinions ?
Speaker reviews done here on ASR ? (Amir basically) or reviews done elsewhere ?
What controversies are you speaking about and was this settled or remained open with diverging opinions ?
You seem to be looking for other peoples thoughts on 'the reviewer's' experience.
Do you mean Amir's experience in particular or other reviewers in general ?
When you mean other reviewers which ones in particular there are so many ?
Have you based your opinion on the reviewer(s) in question on info found on the web, suspicions about him, personal biases or have you been present during a review by the reviewer(s) in question and came to a different conclusion ?
I mean, speakers, room, listening position, music choice, measurements, knowing limitations of measurements, experience in 'linking' measurements with (sighted or blind) observations.
Are the reviewers opinions gospel ? Are other peoples opinions/papers/publications gospel ? Can people make mistakes (be them honest or not) ? Are there financial gains in reviews (free product or otherwise) ? Is the integrity/authority of the reviewer(s) in question being questioned ?

 

[Steve81]

My job is to pass judgement on a speaker in 24 hours. You want me to do it without listening. I tell you that you are wrong. We would be ridiculed left and right if I did that. And would require putting more weight on single research project than is merited (i.e. the score).

Hi Amir,

FWIW, I highly appreciate the work you put in here. Given the practical limitations involved with reviewing a loudspeaker (and various other components) in a relatively short time frame, I'd say you're doing a bang up job. Having been "in the biz" a while back, I know it's not easy, particularly when anyone can come and dump on your testing methodology, your knowledge, your training, your hearing, etc. When you besmirch the reputation of someone's preferred brand, there's always some flaw that renders your opinion moot. To put it another way, if you're not catching any flak as a reviewer, you're probably not doing a very good job.

As far as the "frailty of sighted listening" is concerned, I certainly won't claim it's ideal, but it does have the immense benefit of being practical. I may not agree with your impressions on everything, but I doubt that would change if you tested every speaker blind versus me at home listening sighted.

 

[tuga]

I'm not criticizing here just trying to learn what is known and how it is done with what limits to expect of trained listeners doing sighted comparisons.

I think of what an optometrist does when you get an eye exam. Look at the letters on the wall. He makes a change, "is that better or worse?" A sighted sighted test. Even an untrained person can go from blurry vision to quite good vision to function in the everyday world this way.

I've fine tuned target curves in room correction for a number of people. I listen with them, they describe what they wished was different or maybe I suggest something I could make better. I tweak the target curve, and do much like the optometrist. I switch from the old one to the new one confirming it is better to them or not (as well as at some point seeing they aren't perceiving a difference as the changes get small enough). So a half hour or so and we can make real beneficial tweaks to the sound. Ones you can blind test afterwards to confirm they are audible and if you wish which is preferred. Had you made each tweak and then blind tested it before making the next you'd take probably a 100 hrs to accomplish the same real result. So blind testing is not always needed for every single thing.

But is there work done I'm not aware of checking trained sighted perceptions vs unsighted. I think of the JND metric. Around a db or so in loudness for most people if you just ask like the optometrist. Yet blind in a series of samples people can detect differences of .2 db in loudness.

A reference is important even sighted. That is where much audiophile sighted comparisons fall apart. The lack of ready switching to a reference. A reviewer gets speakers in, sets them up and listens making his pronouncements a couple weeks later. They should do something more like what you do which is put one speaker in channel A with a known reference in channel B.

In my view, the sighted listening bias is more of an issue when listening for preference than when performing critical listening assessments (tracking down shortcomings).
Some people are of course more prone to being influenced by looks and brand and magazine or forum hype than others.

And as far as critical listening is concerned, ready switching comparisons are not effective when listening for some types of shortcomings which may take days to surface.
With critical listening you don't just run a signal/chirp/set-of-tones and you're done with it. You have to listen to a varied, fit-for-purpose musical programme over a long period of time (days) and wait for issues to reveal themselves. This sould be performed in your listening room with your system.
You can flip over to your reference speakers after a week or two for a reality check but this will be more than anything revealing of mostly tonal balance differences.

 

[tuga]

What about listening bias?

If we live with a particular pair of speakers for a year or two then we've become very accustomed to its sound/tonal balance (i.e. Harman staff which owns company speakers and or listens as part of their job). If we like this sound surely speakers with a different balance may sound akward and ultimately less pleasing. For the preference polls it would be interestings to host people who use single-drivers, horns, ESLs...

 

[Vuki]

Sighted listening test impressions are result of bias, preferences and acoustics in different proportions for each listener. Everyones sighted listening impressions are of equal value.

 

[tw99]

I honestly don't get the heat around this issue. If you don't like the way Amir does the subjective tests, just ignore that part of the review. He's clearly not about to change anything there. It also doesn't invalidate the rest of the data, which is the actual USP of this site.

 

[patate91]

View attachment 76693
If you're trying to make the argument that sighted listening cannot provide useful information about listener preferences, the blog post that you provided doesn't support your viewpoint.

This experiment exposed subjects to an extreme condition of potential bias (i.e. company loyalty, as Harman employees were asked to "rate" their own Harman products...). Yet the "effect" of the bias was only 1 unit on the preference rating scale! So we could infer that in the case of Amir, who is exposed to substantially less pressure to inflate his ratings, we can expect that the bias effect of sighted vs. blinded listening would influence his preferences by less than 1 unit.

Also, what's interesting is that for Loudspeaker T, the sighted and blinded preference scores were essentially identical. This implies that listener preferences are not always affected by sighted listening.

Have you read the study? You cannot just take a part of it and dimiss the conclusion.

 

[patate91]

For those who ask about experienced listener the study specifiy in his sub goal : To examine the influence of having experienced in critical listening.

A part of the conclusion says : "What is surprising is that the effect is so strong, and that it applies equaly to experienced and inexperienced listeners."

" The bottom line : if you want to know how a loudspeaker trully sounds, you would be well advised do the listening tests "blind". "

 

[preload]

Have you read the study? You cannot just take a part of it and dimiss the conclusion.

I interpret experimental data all the time and I'm happy to discuss my interpretation here. You never take the authors' conclusions at face value because they are allowed a certain degree of subjectivity. You always have to look at the methods and data yourself.

 

[threadpool]

It's people like patate01 that give the audio community a bad rep. His circumlocution has wasted enough time as it is. This site offers someone with real credentials, providing unparalleled reviews and data for free. If you don't like what he is doing, stop making the personal attacks and leave?

 

[patate91]

That's a lot of questions I'll do my best.

I noticed the thread veered of in the direction of questioning Amirs motives/test methods again.
So many questions keep popping up thinking about the goal of this thread (while leaving dr. Olive out of this as he does not seem to be debated/questioned here.)



Who's opinion are you looking for ?
What is the goal of 'collecting' various opinions ?
Speaker reviews done here on ASR ? (Amir basically) or reviews done elsewhere ?
What controversies are you speaking about and was this settled or remained open with diverging opinions ?
You seem to be looking for other peoples thoughts on 'the reviewer's' experience.
Do you mean Amir's experience in particular or other reviewers in general ?
When you mean other reviewers which ones in particular there are so many ?
Have you based your opinion on the reviewer(s) in question on info found on the web, suspicions about him, personal biases or have you been present during a review by the reviewer(s) in question and came to a different conclusion ?
I mean, speakers, room, listening position, music choice, measurements, knowing limitations of measurements, experience in 'linking' measurements with (sighted or blind) observations.
Are the reviewers opinions gospel ? Are other peoples opinions/papers/publications gospel ? Can people make mistakes (be them honest or not) ? Are there financial gains in reviews (free product or otherwise) ? Is the integrity/authority of the reviewer(s) in question being questioned ?

Who's opinion are you looking for ? People with expertise and knowledge, there's a lot here.

What is the goal of 'collecting' various opinions ? Learning. Everything related to science is interesting.

Speaker reviews done here on ASR ? (Amir basically) or reviews done elsewhere ? ASR makes me discover other site and people which it's great.

What controversies are you speaking about and was this settled or remained open with diverging opinions ? You'll have to read other threads to fully understand. In short word instead of anwsering directly to my/our questions it turns like I/we are attacking Amir's abilitylies and that we don't apppreciate what he's doing. This is false and it can be easily confirmed by reading posts and thread. (Quick exemple : It seems that I'm arrassing the poor Dr Olive (These are not my words). I simply asked a question, politely, there's no arrassement. And finaly why this attitude? What's wrong To get Dr Olive participating here, awnser our questions about his works.)

You seem to be looking for other peoples thoughts on 'the reviewer's' experience. I'm looking for other people expertise. There's a couple here. Some seems to be science oriented.

Do you mean Amir's experience in particular or other reviewers in general ? Amir's experience is not question. The subject is that everyone are subject to biaises, trained or not, useful or not. Simple advices have been provided (Not by me I don't even have credit for that). The main awnser remains : I'm trained I'm LESS subject to biaises. Again no one seems to challenge his abilities. Cognitiv science, and Olive's paper seems to agree that trained or not experienced or not biaises are equelly strong. For heated some people seems to take this to personnaly.

Have you based your opinion on the reviewer(s) in question on info found on the web, suspicions about him, personal biases or have you been present during a review by the reviewer(s) in question and came to a different conclusion ? No I'm not judging Amir. I'm/we are pointing out that some procedure que faulty and we can back it with scientific research.

Are the reviewers opinions gospel ? No. Reviewer's opinion influence other people. We are all subject to the same biaises. Like I said in the OP : hobbies are great way to learn about science. The Covid situation is a good exemple about distrust and bad science can do. For the rest this is again put in the perspective that we été questioning Amir's integrity or abilities : THIS NOT THE CASE, and it never was.

Note that I think you are part of the experienced people here. Your participation is always appreciated.

 

[beefkabob]

I interpret experimental data all the time and I'm happy to discuss my interpretation here. You never take the authors' conclusions at face value because they are allowed a certain degree of subjectivity. You always have to look at the methods and data yourself.

And the great thing is that @amirm gathers and posts the data. That's what elevates this site to science. That and some pretty sound measurement techniques.

 

[patate91]

I interpret experimental data all the time and I'm happy to discuss my interpretation here. You never take the authors' conclusions at face value because they are allowed a certain degree of subjectivity. You always have to look at the methods and data yourself.

So you certainly knows that the way to dismiss studies and there conclusion is done with other scientific studies, opinions are no help.

 

[TimVG]

Just remember that you're criticizing someone who has 1) substantial credentials in this field 2) has invested a great many hours (and counting) providing us with detailed measurements of all kinds of equipment which would be unavailable anywhere else on the planet otherwise 3) has invested well over a 100.000 dollar of his own money in the equipment of providing said measurements.

To be blunt: he doesn't owe any of us sh*t. If he wishes to post his subjective experiences for whatever reason he chooses, that is his call entirely. If you should attempt to correlate that subjective experience to the actual measurements, you may or may not discover a thing or two. You may also choose to ignore the subjective part entirely - that's easy isn't it?

What is with people feeling entitled to just about anything these days without remotely adding anything substantial themselves. If you want to test blind, by all mean, organize it on your own, and do share the results. But please, if you believe you're going to sway Amir into a blind test or further investigating the speaker in question, by pulling crap like this - you haven't been paying attention.

 

[lashto]

I noticed the thread veered of in the direction of questioning Amirs motives/test methods again.
So many questions keep popping up thinking about the goal of this thread (while leaving dr. Olive out of this as he does not seem to be debated/questioned here.
Who's opinion are you looking for ?
What is the goal of 'collecting' various opinions ?
Speaker reviews done here on ASR ? (Amir basically) or reviews done elsewhere ?
What controversies are you speaking about and was this settled or remained open with diverging opinions ?
You seem to be looking for other peoples thoughts on 'the reviewer's' experience.
Do you mean Amir's experience in particular or other reviewers in general ?
When you mean other reviewers which ones in particular there are so many ?
Have you based your opinion on the reviewer(s) in question on info found on the web, suspicions about him, personal biases or have you been present during a review by the reviewer(s) in question and came to a different conclusion ?
I mean, speakers, room, listening position, music choice, measurements, knowing limitations of measurements, experience in 'linking' measurements with (sighted or blind) observations.
Are the reviewers opinions gospel ? Are other peoples opinions/papers/publications gospel ? Can people make mistakes (be them honest or not) ? Are there financial gains in reviews (free product or otherwise) ? Is the integrity/authority of the reviewer(s) in question being questioned ?

you just used ASR's entire quota of question marks for this month. A special gift to relax after all that work

 

[preload]

So you certainly knows that the way to dismiss studies and there conclusion is done with other scientific studies, opinions are no help.

Actually, no, that's really not quite right either. Nobody is "dismissing" the study here, and unless a study's methodology and statistics are definitive (rare), not everyone will have the same interpretation of the data as the author. This is why you can't simply cut/paste an author's conclusion paragraph and consider it the final truth. This is precisely why journal authors spend so much time describing their experimental methods and statistics - its so that others can verify their scientific process.

@patate91, you seem to be trying to present yourself as an expert in psychoacoustics and scientific research. Yet something isn't adding up because myself and others are needing to explain basic things to you, and your replies so far have been quite atypical for anyone who has spent any time in the sciences.

My guess is that you are likely somewhere in your secondary school or university education and have taken a course or two that includes a laboratory section. Am I correct? And if not, please correct me and share what background/education you have in this field or in the sciences?

 

[patate91]

Actually, no, that's really not quite right either. Nobody is "dismissing" the study here, and unless a study's methodology and statistics are definitive (rare), not everyone will have the same interpretation of the data as the author. This is why you can't simply cut/paste an author's conclusion paragraph and consider it the final truth. This is precisely why journal authors spend so much time describing their experimental methods and statistics - its so that others can verify their scientific process.

@patate91, you seem to be trying to present yourself as an expert in psychoacoustics and scientific research. Yet something isn't adding up because myself and others are needing to explain basic things to you, and your replies so far have been quite atypical for anyone who has spent any time in the sciences.

My guess is that you are likely somewhere in your secondary school or university education and have taken a course or two that includes a laboratory section. Am I correct? And if not, please correct me and share what background/education you have in this field or in the sciences?

Not at all, you read what I've wrote, I clearly state that I'm not an expert (clear text). I even proposed to bring Dr Olive to exchange with us.

Opinions < scientific papers < meta analisys (which may lead To scientific consensus). Sure a single study is not enough that's why I shared pkane's post. Now I speak french, so what can I share is limited. With good faith it's easier To find studies about cognitiv biaises.

Now you're trying To bring it down to me personnaly, I won't awnser to that, this is unneccessary.