• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

The frailty of Sighted Listening Tests

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
I think an important point is not discusssed : "Double sighted" listening.

What's the robustness of an evaluation if I get all technical data before I do my sighted evaluation.

For exemple I'll take the one about wine I got earlier. I'm an expert I taste more than 200 wines this year (That's totally false, you'll see as per my descriptions) Someone ask me to evaluate a wine and he tells me : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Let say there's two outcome.

First : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Second : It taste like a Californian wine. Oak, Orange and herbs. Medium robe. Dry ending.

Again how robust will you rate my review?
*EDIT* And what can I do to improved my review?


The sighted evaluation will still be pretty robust. Again, we're not talking about the man on the street. We are talking about someone who

1. Has the listening training and can be reliably proven to achieve a reasonable level (Trained)
2. Paid by corporations and depended on for his work to provide insights into the speakers being evaluated (Practiced)

A trained listener is an actual job, and they are paid to provide professional advice regarding specific audio matters. So within their area of expertise, their words can be trusted.
https://www.hourdetroit.com/community/harman-international-car-audio-tech-test-listeners/
http://www.stroudaudio.com/skilled-listener-training.htm

And no, doesn't mean they only use sighted evaluation. Both sighted and blind tests are done, as have been mentioned by Amir earlier, depending on the stages of work.
 
D

Deleted member 17820

Guest
This, this is the real Frailty of sighted listening or blind listening tests. Age, variable hearing acuity among groups of people, and a not insignificant number of younger people who have accelerated hearing loss due to using headphones which end up being played at louder volumes than loudspeakers on average apparently.

Alas I'm old enough I need to follow Toole and keep my opinions to myself mostly.

I am SOOOOO confused about why this even matters?
Its up to the speaker seller, designer, engineer to build a speaker for a purpose and it's up to the end user to decide if it suits them?

EVEN if you had 20 trained listeners with the same hearing and a perfect A/B test what would the point be? What usefull information would it give you.

AUDIO/sound is consumed, just like food. McDonalds is VERY concerned about selling food. They will do taste testing to see what the majority of people like, then they make food that sells.
Then there are foodies and niche places that serve ALL KINDS OF WEIRD and feakish food that just a fraction of people will eat. A perfect blind test will never account for niches and a perfect mass consumer test will result in a mass market product, not neccesarly a product that has any engaging elements.

IF you were to argue that you could get 10 audiophiles in the same room to agree on anything....well now that sounds amazing!
 
D

Deleted member 17820

Guest
This, this is the real Frailty of sighted listening or blind listening tests. Age, variable hearing acuity among groups of people, and a not insignificant number of younger people who have accelerated hearing loss due to using headphones which end up being played at louder volumes than loudspeakers on average apparently.

Alas I'm old enough I need to follow Toole and keep my opinions to myself mostly.

Hearing loss doesn't make a persons opinion less true or important, it just gives you a special window into what you still can here. Mixers use this aproach in the studio for mixes all the time. One set of monitors highlets mids, the other highs, and mains highlight the whole mix. Each set gives a true and accurate window into part of the mix. If all you have left is low freuency hearing and your complaint is regarding the low freuwnecy of a speaker then that is valid.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,771
Likes
37,635
Hearing loss doesn't make a persons opinion less true or important, it just gives you a special window into what you still can here. Mixers use this aproach in the studio for mixes all the time. One set of monitors highlets mids, the other highs, and mains highlight the whole mix. Each set gives a true and accurate window into part of the mix. If all you have left is low freuency hearing and your complaint is regarding the low freuwnecy of a speaker then that is valid.

Well if you are working with compromised measuring instruments you'd prefer to find someone with good instruments.

There are three ways age reduces your acuity (probably others too)

Signal to noise ratios, many get tinnitus which means around some frequencies you have an elevated noise floor.

Frequency response reduction on the high end. If you can't hear something you can't hear something. I was recording some friends a couple years back and on some songs they used shakers. Most of us thought you could barely hear them, but some teen age girls in the group said they were pretty loud, as in too darn loud. Well the fundamental of those were at 13 khz. So while older folks could hear it, the level was much reduced comapared to someone with good response up near 20 khz still. I dropped speed on the recording by 50% and could hear how loud the sizzling shakers really were. I toned them down a bit.

Another is masking. As your hearing ages it has less ability to mask sounds, basically the filters for such become less sharp. So you can't hear into the music as deeply. Complex sounds become a cacophony when younger listeners don't hear it as such. Less cocktail party effect and more cocktail party noise.

So yes, in some circumstances of evaluating sound quality it would make my opinion less true. Or less useful for those younger. Or do you want all your audio reviews done by those age 80 and above? Of course with the demographics of high end expensive gear maybe I am just entering my golden age when my perceptions will mean more to the right kind of people. ;)
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,771
Likes
37,635
I am SOOOOO confused about why this even matters?
Its up to the speaker seller, designer, engineer to build a speaker for a purpose and it's up to the end user to decide if it suits them?

EVEN if you had 20 trained listeners with the same hearing and a perfect A/B test what would the point be? What usefull information would it give you.

AUDIO/sound is consumed, just like food. McDonalds is VERY concerned about selling food. They will do taste testing to see what the majority of people like, then they make food that sells.
Then there are foodies and niche places that serve ALL KINDS OF WEIRD and feakish food that just a fraction of people will eat. A perfect blind test will never account for niches and a perfect mass consumer test will result in a mass market product, not neccesarly a product that has any engaging elements.

IF you were to argue that you could get 10 audiophiles in the same room to agree on anything....well now that sounds amazing!

If only the audio industry used blind testing to even 10% the level the food industry does. Alas the size of the market probably makes it less of a good value compared to food.

This is also a false comparison on your part. If we had taste reproduction, then we need to find out which taste reproducer is most accurate. Then that accurate reproducer would be the best one for both those MickieD sensibilities and those who prefer freakish food where we'd need to reproduce the taste of something very unusual to a high degree of fidelity. That is like your best speaker. The food industry does approach something like this differently. They make additives that to a high degree of reliability give a taste to food that doesn't otherwise have that taste.
 

krabapple

Major Contributor
Forum Donor
Joined
Apr 15, 2016
Messages
3,197
Likes
3,767
In any case, I'm not sure at all what you are actually objecting to. Research shows that biases involved in sighted evaluations alter people's perception of the sound. What have I written that you think is incorrect, ignorant...or whatever brought about the tone in your reply?

Thanks.

You are, for reasons unfathomable without resorting to uncharitable speculation, leaving out what this 'alteration' amounts to in practice. It doesn't increase accuracy of the perception. In fact, you can get a person to report that the same device sounds different, if you make them think (via sight) that the test actually involves two devices. Such difference is of course entirely imaginary. That's how far 'bias' can go.

THIS IS WHY SENSORY PSYCHOLOGY -- whether its focus is the study of hearing, taste, smell, touch -- RELIES ON 'BLIND' PROTOCOLS.

It is why Toole, Olive, etc use them. When they point out that sight alters people's perception of sound, they are not positing benefits of sighted evaluation. They are warning against its deficits.

So please explain what good it would do to reintroduce the perceptual noise that DBT protocols are designed to filter out?
 

krabapple

Major Contributor
Forum Donor
Joined
Apr 15, 2016
Messages
3,197
Likes
3,767
I am SOOOOO confused about why this even matters?
Its up to the speaker seller, designer, engineer to build a speaker for a purpose and it's up to the end user to decide if it suits them?

EVEN if you had 20 trained listeners with the same hearing and a perfect A/B test what would the point be? What usefull information would it give you.

AUDIO/sound is consumed, just like food. McDonalds is VERY concerned about selling food. They will do taste testing to see what the majority of people like, then they make food that sells.
Then there are foodies and niche places that serve ALL KINDS OF WEIRD and feakish food that just a fraction of people will eat. A perfect blind test will never account for niches and a perfect mass consumer test will result in a mass market product, not neccesarly a product that has any engaging elements.

IF you were to argue that you could get 10 audiophiles in the same room to agree on anything....well now that sounds amazing!

Ah, and here it is, we are finally at the level of reasoning of your average subjectivist 'audiophile' forum. It never fails to appear , eventually.

"People like what they like. So why bother with this science stuff? Just enjoy!"

Just go away.
 

napilopez

Major Contributor
Forum Donor
Joined
Oct 17, 2018
Messages
2,146
Likes
8,718
Location
NYC
I am SOOOOO confused about why this even matters?
Its up to the speaker seller, designer, engineer to build a speaker for a purpose and it's up to the end user to decide if it suits them?

EVEN if you had 20 trained listeners with the same hearing and a perfect A/B test what would the point be? What usefull information would it give you.

AUDIO/sound is consumed, just like food. McDonalds is VERY concerned about selling food. They will do taste testing to see what the majority of people like, then they make food that sells.
Then there are foodies and niche places that serve ALL KINDS OF WEIRD and feakish food that just a fraction of people will eat. A perfect blind test will never account for niches and a perfect mass consumer test will result in a mass market product, not neccesarly a product that has any engaging elements.

IF you were to argue that you could get 10 audiophiles in the same room to agree on anything....well now that sounds amazing!
It matters because, presumably, most people on audio forums go there to find out what sounds good.

If the reliability of the listener is compromised, their ability to make recommendations or statements about audio quality that are applicable to a large number of listeners is reduced. Everyone has the right to enjoy what they like, no matter how colored or inaccurate it may be. It's when they recommend it to others without recognizing the context of their own abilities that it becomes a problem.

I don't totally disagree with your last couple of paragraphs, but the food analogy also doesn't fully work because in food there isn't a 'truth' that people can strive for like there is in audio. Maybe the comparison will work in 300 years, when we can have a kitchen device analogous to speakers that lets us 'play' food from 'live' restaurants.
 

MattHooper

Master Contributor
Forum Donor
Joined
Jan 27, 2019
Messages
7,332
Likes
12,292
You are, for reasons unfathomable without resorting to uncharitable speculation, leaving out what this 'alteration' amounts to in practice. It doesn't increase accuracy of the perception.

Where did I even suggest that sighted evaluation "increased the accuracy of perception?" (In terms of the sound).

(Hint: nowhere).

In fact, you can get a person to report that the same device sounds different, if you make them think (via sight) that the test actually involves two devices. Such difference is of course entirely imaginary. That's how far 'bias' can go.

THIS IS WHY SENSORY PSYCHOLOGY -- whether its focus is the study of hearing, taste, smell, touch -- RELIES ON 'BLIND' PROTOCOLS.

It is why Toole, Olive, etc use them. When they point out that sight alters people's perception of sound, they are not positing benefits of sighted evaluation. They are warning against its deficits.

So please explain what good it would do to reintroduce the perceptual noise that DBT protocols are designed to filter out?

Have you, I suppose, skipped all my other posts in this thread? That seems the most reasonable explanation as to why you think you are telling me things you think I don't know, as if I haven't already written about those issues in this very thread!

The point of blind testing: https://www.audiosciencereview.com/...-of-sighted-listening-tests.15114/post-477584

I'll give it one more whirl: when you want to strictly investigate the sound of a speaker on a listener's perception (or preference), you of course should control for the variable of sighted biases. We agree?

But why?

It's because, as I and almost everyone else on this forum have pointed out before, and now you have re-iterated: Sighted biases can influence perception. We agree?

Therefore: there is not only the phenomenon of how a speaker's sound (only the sound) is perceived. There's also a phenomenon as to how a speaker's sound is perceived COMBINED with variables involved in sighted bias (how it looks, it's design, or how much it costs, various expectation effects).

Right? Otherwise there would be no need to reduce the influence via blind tests when you are ONLY testing the effects of the sound.

Therefore: the effects of sighted bias - the way they change perception of the sound! - can be of interest as well, something to study, and something to consider. (Just as I change the perception of sound in my job).

And, as we've gone through on this forum before, there is an issue inherent in the blind testing studies: they can predict what people are most likely to prefer under the blind conditions of the test, and hence narrow down the variable with respect to just the sound. But life and human brains being messy things, the predictive power for listener preferences or user satisfaction once OUTSIDE the lab becomes much more shaky, due to the conditions under which most people ACTUALLY tend to purchase and listen to their speakers.

This is what I meant in pointing out the blind testing is great for reducing the influence of such variables in the lab, but if in practical normal-life scenarios those sighted variables (influences) are normally commingled with the sound, and this actually alters the perception of the sound (most people buy speakers in sighted auditions, and listen under sighted conditions) that's a *real* change in perception. (Just like an optical illusion is a real perception).

In principle the exact effects of the looks of a speaker could be studied for it's effect on listener perception, just as changes in sound caused by different speaker designs is correlated to listener perception/preferences. Toole et al, as far as I'm aware, have done *some* tests on this, such as the ones I referenced. But that seemed mostly to simply establish the unreliability of sighted tests, helping to justify the program of using blind tests. But the bulk of the research from then had to do with identifying specific speaker designs in regards to only the influence of their sound, not their looks (which is why all the blind test research). I'm not aware of any similarly involved systematic correlations between how a speaker looks (or other bias factors) as to how it alters specific perceptions of the sound. If there were, you'd have a scientific standard for looks as rigorous as the one Harman Kardon has produced for the sound. (Though, perhaps HK has actually done just this work, leading to how their speakers look visually. I'd be curious to know).

Another consequence of the fact that bias effects actually CHANGE the perception of sound is this: It could be completely reasonable to make a purchase based on the speaker you prefer in sighted testing. And that is EVEN IF you would make a different decision if based only on blind testing. So take two speakers that are pretty similar in performance, and under blind conditions I slightly prefer speaker A over B. But in sighted conditions, I perceive speaker B as sounding somewhat better than A. Why? Well, maybe speaker B just looks like a million bucks compared to speaker A. Speaker A is plain looking while speaker B has a gorgeous high quality wood finish, cool drivers, a neato design. Whatever aesthetic issues are causing my change in perception, the FACT IS I'm experiencing a change in perception. And I may want to simply avail myself of that phenomenon. "I may choose speaker A in blind tests, but under the conditions I'll be using this speaker, I perceive the better overall experience listening to speaker B." That is a completely rational way to make a decision. It's advised, made knowledgeably, with a clear understanding of one's goal and which action better fulfills the goal.

So, this is one reason I raise the issue of how blind testing is certainly immensely valuable for weeding out variables to understand and predict the contribution of *just the sound* to listener preferences. But once you are outside the lab those variables exist to either learn more about, or if one wants, to exploit in how one makes a purchase.
 
Last edited:

Dale Gribble

Member
Joined
Aug 7, 2020
Messages
8
Likes
11
You really seem obsessed with snark tactics and rhetoric over charitable interpretation and good faith debate.

You’re not “citing” a position with an 11 year old forum post - you’re finding old comments (not scientific statements of fact...) that may or may not be maintained without any right of reply.

It just seems poor form to me. You obviously don’t agree - but then again, you’re one of the few on this forum who seems incapable of respectful, calm discussion.

This is Rusty. I’m responding here because I cannot log into my Rusty account.

Nothing I have said has been disrespectful. I haven’t used any vulgarity or ad hominem attacks. Disagreeing isn’t disrespectful.

Nor is there anything wrong with quoting someone’s public writing. Amir was an adult in 2011. It’s not as if I’m throwing someone’s posts from when they were a teen in their face. He also has the right to reply to any of his old posts and to my posts.
 

Coach_Kaarlo

Active Member
Joined
Apr 27, 2020
Messages
196
Likes
222
Location
Sydney
You are, for reasons unfathomable without resorting to uncharitable speculation, leaving out what this 'alteration' amounts to in practice. It doesn't increase accuracy of the perception. In fact, you can get a person to report that the same device sounds different, if you make them think (via sight) that the test actually involves two devices. Such difference is of course entirely imaginary. That's how far 'bias' can go.

THIS IS WHY SENSORY PSYCHOLOGY -- whether its focus is the study of hearing, taste, smell, touch -- RELIES ON 'BLIND' PROTOCOLS.

It is why Toole, Olive, etc use them. When they point out that sight alters people's perception of sound, they are not positing benefits of sighted evaluation. They are warning against its deficits.

So please explain what good it would do to reintroduce the perceptual noise that DBT protocols are designed to filter out?

Agree with your point.

Medically the placebo effect runs at about 40% efficacy - that's to say 40% of people report feeling better when given nothing of medicinal value. Most things are only in our heads as it were - what we believe can actually alter our subjective reality.
 
Last edited:

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,306
Location
uk, taunton
There's many examples in this thread of ' the frailty of forums as communication platforms ' especially in complex subject matter . Much is lost much is added that should not be and serves as a distraction.

i think of ASR as amirm's garden , he's opened it up to the public, the locals enjoy it , some plant and help weed , some help with the buying of gardening tools , there's some open discussion on planting but the landscaping is mostly set . Some come and don't like it , maybe they mention why , maybe they leave and find another or make their own .

its plain bad manors to enter the garden , find Amirm and bash him about the head over and over again in some vain attempt to vanquish your displeasure , now if one had planted some boarders , helped weed or otherwise helped the garden be enjoyed by others you'd accept some critique, BUT just random strangers who wonder in ? Why's that acceptable? And to do so persistently?

Reading though this thread I find @Rusty Shackleford had been unnecessarily persistent and adversarial in his pursuit of Amirm , it's become nothing more that a argument to win and whatever the original point of contention that may of held some merit well that has been totally undermined. I'd like to think this is a example of ' the frailty of forums as communication mediums for complex subject matter' , things become adversarial and combative, rather than any personal false on his behalf.

Rather like subjective listening bias I think we are all vulnerable to the limits of this medium, yes @amirm included . There's so many things that underpin social interactions that's are missing on these forums , it would be amazing and rather unnatural if it didn't effect us .

Let's all keep that in mind before we go after eachother, create wild conspiracy theories or otherwise condemn and harass one another. Let's try and respect the garden and the guy who made it as well as those that help maintain it for the benefit of everyone.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
16,054
Likes
36,441
Location
The Neitherlands
i think of ASR as amirm's garden , he's opened it up to the public, the locals enjoy it , some plant and help weed , some help with the buying of gardening tools , there's some open discussion on planting but the landscaping is mostly set . Some come and don't like it , maybe they mention why , maybe they leave and find another or make their own .

This.
 

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,559
Likes
1,704
Location
California
I honestly have no idea what this discussion thread is about. It reads like a series of loosely connected debates around a set of very vague assertions. Perhaps some of these topics can be separated out into separate threads. But that's unlikely to happen because the next post in this thread will be a reply to one of the aforementioned loosely connected debates and that will continue for another couple of posts before someone changes the debate topic again. And so on...
 
Last edited:
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
I honestly have no idea what this discussion thread is about. It reads like a series of loosely connected debates around a set of very vague assertions. Perhaps some of these topics can be separated out into separate threads. But that's unlikely to happen because the next post in this thread will be a reply to one of adorementioned loosely connected debates and that will continue for another couple of posts before someone changes the debate topic again. And so on...

Since you have an expertise it would be appreciate that you anwser to what I quoted you, or the fictiv situation exemples. To make it simple how robust it is, and what can be done to improve it.
 
OP
P

patate91

Active Member
Joined
Apr 14, 2019
Messages
253
Likes
137
What CMS level sommelier are you? If you are a CMS II or above, I might be interested. However, if a CMS IV (Master Sommelier) comes along and says you are full of it, you better believe I'm going to put more weight on their evaluation.
And yes I am aware there was a huge scandal about recent Master Sommelier tests.

Can you share some links about the scandal. I've read a book about food and philosophers. The wine came as an exemple.
 
Top Bottom