The frailty of Sighted Listening Tests

[Deleted member 17820]

wow, what mess of a post did I step into?
"trained listener"? ask any one of my Ex's and they will all tell you nope.

 

[Deleted member 17820]

LOL, I am just thinking that people measure their speakers response 20 feet up a ladder when they listen in a room sitting 1 foot high on a couch.

"effects of visual biases" You mean like when someone buys an exotic car or muscle car because it LOOKS cool and that makes them enjoy it more? So is ENJOYMENT an important part of HiFi? Then why would you think it's scientific to remove a stimuli that leads to enjoyment?>

If thats the case then maybe it would be most scientific to make the listeners sit on a hard floor while they are hungy and thirsty.

 

[krabapple]

Since speakers are enjoyed by also seeing them, I don't see why people would want to hide them while listening. It's well known that visual stimuli change neurotransmitters and cognitive function. Think of a man seeing a good looking girl, everything changes. So visual stimulation affects brain function that also affects how were here and listen.

Changes, as in makes audio discrimination more accurate rather than adding cognitive noise? Prove that please.

 

[krabapple]

If someone produced a speaker that looked just like a good looking girl, I might buy them regardless how they sound...

Jesus. What is this, the Mad Men forum?

 

[krabapple]

This opens up a whole nother can of worms. Blind testing is the best way to determine the quality of sound entering one's ears, but may be flawed for determining the quality of sound one's brain interprets in sighted listening.

Oh good lord.

Look, Toole and Olive actually *studied* the effects of sightedness on audio preference for loudspeakers. Decades ago.

 

[krabapple]

Yes, that's an issue I kept bringing up in other threads.

I work by manipulating the impression of sounds and picture all day long (sound design for film and tv). We very often use substitute sounds rather than real sounds (or the sounds recorded by the on-set microphone), and the combination of what you are seeing and what you are hearing changes your perception. If I play you the sound and let you know it is celery being twisted and snapping, that's what you will hear. If I simply play it with the accompanying visual of a character having his finger twisted and snapped in two, you'll perceive it as being a bone twisting and breaking.
I can take the sound of a vacuum cleaner and that's what you'll hear. Add the visual of a hovercraft or spaceship, and then you'll perceive it as the engine of that craft. Ben Berrt of Star Wars sound design fame took an elephant roar and when placed with the visual stimuli, everyone perceived it as the sound of a Tie Figher's engine.

Blind testing is great for weeding out variables, but if those variables snap back together, are almost always found comingled in normal situations, then that too is worthy of consideration.

Feel free to contribute to the science of psychoacoustics and perception, with your novel findings. Be prepared for some interesting peer review, especially if you demonstrate you aren't familiar with past work in this area.

 

[patate91]

I think an important point is not discusssed : "Double sighted" listening.

What's the robustness of an evaluation if I get all technical data before I do my sighted evaluation.

For exemple I'll take the one about wine I got earlier. I'm an expert I taste more than 200 wines this year (That's totally false, you'll see as per my descriptions) Someone ask me to evaluate a wine and he tells me : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Let say there's two outcome.

First : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Second : It taste like a Californian wine. Oak, Orange and herbs. Medium robe. Dry ending.

Again how robust will you rate my review?
*EDIT* And what can I do to improved my review?

 

[Rusty Shackleford]

I think an important point is not discusssed : "Double sighted" listening.

What's the robustness of an evaluation if I get all technical data before I do my sighted evaluation.

For exemple I'll take the one about wine I got earlier. I'm an expert I taste more than 200 wines this year (That's totally false, you'll see as per my descriptions) Someone ask me to evaluate a wine and he tells me : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Let say there's two outcome.

First : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Second : It taste like a Californian wine. Oak, Orange and herbs. Medium robe. Dry ending.

Again how robust will you rate my review?
*EDIT* And what can I do to improved my review?

 

[patate91]

I'll add to my fictiv situation.

I decide to compare it with one of my prefered wine.

I pour a glass of each I taste them and rate them.

How robust is my evaluation?

 

[richard12511]

Oh good lord.

Look, Toole and Olive actually *studied* the effects of sightedness on audio preference for loudspeakers. Decades ago.

Yes, and they found that it has an effect, hence my point...

 

[MattHooper]

Feel free to contribute to the science of psychoacoustics and perception, with your novel findings. Be prepared for some interesting peer review, especially if you demonstrate you aren't familiar with past work in this area.

Why the snark?

Let's be charitable and presume the other party may not be so naive, and may be making a reasonable point.

First, which specific past work in this area are you referencing? Might it be the work of Toole, Olive etc? For instance the often cited Toole
Hearing is Believing vs Believing is Hearing paper? Sean Olive writes about such experiments here (and I've cited it before here):

http://seanolive.blogspot.com/2009/04/

So people were influenced by things like the cost of the speaker, the size, the apparent cheapness or not of materials etc. Not exactly unexpected, right?

Or is there research that you can cite that gets even more detailed as to what specific speaker VISUAL designs invoke precisely which subjective impressions or ratings in sighted tests?

In any case, I'm not sure at all what you are actually objecting to. Research shows that biases involved in sighted evaluations alter people's perception of the sound. What have I written that you think is incorrect, ignorant...or whatever brought about the tone in your reply?

Thanks.

 

[Racheski]

I think an important point is not discusssed : "Double sighted" listening.

What's the robustness of an evaluation if I get all technical data before I do my sighted evaluation.

For exemple I'll take the one about wine I got earlier. I'm an expert I taste more than 200 wines this year (That's totally false, you'll see as per my descriptions) Someone ask me to evaluate a wine and he tells me : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Let say there's two outcome.

First : This is a Bordeau. Oak, vanilla with an hint of wild berry. The robe is rich dark red. Sweet ending.

Second : It taste like a Californian wine. Oak, Orange and herbs. Medium robe. Dry ending.

Again how robust will you rate my review?
*EDIT* And what can I do to improved my review?

What CMS level sommelier are you? If you are a CMS II or above, I might be interested. However, if a CMS IV (Master Sommelier) comes along and says you are full of it, you better believe I'm going to put more weight on their evaluation.
And yes I am aware there was a huge scandal about recent Master Sommelier tests.

 

[MattHooper]

LOL, I am just thinking that people measure their speakers response 20 feet up a ladder when they listen in a room sitting 1 foot high on a couch.

"effects of visual biases" You mean like when someone buys an exotic car or muscle car because it LOOKS cool and that makes them enjoy it more? So is ENJOYMENT an important part of HiFi? Then why would you think it's scientific to remove a stimuli that leads to enjoyment?>

If thats the case then maybe it would be most scientific to make the listeners sit on a hard floor while they are hungy and thirsty.

This post seems to show some confusion regarding science.

Generally, to narrow down cause and effect, in science you try to remove or control for variables. It's been established through careful testing that what we see and/or what we think we know about a product influences our subjective impression. So If someone likes speaker A over speaker B
the preference may be due to how the speakers sound, or how they look (<-- confounding variable) or some mix of both. To get to a more accurate (and therefore, often, more predictive) explanation, you control for those variables. So in a blind test you take away the sighted bias variable.

If someone still prefers speaker A over B you can have more confidence in attributing the preference to the sound, not to what the speakers look like. If the preference reverses so that the person in the blind conditions chooses speaker B as more preferable, you've learned that it is more likely due to how the speakers look. Either result can suggest avenues of research to gain more understanding and predictability.

It just lets you know what's actually going on, so you know what you are doing when designing or buying a speaker.

Of course most people tend to buy speakers by hearing them in sighted conditions. But knowledge is power. If you have the goal of seeking "accuracy" in your playback (as many do here), then it's quite possible in sighted conditions a listener may perceive the more impressive, more expensive, newer speakers as being more "accurate." Where in fact it's measurably less accurate than the cheaper speaker. This is valuable information for the person seeking accuracy. First, they learn that their sighted listening isn't as reliable as they thought. Second, they learn that they can get the accuracy they seek from the cheaper speaker.

No one has to avail themselves of what is learned in carefully controlled research. One can always just go on whatever creates the more pleasing subjective impression and spend whatever money it took to get it. But it's great to have for anyone who DOES want to use the information to guide their audio designs or purchases.

 

[bobbooo]

Enjoying the read, unusual to find so many informed and similarly aligned views on a forum - one big happy ASR family.

Seriously though, so easy to criticise, so hard to step up and make the difference you believe in. I am grateful for the time and effort some make to help others on here, and equally for @amirm who is trying to effect change in an industry which still worships the sacrificing of virgins (or similar belief orientated dogma).

Apologies if someone has already asked this question - I am half way through the reading of this thread.


But I wonder if anyone had thought critically about perhaps the clearest and most common source of difficulties in listening for differences between speakers - the testers hearing loss!! Fairly linear with age, and certain activities make it worse, and not to mention the male bias towards significant hearing loss generally.

Subjective interpretations of audio from the limited range of hearing an individual has, does not seem insignificant, easy to have tested to confirm tho I guess.

How well does @amirm hear these days?(joking)

Yep, presbycusis (age-related hearing loss) is simply a fact of biology, for which I will quote Dr. Floyd Toole's very honest and humble admission of its effect on his own sound quality judgements with age:

Hearing loss occurs as a result of age and accumulated abuse over the years. Whatever the underlying causes, Figure 3.7 shows that in terms of our ability to make reliable judgments of sound quality, we do not age gracefully. A couple of the data points indicate that young persons are not immune to hearing loss. It certainly is not that we don’t have opinions or the ability to articulate them in great detail; it is that the opinions themselves are less consistent and possibly not of much use to anyone but ourselves. In my younger years, I was an excellent listener—one of the best, in fact. However, listening tests as they are done now track not only the performance of loudspeakers but of listeners—the metric shown in Figure 3.9. At about age 60, it was clear that it was time for me to retire from the active listening panel. Variability had climbed and, frankly, I found it to be a noticeably more difficult task. It is a younger person’s pursuit. Music is still a great pleasure, but my opinions are now my own. When graybeards expound on the relative merits of audio products, they may or may not be relevant. But be polite—the egos are still intact.

In Figure 3.8, it is seen that fidelity ratings for loudspeakers D and U are closely grouped by both sets of listeners, but the ratings in the shaded areas are simply lower. Things change for loudspeakers V and X, where the close groupings of the low-variability listeners contrast with the widely dispersed ratings by the high-variability listeners. This is a case where an averaged rating does not reveal what is happening. Listener ratings simply dispersed to cover the available range of values; some listeners thought it was not very good (fidelity ratings below 6), whereas others thought it was among the best loudspeakers they had ever heard (fidelity ratings above 8). Listeners simply exhibited strongly individualistic opinions. Hearing loss is very likely involved. Be careful whose opinions you trust.

Note, from figure 3.7, above around age 50 the mean variability of speaker rating judgements for the majority goes above 0.5, which translates to the increase in rating range given for speakers V and X in figure 3.8.

 

[MattHooper]

Often I wish that some of you would devote the majority of your energy to issues like global warming, racial injustice, or sexual assault. Good night.

What a strange (and sanctimonious) comment.

How do you know people here aren't devoting energy to any of those causes?

Are you devoting the majority of your energy to those issues?

You have 69 posts. Wouldn't that time have been better spent fighting global warming? (And not contributing to it by using your computer to post on an audio forum?). Have you made some cut-off point calculation as to how many posts are ok before someone isn't doing enough for those causes?

Yeesh.

 

[youngho]

What a strange (and sanctimonious) comment.

https://dilbert.com/strip/2018-05-30

Have at it, I'm unsubscribing here.

MattHooper, I will not respond further. I have already contributed to this thread earlier. Goodbye.

 

[NDC]

How on earth is it “intellectually dishonest” or “below the belt” to quote someone’s public posts? Moreover, what does 2009 have to do with anything? This forum is full of citations, including from Amir, that are far older than 11 years!

You really seem obsessed with snark tactics and rhetoric over charitable interpretation and good faith debate.

You’re not “citing” a position with an 11 year old forum post - you’re finding old comments (not scientific statements of fact...) that may or may not be maintained without any right of reply.

It just seems poor form to me. You obviously don’t agree - but then again, you’re one of the few on this forum who seems incapable of respectful, calm discussion.

 

[Blumlein 88]

Yep, presbycusis (age-related hearing loss) is simply a fact of biology, for which I will quote Floyd Toole's very honest and humble admission of its effect on his own sound quality judgements with age:


View attachment 76977
View attachment 76978


Note, from figure 3.7, above around age 50 the mean variability of speaker rating judgements for the majority goes above 0.5, which translates to the increase in rating range given for speakers V and X in figure 3.8.

This, this is the real Frailty of sighted listening or blind listening tests. Age, variable hearing acuity among groups of people, and a not insignificant number of younger people who have accelerated hearing loss due to using headphones which end up being played at louder volumes than loudspeakers on average apparently.

Alas I'm old enough I need to follow Toole and keep my opinions to myself mostly.

 

[MattHooper]

https://dilbert.com/strip/2018-05-30

Have at it, I'm unsubscribing here.

Sure.

I don't think you are a troll.

But are you aware that you just exhibited trolling behavior?

That is: drop in with a brief inflammatory comment, highly judgemental about the efforts and character of others here. Then when your comment is questioned or critiqued, instead of engaging with someone's response, ignore the points made and simply respond with a link to an insult implying the person is merely an idiot? Then leave.

Is that really the standard of interaction you want to see around here? If not...why contribute that type of stuff?

Cheers.

 

[Coach_Kaarlo]

Yep, presbycusis (age-related hearing loss) is simply a fact of biology, for which I will quote Floyd Toole's very honest and humble admission of its effect on his own sound quality judgements with age:


View attachment 76977
View attachment 76978


Note, from figure 3.7, above around age 50 the mean variability of speaker rating judgements for the majority goes above 0.5, which translates to the increase in rating range given for speakers V and X.

Thank you for making my point in a far more insightful and factual way.

I think it highlights the inherent flaws in biased discussions around individual and group biases - as opposed to THE scientific method. One is only aware of that which one can be aware of, meaning Olive & Toole wrote about many more sources of error than just blind testing, but most contributions to this discussion have focused on sighted v blind tests. If one has not had one's hearing checked recently there may be an elephant in the room.

IMHO the more one learns about neuroscience, biology, evolution, psychology, behavioural economics, etc etc - it becomes a source of sheer amazement that we manage to achieve half of that which we do (the human race). We are both incredible and deeply flawed animals - nothing more so than our ACTUAL hearing abilities.