The frailty of Sighted Listening Tests

[LDKTA]

FWIW, the AES guidelines for speaker evaluation best practices are that listening should be blind and that trained listeners are preferred. The mark of a trained listener is their reliability, not validity. AES also mentions hearing impairment being a factor, which clearly opens up a whole other can of wormsView attachment 77111View attachment 77112
The contention in this thread is about the validity of trained listeners’ sighted impressions, a point that seems likely to remain debated.

Did you know that reliability is a synonym for validity? It is.

I am well aware of all of this. No one in this thread has discounted blind listening.

 

[Rusty Shackleford]

FWIW SVS is the opposite of small. They're multinational and have dozens of retail partners in the US alone (both big box B&M and online). They're likely among the largest brands (in both units and revenue) Amirm has measured to date. They're not Klipsch, but they're much closer to them than they are to someone like Salk or Ascend by far.

I honestly didn’t know this. Very interesting. Now I’m curious to read more about their market share, if they’re publicly traded, etc.

 

[Rusty Shackleford]

Did you know that reliability is a synonym for validity? It is.

In this context, they aren’t.

 

[Deleted member 17820]

Let me give you guys a scenario.

A perfect, double blind listening test is set up.
Every perfect and trained listener chooses speaker A.

They say speaker A is the best sounding speaker they have ever heard.

The blind is pulled back and speaker A, although the most extremely superlative sounding speaker ever made is also the most ugly by an order of magnitude.

No one can stand to look at it
It's so ugly it makes some people feel ill
It brings up bad memories of people's childhood.

What is so "VALID" about blind testing in this case? In this case the blind testing can only tell you the speaker is ENJOYABLE when not seen. So the "perfect" blind test is only beneficial to the person who will use the speakers where they can't be seen. BUT even then the knowledge of how ugly they are STILL might outweigh the PERFECT sound. Therefore these speakers can only be enjoyed when installed and never seen even once.

 

[Racheski]

I have answered this already: https://www.audiosciencereview.com/...hted-listening-tests.15114/page-6#post-476281

Take the above to a natural extreme and you have the answer. A trained listener is able to outperform the general public by a wide margin and with it, allow developers to produce products that please even the best listeners. They are critical in finding design issues which can be caught and resolved. Otherwise, you would be at mercy of luck to find listeners from general population to have same level of discrimination.

Extreme objectivists hate trained listeners because they can invalidate their generalizations about what can be heard and not. So I get your angst but that doesn't change the reality of the important role trained listeners play for companies and research organizations that use them such as Harman, Dolby, Fraunhofer Institute, Microsoft, etc.

Trained listeners need to be our friends, not enemies. They let us find the truth in audio using our ears so that we don't get embarrassed one day when a subjectivist can hear something we can't. We use the best instrumentation for measurements. Why not for listening? Why the put down as you wrote?

I don't think @krabapple was doubting that you are a trained listener nor the value that trained listeners bring to speaker design and evaluation. We understand that you have decades of experience, have worked with trained listeners extensively during your career, and have taken the Harman test multiple times. I'm not doubting your status as a trained listener. What is not clear, at least to me, is exactly when someone can be considered a trained listener because I am not familiar with any organizations that codify what it means to be a trained listener. I know when someone can be considered a medical doctor (they graduate from medical school), or when someone becomes a master sommelier (they pass the Master Sommelier test). But obviously I should not be considered a trained listener like yourself simply by taking the online Harman course, just like if I take a CPR certification class I should not be considered a paramedic.

Is there a more official certification for becoming a trained listener that we should be aware of?

 

[amirm]

FWIW, the AES guidelines for speaker evaluation best practices are that listening should be blind and that trained listeners are preferred. The mark of a trained listener is their reliability, not validity. The contention in this thread is about the validity of trained listeners’ sighted impressions, a point that seems likely to remain debated.

Clark, the author of the specification is confused about what the Bech paper is about as I noted here: https://www.audiosciencereview.com/...hted-listening-tests.15114/page-6#post-476281

Experience with a test as Bech investigated is not the same as being a trained, critical listener. Bech did not at all investigate what professional trained listeners are capable of so that reference is incorrect.

Clark popularized the ABX test so naturally puts a lot of emphasis on that. His cohort though, the late Arny Kruger argued with me a ton about blind test being necessary in speaker testing. I can't find his posts on that front but did on room acoustics:

As you see, he firmly believes in double blind tests not needed when differences are large which speakers clearly are.

 

[Deleted member 17820]

I don't think @krabapple was doubting that you are a trained listener nor the value that trained listeners bring to speaker design and evaluation. We understand that you have decades of experience, have worked with trained listeners extensively during your career, and have taken the Harman test multiple times. I'm not doubting your status as a trained listener. What is not clear, at least to me, is exactly when someone can be considered a trained listener because I am not familiar with any organizations that codify what it means to be a trained listener. I know when someone can be considered a medical doctor (they graduate from medical school), or when someone becomes a master sommelier (they pass the Master Sommelier test). But obviously I should not be considered a trained listener like yourself simply by taking the online Harman course, just like if I take a CPR certification class I should not be considered a paramedic.

Is there a more official certification for becoming a trained listener that we should be aware of?

IMO a trained listener is someone who can hear the good and the bad, and it is almost always the bad things that can easily be measured. Sound engineers have to be "trained listeners" because their whole job is to fix problems with the sound and to shape it, often times QUICKLY durring set up and sound check. A trained listener can see if problems exist in a system, consistently.

A buddy of mine brought over some bookshelf speakers for me to burn in and listen to. We put pink noise into each one, alternating left and right, and we both heard a big issue with one speaker. We then played pink noise and moved our ears up and down from the tweeter to the mid. We heard a suck-out of sound where the woofer crossed over the the tweeter in one speaker. We then got out the mics and measured everthing. It turns out one tweeter did have a 5db dip right at the crossover frequency. We both heard this at a distance with pink noise as a tonal difference. Up close we both heard it as a driver integration problem. Testing with a mic showed it was a slight? variation with one tweeter.

Just as AMIRM talks about "listening in mono is best for eval", my buddy and I found the issue listening in mono, when you listen to these speakers in stereo the brain kind of smooths the differences and issue!

 

[amirm]

What is not clear, at least to me, is exactly when someone can be considered a trained listener because I am not familiar with any organizations that codify what it means to be a trained listener.

If I were to guess, I would say the number of trained, critical listeners in the world is in the order of hundreds if that. Most companies only have one or two people that achieve this status. So this is not a broad category of job to have formal training or certifications.

Usually internal employees are groomed/trained on the task. At some point it becomes obvious that they are way above general class of testers and become the key gate keeper on fidelity questions of audio technology.

I think the word "training" is confusing and hence the reason I like to call it critical listeners. Training can help everyone become a better listener but to become a critical listener where you find things that others cannot, is unique and is not just a matter of reading a book and passing a test.

 

[Coach_Kaarlo]

FWIW, the AES guidelines for speaker evaluation best practices are that listening should be blind and that trained listeners are preferred. The mark of a trained listener is their reliability, not validity. AES also mentions hearing impairment being a factor, which clearly opens up a whole other can of wormsView attachment 77111View attachment 77112
The contention in this thread is about the validity of trained listeners’ sighted impressions, a point that seems likely to remain debated.

Hearing ability is critical, and a measurable basis for claiming TRAINED LISTENER ABILITY, as mentioned a while back by a few of us.....

 

[Rusty Shackleford]

Clark, the author of the specification is confused about what the Bech paper is about as I noted here: https://www.audiosciencereview.com/...hted-listening-tests.15114/page-6#post-476281

Experience with a test as Bech investigated is not the same as being a trained, critical listener. Bech did not at all investigate what professional trained listeners are capable of so that reference is incorrect.

Clark popularized the ABX test so naturally puts a lot of emphasis on that. His cohort though, the late Arny Kruger argued with me a ton about blind test being necessary in speaker testing. I can't find his posts on that front but did on room acoustics:

View attachment 77113

As you see, he firmly believes in double blind tests not needed when differences are large which speakers clearly are.

Just to be clear, you’re arguing that the official AES guidelines are wrong? Could you please post the entire Bech paper as a PDF?

However, let’s say that trained listeners are not just more reliable, but also more valid, as in able to accurately identify an objective stimulus. They can, for example, tell which frequency has been cut in a song more more accurately than an untrained listener. How to Listen, SoundGym, etc. all have “games” designed to train one in this. So I can absolutely believe that trained listeners are more valid in that sense.

I think the question we’re all grappling with, though, is how do we determine validity when there is no clear objective benchmark?

In the case of the similarly measuring speakers, the subjective listening is used to differentiate when the objective standard is indeterminate. Things get trickier if a trained listener says a poor measuring speaker sounds great or if two similarly measuring DACs or amps sound different. Etc.

 

[Vuki]

Let me give you guys a scenario.

A perfect, double blind listening test is set up.
Every perfect and trained listener chooses speaker A.

They say speaker A is the best sounding speaker they have ever heard.

The blind is pulled back and speaker A, although the most extremely superlative sounding speaker ever made is also the most ugly by an order of magnitude.

No one can stand to look at it
It's so ugly it makes some people feel ill
It brings up bad memories of people's childhood.

What is so "VALID" about blind testing in this case? In this case the blind testing can only tell you the speaker is ENJOYABLE when not seen. So the "perfect" blind test is only beneficial to the person who will use the speakers where they can't be seen. BUT even then the knowledge of how ugly they are STILL might outweigh the PERFECT sound. Therefore these speakers can only be enjoyed when installed and never seen even once.

Valid would be the result of the listening test, invalid would be impression of that loudspeaker esthetic value.
Sighted test listening impressions are objectively valid only for the listener - trained or not.

 

[Deleted member 17820]

Valid would be the result of the listening test, invalid would be impression of that loudspeaker esthetic value.
Sighted test listening impressions are objectively valid only for the listener - trained or not.

A valid result of a listening test STILL needs a claim, for example

1. This speaker has boosted treble
2. This speaker has smooth off axis response
3. This speaker breaks apart at 100DBa spl

However claims such as
1. This speaker has good rythm and pace
2. This speaker has good imaging and sounstage

THOSE claims we hear all the time are things I have not yet found a measurment for? Is there a way to measure if speakers have "pinpoint accuracy" or "Deep" soundstage? Aren't these things a very complicated phenomena??

 

[Rusty Shackleford]

Critical listeners have that ability as I have shown in repeated double blind tests that folks like you can't even dream of passing letting alone doing so.

Once again, responses like this are condescending and rude. You have no idea what skills, experience, or credentials “folks like me” or others on this forum do or don’t have.

 

[LDKTA]

In this context, they aren’t.

In this context, they are. Why? Because sighted listening of loudspeakers is valid. You're ingesting the information and not even fully understanding the implications of said information. Although, in many cases, the validity of sighted listening tests have been explicitly stated. Even Sean Olive has explicitly stated that sighted listening is valid. Toole will tell you the same. Find me a single engineer or researcher that claims sighted listening is invalid. You will not find one. It's been acknowledged multiple times in this thread that blind testing is the golden standard and the subjective part of Amir's reviews (at the very end) are just that. Subjective. Even when his opinion clashes with the measurements, it only matters so much because that is how opinions work.

As stated above, at some point, it is absolutely imperative that you just listen. Measurements may serve as a check and balance, however, they will not always tell you whether you will like a loudspeaker or not.

 

[Rusty Shackleford]

In this context, they are. Why? Because sighted listening of loudspeakers is valid. You're ingesting the information and not even fully understanding the implications of said information. Although, in many cases, the validity of sighted listening tests have been explicitly stated. Even Sean Olive has explicitly stated that sighted listening is valid. Toole will tell you the same. Find me a single engineer or researcher that claims sighted listening is invalid. You will not find one. It's been acknowledged multiple times in this thread that blind testing is the golden standard and the subjective part of Amir's reviews (at the very end) are just that. Subjective. Even when his opinion clashes with the measurements, it only matters so much because that is how opinions work.

As stated above, at some point, it is absolutely imperative that you just listen. Measurements may serve as a check and balance, however, they will not always tell you whether you will like a loudspeaker or not.

You’re saying they’re “valid” in that they tell you if you, personally, like how they sound. Yes, of course that’s true. It’s tautological. That’s not what I thought we were discussing.

 

[whazzup]

It really boils down to an issue of trust and lack of knowledge in the inner workings. Here the only trained / critical listener is Amir.

40 other (working) critical listeners to talk to (if you can find them ):
https://www.hourdetroit.com/community/harman-international-car-audio-tech-test-listeners/

Audio bloggers / vloggers should take the opportunity to get them to talk more about their work....

And @Rusty Shackleford's comment about Audiophiliac gaining critical listener credentials, if someone chooses to be a bad actor and abuses his cred, that's his choice....

 

[amirm]

Could you please post the entire Bech paper as a PDF?

No. It is against AES rules to do that. Pay the money and get a copy. It is the main peer reviewed study that is quoted by both Clark and Olive.

 

[solderdude]

Let me give you guys a scenario.

A perfect, double blind listening test is set up.
Every perfect and trained listener chooses speaker A.

They say speaker A is the best sounding speaker they have ever heard.

The blind is pulled back and speaker A, although the most extremely superlative sounding speaker ever made is also the most ugly by an order of magnitude.

No one can stand to look at it
It's so ugly it makes some people feel ill
It brings up bad memories of people's childhood.

What is so "VALID" about blind testing in this case? In this case the blind testing can only tell you the speaker is ENJOYABLE when not seen. So the "perfect" blind test is only beneficial to the person who will use the speakers where they can't be seen. BUT even then the knowledge of how ugly they are STILL might outweigh the PERFECT sound. Therefore these speakers can only be enjoyed when installed and never seen even once.

When they were trained listeners and truly found the speakers were 'the best sounding one they ever heard' I would say the blind test was very valid. Not merely enoyable when not seen but close to perfection when not seen. That is the strength of blind tests done with trained listeners.
Regardless of the other speaker(s) it was up against.
The maker won't sell much of them but if the price wasn't absurd I bet there would be takers that use them and simply hang a curtain in front of them.

 

[amirm]

Just to be clear, you’re arguing that the official AES guidelines are wrong?

No, I am saying it is low quality specification in the part you quoted to reference a study about one thing, and confuse it with another. That confusion has been throughout this thread as well and hence me commenting on it. Experience in testing is NOT the same as being professionally trained.

 

[Rusty Shackleford]

It really boils down to an issue of trust and lack of knowledge in the inner workings. Here the only trained / critical listener is Amir.

40 other (working) critical listeners to talk to (if you can find them ):
https://www.hourdetroit.com/community/harman-international-car-audio-tech-test-listeners/

Audio bloggers / vloggers should take the opportunity to get them to talk more about their work....

And @Rusty Shackleford's comment about Audiophiliac gaining critical listener credentials, if someone chooses to be a bad actor and abuses his cred, that's his choice....

I’m actually well aware of what’s in that Hour Detroit article. I know people involved.

However, I think the notion that there are only a few hundred such people is too limiting. These are generally people already working in the company, who then train and pass certain tests. If there were only a few hundred people in the world who could detect what they can, what would be the point? It’s that they’re consistent in detecting things and speaking about them in clear ways, not that no one else in the world could detect them. After all, besides people with above-average hearing, critical consumers, etc., there are people in all aspects of professional audio who’ve undergone extensive training and make a living based on their ability to hear fine differences. Does anyone think a world-class mastering engineer could not be one of the Harman/Ford listeners? Of course not.