The frailty of Sighted Listening Tests

[krabapple]

If your view is that this is Amir’s “garden,” so he is above being challenged on his claims, then this site really should be, as someone above noted, Amir’s Subjective Opinions, not Audio Science Review.

How about 'Amir's Garden of Objective and Subjective Audio Delights"?

OK, maybe not.

 

[krabapple]

It pains me to say this, but perhaps part of the problem is our beloved pink panther.

It is the first image we see when we log on to a new review, and we all understand the meaning of each panther being used i.e. headless vs. cookie jar vs. golfing panther.

I've never had a clue what the Pink Panthers are supposed to mean, much less beloved them.

 

[LDKTA]

For what feels like the millionth time, what followed were assertions that, as a trained listener, he could come to valid sighted, subjective opinions that others can’t.

This thread started with the OP asking if that is true, given that Olive said that speaker evaluation “must be done blind.” Since then, Amir has said that Olive’s study wasn’t up to a high enough standard, and has challenged the “must be done blind” conclusion.

This new hypothesis, that trained listeners can come to valid sighted conclusions, necessarily requires evidence. It also prompts the question as to what training counts. (We seem to have come down on How to Listen, since that’s what Amir used.)

This is all important for consistency and for evaluating claims. What’s the proper way to respond if someone says “I’ve excelled in my How to Listen training, and ___ sounds better than ___”? Is that now a claim that should not be met with the “your listening wasn’t blind” rejoinder? These are honest, important questions.

I am fully aware of the limitations of what I volunteer them all the time.

The position taken is that my subjective tests should not even be reported because by definition they are wrong. This is the absurd position we are arguing. Not that my subjective results are guaranteed to be right. If I thought this, I would not measure!

So take a seat and listen and learn the topic before jumping in with anger and emotion with zero contribution to the topic. Heaven knows we already have too many like you spoiling this dish.

Above this quote was an image with commentary from Sean Olive. Do you not feel as if you're beating a dead horse?

If the study in question was posted in a convention paper, it certainly wasn't "up to a high enough standard." He seems to be well aware of the "must be done blind" conclusion. You are grasping at straws, though I do feel that you've asked *some* reasonable questions throughout this thread.

My response to someone making such a claim would be, "the Harman How-to-Listen training is not only out of date but only allows one to hear certain artifacts that may or may not be present in X, Y, or Z." TO ME, meeting that claim with "your listening wasn't blind" is a rejoinder only worthy of electronics such as two DACs that are measurably transparent. Loudspeaker comparisons are far more complex than you may think and while blind testing significantly reduces biases, we do NOT listen to loudspeakers blinded on a day to day basis. If that option was readily available to everyday consumers seeking to truly known where their preference stands, don't you think like minded members in this group would take that option? Through decades of real world experience, Amir has proven beyond a reasonable doubt that his listening ability is above average. Trained listeners don't exist for fun, these are individuals with the capability to discern even the most minute artifacts in program material or whatever else they are trained to listen for. Whether it is sighted or unsighted is wholly irrelevant in regards to his subjective listening evaluation because it is supposed to be subjective. Again, like the OP, it appears to me that you just want it YOUR way.

 

[Haint]

...This forum is focused on informing consumers, thereby impacting purchasing decisions. Companies viewed to have performed poorly have been mercilessly criticized and mocked. The defense of that was that such conclusions were based on measurements, not opinions. That may or may not have been a valid defense, but it was a clear one. However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. Despite having remarkably similar preference scores, one would come away from reading ASR thinking that the the M106 (5.79) is a good, recommended speaker and the SVS (5.70) is a poor, not recommended speaker. Now, that very well might be the case! Maybe the preference scoring model is invalid, despite its good lineage. Maybe listening, even sighted listening weeks apart, is superior to the preference score at evaluating speakers. I just think we should be clear about the implications of that. Do we think blind tests or, at least, level-matched listening with both speakers being present at the same time, playing the same music, is necessary to come to a critical comparison, or not? I do think that, if they’re going to trump the data, the subjective reviewers should be longer, level-matched, and done against a consistent “good” reference speaker. But that’s just my view.

With influence — either over the epistemological contours of acceptable reviewing or over readers’ purchasing decisions and, thereby, companies’ business — comes responsibility. That’s my point, as simply put as I can make it.

FWIW SVS is the opposite of small. They're multinational and have dozens of retail partners in the US alone (both big box B&M and online). They're likely among the largest brands (in both units and revenue) Amirm has measured to date. They're not Klipsch, but they're much closer to them than they are to someone like Salk or Ascend by far.

 

[krabapple]

It's about the frailty of sighted listening while gardening, debated under the theoretical framework established by Olive that drinking wine during experiments with Harman employees will bias trained listeners. Clear as mud for me.

I think it can be summarized as:

sighted evaluations of audio gear are generally so fraught with bias as to be useless as data points...unless done by trained listeners.

Agreed-upon criteria that make one a 'trained listener' still to be determined...

 

[LDKTA]

What? The goal of providing information about gear is to inform consumer decisions, which necessarily impact businesses. This isn’t a value judgment. It’s a simple fact of how the marketplace works.

That is besides the point. I can inform you that "X" is objectively better than "Y" and if you choose to purchase "Y" based on your personal liking to "Y," am I supposed to make a big deal about it? Should I say that your personal liking is wrong? "The goal of providing information about gear IS to inform consumer decisions," but that doesn't mean an *informed consumer* must make the same decision that they've been informed to make. The data is here for us to interpret and it is also up to us to make purchasing decisions we find most applicable to our penchant (irrespective of one's opinion on that product).

 

[Rusty Shackleford]

Above this quote was an image with commentary from Sean Olive. Do you not feel as if you're beating a dead horse?

If the study in question was posted in a convention paper, it certainly wasn't "up to a high enough standard." He seems to be well aware of the "must be done blind" conclusion. You are grasping at straws, though I do feel that you've asked *some* reasonable questions throughout this thread.

My response to someone making such a claim would be, "the Harman How-to-Listen training is not only out of date but only allows one to hear certain artifacts that may or may not be present in X, Y, or Z." TO ME, meeting that claim with "your listening wasn't blind" is a rejoinder only worthy of electronics such as two DACs that are measurably transparent. Loudspeaker comparisons are far more complex than you may think and while blind testing significantly reduces biases, we do NOT listen to loudspeakers blinded on a day to day basis. If that option was readily available to everyday consumers seeking to truly known where their preference stands, don't you think like minded members in this group would take that option? Through decades of real world experience, Amir has proven beyond a reasonable doubt that his listening ability is above average. Trained listeners don't exist for fun, these are individuals with the capability to discern even the most minute artifacts in program material or whatever else they are trained to listen for. Whether it is sighted or unsighted is wholly irrelevant in regards to his subjective listening evaluation because it is supposed to be subjective. Again, like the OP, it appears to me that you just want it YOUR way.

I don’t know what this post is getting at. Most listening of all equipment isn’t done blind “on a day to day basis”! Does that invalidate blind testing?

In terms of training, Amir has said he self-trained and then used How to Listen. From what he’s said, his training was focused on compression artifacts. There are obviously many types of trained listeners, many people who make their living based on their hearing ability. The question is simply what training “counts“ and how we evaluate their claims if they clash with measurements and with each others’ claims. (There are world-renowned mastering engineers who praise speakers that would get the no-head Panther.)

 

[Rusty Shackleford]

That is besides the point. I can inform you that "X" is objectively better than "Y" and if you choose to purchase "Y" based on your personal liking to "Y," am I supposed to make a big deal about it? Should I say that your personal liking is wrong? "The goal of providing information about gear IS to inform consumer decisions," but that doesn't mean an *informed consumer* must make the same decision that they've been informed to make. The data is here for us to interpret and it is also up to us to make purchasing decisions we find most applicable to our penchant (irrespective of one's opinion on that product).

I’m sorry, but this strikes me as dodging responsibility. Clearly the goal of any review or measurement is to provide consumers with actionable information. Aren’t there countless threads here bashing certain reviewers and companies for peddling misinformation and snake oil? If consumers are all ultimately responsible for their decisions, regardless of what info they’ve absorbed, then there’s no need to bash those people, because they are not responsible for what consumers do.

 

[Deleted member 17820]

Here lies the problem. Many people say there are things we can't measure that affect what we here, including transparency.
1. A person can say we can measure everything that matters to hearing, but they can't prove it.
2. Another person can say there are things that matter, but we don't know what they are or how to measure them yet.
The basis of the claim rests on undiscovered phenomena and is therefore a matter of faith and not science, although scientific history DOES show evidence that what we believe to be a full extent of knowledge is actually just the tip of the iceberg.

There are measurable things we know to impact transparency but there are also possibly things we don't know, so to say a dac, amp, or any other piece of gear is transparent based on measurements alone is, IMO, a bold and presumptuous statement.

This isn't just my belief, but also that of an electrical engineer, audio designer, and genius friend of mine that has said numerous times he has measured gear that should sound great but is horrible while at the same time he has had gear that measures horrible and sounds great.
He has 50+ years experiences and is both an eletrical genius, retired from Seimens, and also an acoustician who builds studios.

To say that we need a pristine playback system to capture the detail of a dirty, grungy distorted guitar sound is logical suicide.

A dirty and distorted electric guitar sounds great therefore we need a low distortion system to enjoy it? LOGICAL SUICIDE.

As a sound engineer I can tell you that all music has distortion. Compression, limiting, microphone selection, EQ, just about EVERY aspect of music production changes the signal and shapes the sound. Without drastic change to "original sources" you would have garbage for music. Complementary EQ, dynamic processing, Reverb and FX are what make up most of what we know as music.

 

[amirm]

I think it can be summarized as:

sighted evaluations of audio gear are generally so fraught with bias as to be useless as data points...unless done by trained listeners.

That's absurd. No one has taken that position. I don't even do listening tests on bulk of electronics.

The issue is for speakers. Someone has to listen at some point. I can't do double blind research of them every day. So I use sighted, but informed listening, paying attention to problematic areas identified in objective measurements. I follow research recommendations for keeping listening room constant, mono listening, critical content, and listener training. And I have shown time and time again that who makes the speaker and what it looks like is no impact on me.

You all can buy a few of these speakers and do your double blind tests. If you are not going to bother, then don't bother me with it. And certainly not by completely distorting what we are saying by generalizing that way.

 

[LDKTA]

I’m sorry, but this strikes me as dodging responsibility. Clearly the goal of any review or measurement is to provide consumers with actionable information. Aren’t there countless threads here bashing certain reviewers and companies for peddling misinformation and snake oil? If consumers are all ultimately responsible for their decisions, regardless of what info they’ve absorbed, then there’s no need to bash those people, because they are not responsible for what consumers do.

Dodging responsibility? "Clearly the goal of any review or measurement is to provide consumers with actionable information." Sure, but this isn't Stereophile. You want it done your way because Amir's way isn't pleasing enough for you and I understand where you're coming from but so what? Is Amir "peddling misinformation and snake oil?" I've yet to see it... There is no need to bash those people. There are those that bash them and those who attempt to inform them. The companies and reviewers spewing misinformation/disinformation ARE responsible for their actions and they should be held accountable for misleading their consumers into purchasing their products. This is NOT what is going on here.

 

[Rusty Shackleford]

In an effort at a peace offering, I will say to @amirm that I’m very grateful for your speaker measurements.

 

[amirm]

Agreed-upon criteria that make one a 'trained listener' still to be determined...

I have answered this already: https://www.audiosciencereview.com/...hted-listening-tests.15114/page-6#post-476281

Take the above to a natural extreme and you have the answer. A trained listener is able to outperform the general public by a wide margin and with it, allow developers to produce products that please even the best listeners. They are critical in finding design issues which can be caught and resolved. Otherwise, you would be at mercy of luck to find listeners from general population to have same level of discrimination.

Extreme objectivists hate trained listeners because they can invalidate their generalizations about what can be heard and not. So I get your angst but that doesn't change the reality of the important role trained listeners play for companies and research organizations that use them such as Harman, Dolby, Fraunhofer Institute, Microsoft, etc.

Trained listeners need to be our friends, not enemies. They let us find the truth in audio using our ears so that we don't get embarrassed one day when a subjectivist can hear something we can't. We use the best instrumentation for measurements. Why not for listening? Why the put down as you wrote?

 

[Rusty Shackleford]

Dodging responsibility? "Clearly the goal of any review or measurement is to provide consumers with actionable information." Sure, but this isn't Stereophile. You want it done your way because Amir's way isn't pleasing enough for you and I understand where you're coming from but so what? Is Amir "peddling misinformation and snake oil?" I've yet to see it... There is no need to bash those people. There are those that bash them and those who attempt to inform them. The companies and reviewers spewing misinformation/disinformation ARE responsible for their actions and they should be held accountable for misleading their consumers into purchasing their products. This is NOT what is going on here.

This is question begging. Is it not true that many people you might consider to be “spewing misinformation/disinformation” based on their subjective listening might honestly think what they’re saying is true? The task is demonstrating through evidence the conditions under which something like sighted listening is valid.

 

[Deleted member 17820]

In an effort at a peace offering, I will say to @amirm that I’m very grateful for your speaker measurements.

I just found ASR while ago and have seen a war of sorts online, I have even been told on this forum to "go away" and accused of being a subjectavist. HAHAHA
I wish people could just see that everything is a data point. blind listening, sighted listening, mono, stereo, all just data points. More data points may help you or they may not.
I for one have found AMIRM's testing to be both helpful and true on much of the gear I have purchased, Regardless of the "perfection" of his testing methods.

 

[Deleted member 17820]

This is question begging. Is it not true that many people you might consider to be “spewing misinformation/disinformation” based on their subjective listening might honestly think what they’re saying is true? The task is demonstrating through evidence the conditions under which something like sighted listening is valid.

Hi Rusty, I wish ppl would get to the bottom of this.
For something to be "valid" it must be true. For something to be true there must be a claim!
What is the claim anyone here is making?

Also "bias" may be the wrong term. Bias denotes a preference but would untrained listeners have a preference for how a speaker looks? And there is no reason that the "bias" of untrained listeners would have any logic to it at all, is there??

 

[LDKTA]

This is question begging. Is it not true that many people you might consider to be “spewing misinformation/disinformation” based on their subjective listening might honestly think what they’re saying is true? The task is demonstrating through evidence the conditions under which something like sighted listening is valid.

It is obviously true that these people may honestly believer what they're saying is true.

The fact of the matter is sighted listening IS valid and it will always be valid no matter how much don't want it to be. You have to listen sighted. Loudspeakers are NOT DACs, amplifiers, preamplifiers, etc. It is actually frustrating to read your commentary at this point. Why would sighted listening not be valid for loudspeakers? Can they be misleading? Sure, but that does not make them invalid by any means.

 

[Rusty Shackleford]

FWIW, the AES guidelines for speaker evaluation best practices are that listening should be blind and that trained listeners are preferred. The mark of a trained listener is their reliability, not validity. AES also mentions hearing impairment being a factor, which clearly opens up a whole other can of worms

The contention in this thread is about the validity of trained listeners’ sighted impressions, a point that seems likely to remain debated.

 

[LDKTA]

I don’t know what this post is getting at. Most listening of all equipment isn’t done blind “on a day to day basis”! Does that invalidate blind testing?

In terms of training, Amir has said he self-trained and then used How to Listen. From what he’s said, his training was focused on compression artifacts. There are obviously many types of trained listeners, many people who make their living based on their hearing ability. The question is simply what training “counts“ and how we evaluate their claims if they clash with measurements and with each others’ claims. (There are world-renowned mastering engineers who praise speakers that would get the no-head Panther.)

No, it doesn't invalidate blind testing. Context.

I hear you...

 

[Deleted member 17820]

FWIW, the AES guidelines for speaker evaluation best practices are that listening should be blind and that trained listeners are preferred. The mark of a trained listener is their reliability, not validity. The contention in this thread is about the validity of trained listeners’ sighted impressions, a point that seems likely to remain debated.

View attachment 77111View attachment 77112

WHAT IS THE CLAIM TO VALIDITY?