• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

The frailty of Sighted Listening Tests

D

Deleted member 17820

Guest
When they were trained listeners and truly found the speakers were 'the best sounding one they ever heard' I would say the blind test was very valid. Not merely enoyable when not seen but close to perfection when not seen. That is the strength of blind tests done with trained listeners.
Regardless of the other speaker(s) it was up against.
The maker won't sell much of them but if the price wasn't absurd I bet there would be takers that use them and simply hang a curtain in front of them.

Curtain worthy? These vintage Polk Aduio(?) are for sale on CL sf if you are interested, HAHAHA.
 

Attachments

  • POLK AUDIO.jpg
    POLK AUDIO.jpg
    33.8 KB · Views: 91
Last edited by a moderator:

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
No. It is against AES rules to do that. Pay the money and get a copy. It is the main peer reviewed study that is quoted by both Clark and Olive.

I already have a copy, but I thought it would be helpful for all to see. I didn’t know your views on posting articles, which is why I asked.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
Let me give you guys a scenario.

A perfect, double blind listening test is set up.
Every perfect and trained listener chooses speaker A.

They say speaker A is the best sounding speaker they have ever heard.

The blind is pulled back and speaker A, although the most extremely superlative sounding speaker ever made is also the most ugly by an order of magnitude.

No one can stand to look at it
It's so ugly it makes some people feel ill
It brings up bad memories of people's childhood.

What is so "VALID" about blind testing in this case? In this case the blind testing can only tell you the speaker is ENJOYABLE when not seen. So the "perfect" blind test is only beneficial to the person who will use the speakers where they can't be seen. BUT even then the knowledge of how ugly they are STILL might outweigh the PERFECT sound. Therefore these speakers can only be enjoyed when installed and never seen even once.
This did make me laugh. You have just confirmed the whole reason to do blind testing. The purpose of the test is to assess sound quality not aesthetics.

Once you have an accurate impression of sound quality you can still make a decision as to whether it is aesthetically acceptable to you. If you do this sighted you will gain an inaccurate opinion of the sound quality due to that cognitive bias and are going precisely nowhere.
 
Last edited:

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
IMO a trained listener is someone who can hear the good and the bad, and it is almost always the bad things that can easily be measured. Sound engineers have to be "trained listeners" because their whole job is to fix problems with the sound and to shape it, often times QUICKLY durring set up and sound check. A trained listener can see if problems exist in a system, consistently.

A buddy of mine brought over some bookshelf speakers for me to burn in and listen to. We put pink noise into each one, alternating left and right, and we both heard a big issue with one speaker. We then played pink noise and moved our ears up and down from the tweeter to the mid. We heard a suck-out of sound where the woofer crossed over the the tweeter in one speaker. We then got out the mics and measured everthing. It turns out one tweeter did have a 5db dip right at the crossover frequency. We both heard this at a distance with pink noise as a tonal difference. Up close we both heard it as a driver integration problem. Testing with a mic showed it was a slight? variation with one tweeter.

Just as AMIRM talks about "listening in mono is best for eval", my buddy and I found the issue listening in mono, when you listen to these speakers in stereo the brain kind of smooths the differences and issue!

Good point, sound engineers are trained listeners but in a different context. So the training might be the same, but the experience gained after is different. Like an ophthalmologist vs a cardiologist.

And I got the same problem recently too! A pair of used speakers sounded odd, turns out one speaker's tweeter was barely making a sound and further checking revealed a bad crossover. In this example, I really didn't need to do any blind testing. Again, it's about the context.



I’m actually well aware of what’s in that Hour Detroit article. I know people involved.

However, I think the notion that there are only a few hundred such people is too limiting. These are generally people already working in the company, who then train and pass certain tests. If there were only a few hundred people in the world who could detect what they can, what would be the point? It’s that they’re consistent in detecting things and speaking about them in clear ways, not that no one else in the world could detect them. After all, besides people with above-average hearing, critical consumers, etc., there are people in all aspects of professional audio who’ve undergone extensive training and make a living based on their ability to hear fine differences. Does anyone think a world-class mastering engineer could not be one of the Harman/Ford listeners? Of course not.

Cool, get them to chip in too!

"Does anyone think a world-class mastering engineer could not be one of the Harman/Ford listeners?"
See above point between ophthalmologist vs a cardiologist. Of course the mastering engineer can switch jobs and eventually be a harman listener.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
I already have a copy, but I thought it would be helpful for all to see. I didn’t know your views on posting articles, which is why I asked.
Follow AES/fair use rules which allow excerpts with attribution but not full articles.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
Curtain worthy?

Depends on how it sounds, whether or not a nice looking grille can be put in front of it and where it is to be used.
Perhaps in a studio no one will be bothered. In most homes there probably won't be much spouses that would like to have this prominently in their living room though. Then again, not many speakers will make spouses happy soon. :D
 
D

Deleted member 17820

Guest
Depends on how it sounds, whether or not a nice looking grille can be put in front of it and where it is to be used.
Perhaps in a studio no one will be bothered. In most homes there probably won't be much spouses that would like to have this prominently in their living room though. Then again, not many speakers will make spouses happy soon. :D
I swear I am like a speaker magnet, the more I get rid of the more that show up, wish it was women....
 

aac

Active Member
Joined
May 17, 2020
Messages
217
Likes
163
The only problem I see with Amir's reviews is when it goes into drawing lines and text on graphs, I'd rather had none at all as sometimes they look weird when you compare different speakers.
 

RayDunzl

Grand Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
13,198
Likes
16,981
Location
Riverview FL

Racheski

Major Contributor
Forum Donor
Joined
Apr 20, 2020
Messages
1,116
Likes
1,699
Location
Chicago
If I were to guess, I would say the number of trained, critical listeners in the world is in the order of hundreds if that. Most companies only have one or two people that achieve this status. So this is not a broad category of job to have formal training or certifications.

Usually internal employees are groomed/trained on the task. At some point it becomes obvious that they are way above general class of testers and become the key gate keeper on fidelity questions of audio technology.

I think the word "training" is confusing and hence the reason I like to call it critical listeners. Training can help everyone become a better listener but to become a critical listener where you find things that others cannot, is unique and is not just a matter of reading a book and passing a test.
Yeah but “critical listener” is too much of an insider term - a layperson will assume it only means to listen carefully. I’m going to call it a TEC (Trained, Experienced, & Critical) Listener.
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
As I posted in the first post in this account earlier in this thread, I couldn’t log into my original account. I’d enter my password, and it would load indefinitely. I’m not sure why. I tried clearing my browser data, using a different browser, even a different device. I will try again to log into my original account at some point to see if the issue is resolved. But I never stated I was banned. Nor do I want a new account. I like my old one! Haha.
My mistake, sorry mate .
 

Killingbeans

Major Contributor
Joined
Oct 23, 2018
Messages
4,088
Likes
7,544
Location
Bjerringbro, Denmark.
Not to belabor this, but isn't the whole point of this debate that Amir was claiming that his subjective impression were correct because he's a "trained listener"? That's how we got into what counts as training, if trained listeners are actually more valid or simply more reliable, etc. If he were simply saying "I like this better, but those are just my personal tastes" I agree it wouldn't be an issue.

None belabor taken :)

I haven't read all of this thread, so I can't comment on a specific claim Amir has made. If he said that his training makes his personal taste more 'correct' than that of others, then of course it's BS. But I can't imagine he literally did that?

The whole point of the training is to make the estimate, of whether or not something measured is also audible, less rough around the edges. If that's what he claims to be doing, then I have no problem with it at all.

If you consider the scientific model, that Amir uses to set the bar for what should be audible in a quality product, as a matter of personal taste, then it opens a whole different can of worms.

I also don't think subjective reviews have no standards, insofar as that means "best practices." Certainly, things like level matching, having the equipment compared in possession at the same time, real-time switching, using consistent music, specifying what you are or aren't hearing in clear and precise language, etc. can make a subjective review more or less informative.

As far as I'm concerned, a subjective review means that all bets are off. There's no need to paint it in way that implies credibility. It's much like writing a book or a play. There's certain tricks that can maximize the suspension of disbelief, but it's still fiction. "Best practices" are great if you want sell something, or just like to have a mysterious aura of "professionalism" around you. I don't think Amir is aiming at being that specific kind of entertainer.
 

MattHooper

Master Contributor
Forum Donor
Joined
Jan 27, 2019
Messages
7,194
Likes
11,806
Please, please don't feel obligated to donate. It is not expected and certainly not in your situation. I hope the core message of what I wrote is not misunderstood.

Many of you contribute to the forum with greats posts which often have more value than financial contribution.

It is when the contributions are negative that I find the equation wrong.

Too late. I donated right after making my post. Though I did it via the single-donation paypal option as I don't have the spare funds to set up a monthly payment Patreon account at the moment.

But thank you for your message, I appreciate it.
 

aarons915

Addicted to Fun and Learning
Forum Donor
Joined
Oct 20, 2019
Messages
685
Likes
1,140
Location
Chicago, IL
I've kept up with this whole thread and there have been good points made on both sides but my conclusions based on everything are that:

The Harman training trains you to be a critical listener and makes you much more reliable and consistent than untrained listeners but both still come to the same conclusions regarding sound quality. Harman doesn't claim that the training affects bias in any way and all of the recent sound studies on headphones are still double blind, showing blind listening is still necessary to reduce bias.

It doesn't in any way mean that Amir is biased toward Harman but considering he owns Harman products, sells Harman products and is friendly with Harman engineers means he is probably more susceptible to Harman bias than any other brand. Of course it could just be that Harman products really do just sound great due to their research and superior engineering, all Revel speakers I've heard do sound good. We also have to remember that Amir's subjective impressions have largely been supported by the measurements, except with small speakers that are obviously being driven beyond their comfort zone.

With any subjective reviews online, if I consistently agree with someone's conclusions about a speaker I start putting more stock into their reviews but if someone doesn't like any of the speakers I have thought sounded good I know not to put as much stock into their opinions. I find most people are in 2 camps regarding subjective opinions on speakers, 1 type focuses on the midrange and higher frequencies while the other is all about bass and very high SPL, that may be simplifying things a bit but we know which category our host falls into. Either way, the measurements are the real gift of the website, if you don't feel the same about his subjective impressions, the measurements are the best we're going to get.
 

Putter

Senior Member
Forum Donor
Joined
Sep 23, 2019
Messages
492
Likes
771
Location
Albany, NY USA
I spent 40 years carefully training myself to not listen, ask anyone of my friends and family, they can vouch for my credentials.

Sometimes I think that may be the audiophile curse which is that you KNOW that your equipment isn't as good as the reviewers/friends/dealers/etc. and use your cognitive biases to find problems that aren't there, like power cords, bad electrical supply, cheap interconnects and speaker cables or theoretical problems like the amp uses too much feedback or 'stair-step' digital conversion.

I realize that in most cases these 'problems' don't exist or are easily fixed like better gauge speaker cables or better shielded interconnects, but it does tend to lead one down the rabbit hole of upgrading and expecting an improvement so one hears it until you read a review that says this cable/amp/power conditioner/..... is better and gives a blacker background to the music. I've rambled a bit OT, but the natural skeptic/scientist in me always doubted the truth of the golden eared audiophile, but not enough to ignore them.
 

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,554
Likes
1,701
Location
California
Let me give you guys a scenario.

A perfect, double blind listening test is set up.
Every perfect and trained listener chooses speaker A.

They say speaker A is the best sounding speaker they have ever heard.

The blind is pulled back and speaker A, although the most extremely superlative sounding speaker ever made is also the most ugly by an order of magnitude.

No one can stand to look at it
It's so ugly it makes some people feel ill
It brings up bad memories of people's childhood.

What is so "VALID" about blind testing in this case? In this case the blind testing can only tell you the speaker is ENJOYABLE when not seen. So the "perfect" blind test is only beneficial to the person who will use the speakers where they can't be seen. BUT even then the knowledge of how ugly they are STILL might outweigh the PERFECT sound. Therefore these speakers can only be enjoyed when installed and never seen even once.

With this hypothetical scenario, is your point that sighted tests are superior to blinded tests in determining the overall enjoyment from a speaker? If so, I would agree that the answer is yes, in some situations. For instance, if a loudspeaker is so ugly that your spouse makes you cover it up with a bedsheet everytime you listen to it, then yeah, that's going to affect your enjoyment. However, the question we're trying to answer with blinded tests is not overall enjoyment. The question is how preferred is the loudspeaker based JUST on its sound quality and nothing else (to the extent possible). Two different questions, two different methods to answer them.
 
Top Bottom