• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

The frailty of Sighted Listening Tests

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
Where does one go to apply and be trained and certified as a listener? I think what people are questioning is this certification program and its validity.

Well, except the unannouced trolls / people with vested interests, that is.

The How to Listen software was developed by Harman to train its listeners. It’s freely available online. Similar training is available from other sites and apps, too.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
I invite everyone to compare my posts to yours. Nothing I have said is “bullying.” You, on the other hand, have referred to other reviewers as “Joe Blow” and flatly stated “I know more than you,” even though you don’t know the identities of everyone on this forum, their backgrounds, or (apropos of the debate here) their hearing training.

As with the MQA thread, when you’re challenged with contradictory research, you become abusive and simply assert your superiority, rather than demonstrate it with evidence or clear reasoning.
Oh, I return fire with fire with obnoxious people who just want to fight. Laying low encourages them to become more of a bully. Look at how little self-awareness you have in making personal remarks about me while saying you have done no such thing.
 

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
Oh, I return fire with fire with obnoxious people who just want to fight. Laying low encourages them to become more of a bully. Look at how little self-awareness you have in making personal remarks about me while saying you have done no such thing.

The comments of yours I quoted were not in response to me, but please tell me where I made “personal remarks” about you.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
The comments of yours I quoted were not in response to me, but please tell me where I made “personal remarks” about you.
Let's see. You entered the thread with an insult against the moderator of the forum. That alone would qualify to whack you on the back of the head. Then you made comments like this:

1596690377252.png


Authority fallacy nonsense? As I said, you have zero common sense and self awareness. You seem extremely unhappy and agitated about what I do, and what is going on here. I am within an inch of showing you the door and lead you to happiness elsewhere without me.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
My concerns are:

1) If Amir claims that his sighted listening can detect things the measurements cannot, that’s an epistemological claim that runs counter the the accepted wisdom on this site. It means that there are some things better determined by listening, even sighted listening, than by measurements. This surely opens up room for broader subjective claims, which can no longer be shut down with objective claims.

In case of transducers and acoustics why would this not be the case ? One would need a lot more different measurements and have a lot of practical experience linking measurements to what's heard to be able to state... measurements are enough.

For electronics, however, it is a whole different story. It is very easy to measure this and when one knows about thresholds etc. one can easily hold the claim that measurements is all that is needed. Enough, conclusive, defined and well performed measurements that is.

2) If Amir claims that his sighted listening, as opposed to others’, is valid only because he is a trained listener, then there needs to be some evidence to support that. We also need the type of training that’s acceptable clearly specified. Perhaps we also need hearing test results. But it cannot be that only Amir is able to make valid subjective claims based on sighted listening. Openness, fairness, and reproducibility matter.

I don't think trained listeners always get a certificate (they might ?) stating they completed coarse this or that including a rating how 'accurate' they were. On would have to trust they did I reckon.
Then one can also be a trained listener by being self taught.
Let's differentiate between sound quality evaluations and comparisons.
Let's also differentiate between electronics testing and transducer testing.
Let's also differentiate between transducers in a room. A dead room is not the same as a treated room, a treated room is not the same as another treated room and again not anything like a room at home. Speaker positioning, room aspects, listening position. Any of these aspects can change the sound of a speaker.
It won't change the sound of electronics. Bias and sighted and/or not properly done tests will.|
These are the tests where one can adhere to blind tests and even trained listeners (trained to look for specifics).
Amir can only tell with certainty how he operates and knows jack about other peoples claims. He just tells us how he sees it.

3) Amir has taken pride at ASR’s reviews impacting manufacturers, positively and negatively. He took credit for Schiit’s changes in measurements and introduction of new product lines. Given that, the “recommended” versus “not recommended” distinctions matter. So if he’s going to say that, between two very similarly measuring speakers, one is “recommended” and one is a “not recommended” — and further claim that this is more than just his personal preference and is of greater value than others’ — there needs to be some proven validity to those distinctions, which brings us back to 1 and 2.

Schiit does have a good measuring cheap line. Schiit did buy an AP555. ASR certainly is a hot topic with fanboys and questions has been asked to Jason and others. Some aspects have been put out in the open which didn't before, or at least not so openly.
It is more than likely actions of ASR did result in certain changes. Jason openly admitted that the Heresy etc. was created for the measurement guys to show they can pull it off. Which measurement adoring folks ?
Nah.... ASR did have some influence there for sure.

Doesn't Amir always state why a certain device is recommended or not recommended ?
Yes, people buy on 'recommeded' and don't buy 'not recommended' gear but this is their decision and their responsibility.
The (prospective) buyer of gear can freely take recommendations at heart, look for other reviews or info and decide on their own.
Those that ONLY look at Amirs recommendations do that of conviction. Those that look at other 'known/renowned' reviewers and take their word as gospel also do that from their own conviction. I would lay that responsibility with the one planning a purchase based on their own.
They can always return stuff if the don't like it. Consider this a learning curve.

There are Amir (ASR) critics and fans. The thread is more about that than the actual benefits or pitfalls of blind/sighted testing and when these are warranted/essential or not.
 
Last edited:

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
Let me make a few things clear:

1. I will not stop listening to speakers and reporting on what I find. It is a requirement for the site performing reviews of speakers to listen to them. In addition, I will continue to give a recommendation/or not in each review. Membership has hugely appreciated this as has the industry as a whole.

2. I personally find the process of listening, EQing and manipulating speaker response incredibly educational. It has been massively useful and rewarding. You can be part of it by reading what I post in that regard, or not.

3. I know some of the times my "like or not like" may be in error. So far they are 75% or better in agreement with measurements. So if they are wrong a lot, so are the measurements! The risk is known to anyone with common sense that a person's subjective opinion is not ironclad. You paid nothing to get my subjective opinion. So don't complain about it not being a good deal.

4. Over time we may develop our own refinements to objective measurements to predict listener preference and happiness. Things like SPL testing, distortion, etc. will get us closer.

5. Threads like this take away time from doing reviews. I was going to post a speaker review this morning and here is the evening and I have not.

6. Listening to speakers and evaluating them takes a lot of time and effort. It also brings with it a lot of grief as we see. Despite all of these negatives that brings on me, I am doing the work because I know the value it brings. Whether you are convinced or not, is not an issue for me. You have the strong fallback of ignoring this small section of the speaker reviews and paying attention to measurements.

So there. Now I am going to go work on more reviews.
 

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
Let's see. You entered the thread with an insult against the moderator of the forum. That alone would qualify to whack you on the back of the head. Then you made comments like this:

View attachment 76826

Authority fallacy nonsense? As I said, you have zero common sense and self awareness. You seem extremely unhappy and agitated about what I do, and what is going on here. I am within an inch of showing you the door and lead you to happiness elsewhere without me.

I did not “insult” a moderator. I said that his reasoning for changing the title of the thread, which was copied from Olive‘s blog post, didn’t make sense.

“Appeal to authority” is a specific logical fallacy.

It is also a plain fact that you don’t know whether you know more about any given topic than everyone on this site. Unless you can provably claim to know more about audio than everyone on the planet, it is a self-evidently true statement!

Nothing I have said is a remotely justifiable reason to ban me. I’m quite confident that any reasonable person who reads my posts will feel the same.
 
Last edited:

Rusty Shackleford

Active Member
Joined
May 16, 2018
Messages
255
Likes
550
In case of transducers and acoustics why would this not be the case ? One would need a lot more different measurements and have a lot of practical experience linking measurements to what's heard to be able to state... measurements are enough.

For electronics, however, it is a whole different story. It is very easy to measure this and when one knows about thresholds etc. one can easily hold the claim that measurements is all that is needed. Enough, conclusive, defined and well performed measurements that is.



I don't think trained listeners always get a certificate (they might ?) stating they completed coarse this or that including a rating how 'accurate' they were. On would have to trust they did I reckon.
Then one can also be a trained listener by being self taught.
Let's differentiate between sound quality evaluations and comparisons.
Let's also differentiate between electronics testing and transducer testing.
Let's also differentiate between transducers in a room. A dead room is not the same as a treated room, a treated room is not the same as another treated room and again not anything like a room at home. Speaker positioning, room aspects, listening position. Any of these aspects can change the sound of a speaker.
It won't change the sound of electronics. Bias and sighted and/or not properly done tests will.|
These are the tests where one can adhere to blind tests and even trained listeners (trained to look for specifics).
Amir can only tell with certainty how he operates and knows jack about other peoples claims. He just tells us how he sees it.



Schiit does have a good measuring cheap line. Schiit did buy an AP555. ASR certainly is a hot topic with fanboys and questions has been asked to Jason and others. Some aspects have been put out in the open which didn't before, or at least not so openly.
It is more than likely actions of ASR did result in certain changes. Jason openly admitted that the Heresy etc. was created for the measurement guys to show they can pull it off. Which measurement adoring folks ?
Nah.... ASR did have some influence there for sure.

Doesn't Amir always state why a certain device is recommended or not recommended ?
Yes, people buy on 'recommeded' and don't buy 'not recommended' gear but this is their decision and their responsibility.
The (prospective) buyer of gear can freely take recommendations at heart, look for other reviews or info and decide on their own.
Those that ONLY look at Amirs recommendations do that of conviction. Those that look at other 'known/renowned' reviewers and take their word as gospel also do that from their own conviction. I would lay that responsibility with the one planning a purchase based on their own.
They can always return stuff if the don't like it. Consider this a learning curve.

There are Amir (ASR) critics and fans. The thread is more about that than the actual benefits or pitfalls of blind/sighted testing and when these are warranted/essential or not.

I agree with much of what you said.

However, if we open the door to trained listeners being able to hear things others can’t, it doesn’t necessarily end at transducers. I’ve yet to see any peer-reviewed research demonstrating that it’s impossible to reliably discern differences between amps and DACs that measure similarly. I don’t even know how you could definitively prove that. The only thing you could do is take someone who claims they can do so sighted and have them do a blind test. (Obviously, that doesn’t mean that some people who think they hear differences aren’t wrong. People fool themselves all the time.) We can also debate what the parameters of “similar measuring” are. But even if they don’t vary as much as transducers, roughly similar DACs can still vary in measurements. Moreover, a true subjectivist would say that there’s something not captured by the given suite of measurements, which is essentially what’s being argued with regard to speakers.

In terms of ASR impacting purchasing, isn’t informing consumers it’s stated goal? Isn’t the claim that Amir’s reviews are different from other sites’ because they’re “science,” as stated in the title of the site? The site cannot simultaneously claim to be above mere subjective reviewing, positioning itself as guarding consumers against “audiophoolery,” and then disclaim any responsibility for impacting people’s purchasing decisions. That’s the stated goal! Given that, it seems irresponsible to levy “recommend” and “cannot recommend” based on subjective evaluations, unless Amir is willing to say his opinion is just that, no more valuable than any number of other sites and not necessarily discerning something missing in the measurements.
 
Last edited:

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
I agree with much of what you said.

However, if we open the door to trained listeners being able to hear things others can’t, it doesn’t necessarily end at transducers. I’ve yet to see any peer-reviewed research demonstrating that it’s impossible to reliably discern differences between amps and DACs that measure similarly. I don’t even know how you could definitively prove that. The only thing you could do is take someone who claims they can do so sighted and have them do a blind test. Obviously, that doesn’t mean that some people who think they hear differences aren’t wrong. People fool themselves all the time. We can also debate what the parameters of “similar measuring” are. But even if they don’t vary as much as transducers, roughly similar DACs can still vary in measurements. Moreover, a true subjectivist would say that there’s something not captured by the given suite of measurements, which is essentially what’s being argued with regard to speakers.

Note that while you do not want to 'open the door' to trained listeners being able to hear things others can't, there are specific companies like Dolby, Microsoft and Harman who actually opened their doors, trained and paid them. Or are you taking the position that such people, and job positions, do not actually exist?

Update:
Did a quick google search and this came up:
http://www.stroudaudio.com/skilled-listener-training.htm

Is this considered a real thing from your pov? That someone can be trained and go on to work as a skilled / trained listener?

Update2:
Is Jodie Lampert a real person with a real job (among '40 other trained listeners' as mentioned in the article)?
https://www.hourdetroit.com/community/harman-international-car-audio-tech-test-listeners/
 
Last edited:

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
I agree with much of what you said.

However, if we open the door to trained listeners being able to hear things others can’t, it doesn’t necessarily end at transducers. I’ve yet to see any peer-reviewed research demonstrating that it’s impossible to reliably discern differences between amps and DACs that measure similarly. I don’t even know how you could definitively prove that. The only thing you could do is take someone who claims they can do so sighted and have them do a blind test. Obviously, that doesn’t mean that some people who think they hear differences aren’t wrong. People fool themselves all the time. We can also debate what the parameters of “similar measuring” are. But even if they don’t vary as much as transducers, roughly similar DACs can still vary in measurements. Moreover, a true subjectivist would say that there’s something not captured by the given suite of measurements, which is essentially what’s being argued with regard to speakers.

In terms of ASR impacting purchasing, isn’t informing consumers it’s stated goal? Isn’t the claim that Amir’s reviews are different from other sites’ because they’re “science,” as stated in the title of the site? The site cannot simultaneously claim to be above mere subjective reviewing, positioning itself as guarding consumers against “audiophoolery,” and then disclaim any responsibility for impacting people’s purchasing decisions. That’s the stated goal! Given that, it seems irresponsible to levy “recommend” and “cannot recommend” based on subjective evaluations, unless Amir is willing to say his opinion is just that, no more valuable than any number of other sites and not necessarily discerning something missing in the measurements.

The problem is that people who claim to hear differences that do not seem to be very likely probably won't volunteer.
Not talking about transducers but amps, pre-amps, DAC's etc. There are multiple ways to test.
Power amps, of course, are load dependent and could in certain circumstances differ from lab conditions. Might even reach audible thresholds.
That doesn't mean that you can hear things that are not measurable, demonstrable.
Amir has offered money for those that can demonstrate this ability.
That should be like taking candy from a baby. The offered reward by Amir is enough to compensate for expenses made to visit Amir.
Why hasn't anyone taken this opportunity ?

The test HAS to be administered by someone who knows how to test correctly. We all know very, very few people compare or test correctly.
In fact the very very vast majority that makes claims of audibility does so based on obviously sighted and incorrectly performed 'tests'.
Valid to them, not valid to act as definitive proof in court.


I think the way you look at Amir and ASR and its goals differ from the one set out by Amir.... The way you look at it seems to stem more from an 'haters' or 'sceptics' standpoint towards measurements.
I believe Amir's goal is to 'peer review' manufacturers specs, to weed out stuff that doesn't perform well. Also to inform people that are like-minded. Maybe to point out flaws to manufacturers. Its goal isn't to please or ridicule people that prefer subjective and sighted testing.
What ASR can do is point out why certain testing is SUSPECT (and thus not very valid).

of course... my p.o.v. not necesarilly yours or others. It's why I like it here. There are 'like minded' folks here and those that want to learn a bit.
Also those who oppose ASR views who like to put a magnifying glass on everything and every word but refuse to test properly.
That's life. Everyone should be free. Life isn't always fair either.
 

williamwally

Member
Forum Donor
Joined
Jan 6, 2020
Messages
68
Likes
141
Location
IL
So what are the realistic solutions to the problems people are finding in these reviews?

Conduct 70+ scientifically rigorous double blind studies within a year? Gluck
Get rid of recommendations which are there only because so many people wanted them, and most still do?

And on all these criticisms, who does it better?
If this is all so bad, what sites, what reviewers, what methods etc, give scientifically sound measurements along with subjective evaluations by someone trained to hear deficiencies/distortions etc and has been paid highly in the past to do so and happens at the pace that's happening here?
 

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,449
Likes
4,209
Note that while you do not want to 'open the door' to trained listeners being able to hear things others can't, there are specific companies like Dolby, Microsoft and Harman who actually opened their doors, trained and paid them. Or are you taking the position that such people, and job positions, do not actually exist?
http://www.stroudaudio.com/skilled-listener-training.htm Is this considered a real thing from your pov? That someone can be trained and go on to work as a skilled / trained listener? Is Jodie Lampert a real person with a real job (among '40 other trained listeners' as mentioned in the article)?
https://www.hourdetroit.com/community/harman-international-car-audio-tech-test-listeners/
Nowhere, on either of those links, does it say that they are being trained to do sighted listening. Until shown otherwise, I assume the training is to allow them to do blind listening more efficiently.
 
Last edited:

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
Nowhere, on either of those links, does it say that they are being trained to do sighted listening. Until shown otherwise, I assume the training is to allow them to do blind listening more efficiently.

Hmm....and nowhere did the links mention that the training is to allow them to do 'blind listening more efficiently'??
What makes you assume one over the other then?
 

Tks

Major Contributor
Joined
Apr 1, 2019
Messages
3,221
Likes
5,494
So what are the realistic solutions to the problems people are finding in these reviews?

Conduct 70+ scientifically rigorous double blind studies within a year? Gluck
Get rid of recommendations which are there only because so many people wanted them, and most still do?

And on all these criticisms, who does it better?
If this is all so bad, what sites, what reviewers, what methods etc, give scientifically sound measurements along with subjective evaluations by someone trained to hear deficiencies/distortions etc and has been paid highly in the past to do so and happens at the pace that's happening here?

Best solution would be a disclaimer on the entailment's of virtually any subjective deduction of a sound comparison or experience. That being one with a preface (similar to the repeating one about the Klippel measurement system in each speaker review) explaining what a subjective notion is influenced by. So not just saying "psychological bias", but instead some specifics for newcomers, like how time-of-day, emotional state, and pre-existing preferences and attitudes towards experiences can influence something as broad as a recommendation.

This would be in conformity with the various tones of each review (the differing nature between a device that cooperates as it should, versus a device with trash drivers, awful build quality, moronic UI, bad internals, or just awful software functionality - I've seen review metrics withheld because of devices like this or a bad day, or a rush to finish a review). And it falls in line with the basic understanding of how varied any subjective impression can be of a product. So imagine you're having a shit day, you're having to lug around some massive AVR or some other weird device, you're calling the company because some clown was forced to hurry up and get a product out the door that now doesn't function as advertised even.. Yeah, the subjective final verdict that will dictate a recommendation or not is going to be effected by all of this.

Amir naturally doesn't get a pass from all slip-ups seeing as how he's making his work available for comment to the general public. So even if he's aware of these little things -and- he has training, it won't do much to help him in the end in the same way no amount of effort has aided my brain from being immune to something like this. But this goes for basically anyone. The problem Amir potentially runs into, is as that other dude mentioned. Saying his training (or his experience) is allowing him to hear things measurements aren't able to reveal (potentially insinuating there is a metric he can't put into words, but is nonetheless claiming an unknown or unbenounced to others; artifact exists that current measurement systems are incapable of capturing for observation). That's when you start getting into serious waters where the skeptics raise from their seats and demand either further investigation or demonstration of such claims in comparative form so others can potentially expose themselves to an emulation of such an experience. Now of course Amir doesn't have to do any of that, in the same way he doesn't have to actually listen to anyone at all in reality.

I personally don't care about the subjective notions beyond just to see how consistent his subjective thoughts are. In terms of reviews, the only time they're interesting is when there is a Frequency Response comment that speaks about how "because there is no bass, even with the rolled off treble, the device is still seemingly harsh", those are always interesting since that's where most variability in sound comes from in my opinion. But in terms of recommendations, yeah, those are influenced by so many things it's not even worth listing (foods I used to hate, I now love - music I used to love, now find boring, genre's I used to hate, now I only listen to), all because of who knows how many factors...
 

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,449
Likes
4,209
@whazzup Occam's Razor, "until shown otherwise".

Very interested in your reason for assuming what you assumed.....?
 

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
@whazzup Occam's Razor, "until shown otherwise"

Very interested in your reason for assuming what you assumed.....?

Well, at least we know there're at least 40 other Amirs.... anyone care to contact / harass them to ask about whether they test speakers blind every time? :D

Because of practicality. Because it's a job. Because speed in carrying out a job is important. It's a logistical issue (to test blind every single time). Does not make sense at all.
 

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,449
Likes
4,209
Until you do so, it remains more sensible to assume blind. OK?
 

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,449
Likes
4,209
What are you going to do when Michael Fremer pipes up and says "I did that test. Now you all have to defer to my (sighted) views." Hmmm? He already writes things like this:
"I've taken many blind tests including at Harman's test facility and demonstrated the reliability of my listening, and both JA and I took part in a 5 amplifier blind test at an AES after I agreed to a challenge made by former Audio writer Dave Clark who claimed all amps that measure the same sound the same. I got all identifications correct and then though it wasn't part of the test, once we knew the identities of all five amps, I correctly identified ("blind") which was which. Nonetheless, though I scored 100% and JA scored 80%, because the population did no better than chance, we were both declared "outliers" and our results were tossed! What a joke. Interestingly, Clark included a VTL 300 tube amp in his test, an amp that measured and sounded very different than the others, which were solid state. Yet most of the AES crowd couldn't hear that one either! Why? Because under double blind conditions inexperienced test takers usually screw up and them (sic) did."

You want to open that door? :)
 
Last edited:

whazzup

Addicted to Fun and Learning
Joined
Feb 19, 2020
Messages
575
Likes
486
Until you do so, it remains more sensible to assume blind. OK?
You probably missed my updated post. But no, Amir's words has been proven by 3rd party data (like the articles) time and time again. And his words have a clear logic, unlike people who opposes in very general terms and then fails in providing backing data.

What are you going to do when Michael Fremer pipes up and says "I did that test. Now you all have to defer to my (sighted) views." Hmmm?
If he actually scored a job at Harman, trained there for 8 months as a listener and then willing to stake his professional certificate / reputation on the line for saying what he says, sure. I probably won't be the first to call him out, his peers probably will.

So anyone else can verify that 'his results are tossed'? That he took the harman tests? That he did the 8 months of training?
 
Last edited:

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,449
Likes
4,209
You probably missed my updated post. But no, I don't see your wish to open that door as helpful.
 
Top Bottom