The frailty of Sighted Listening Tests

[amirm]

As you just said: blind test. That is not what is at question here, is it?

You really missed the point of that detailed post by such a mile?

I was explaining that showing spinorama measurements is not prescriptive enough and indeed, can be misleading given that blind tests did not agree with them. My sighted tests will give us a shot at hopefully determining what the elaborate, time consuming blind test produced.

 

[blueone]

You really missed the point of that detailed post by such a mile?

I was explaining that showing spinorama measurements is not prescriptive enough and indeed, can be misleading given that blind tests did not agree with them. My sighted tests will give us a shot at hopefully determining what the elaborate, time consuming blind test produced.

I think you should have a new ASR posting rule. You have to be a Forum Donor at the $100 level or higher to have a running argument with you about review efficacy. I've noticed there is a negative correlation between donors and complainers.

[Only kidding. A little...]

 

[amirm]

I think you should have a new ASR posting rule. You have to be a Forum Donor at the $100 level or higher to have a running argument with you about review efficacy. I've noticed there is a negative correlation between donors and complainers.

[Only kidding. A little...]

No, your are right on point. Usually the most obnoxious and argumentative people I see on the forum are the ones who have neither contributed to the forum financially or with loaned gear. I notice it all the time. It tells me that they can't stand me and what we are doing here and the argument they are having, is mostly sourced due to that.

 

[aarons915]

If I didn't listen, we would walk right into these subjectivist arguments.

As I have mentioned, for things that show no measured performance difference, we are not going to do anything but blind testing.

You all need to get used to two different domains of products we test: those that by definition sound different from each other (speakers), and everything else. One size does not fit all.

Maybe you can show me a study that says listening to speakers is no longer required as measurements are completely descriptive.

Agreed, I don't think you should do away with the subjective portion of the test. I think the arguments come out every time you don't like a speaker that seems to measure well. What do you think about instead of just saying a speaker is "good or bad", you compare it to a reference speaker so that every speaker is compared to a standard? I think the Revel M16 would be a good candidate but any decent speaker would do. I wonder sometimes if regularly listening to the Salon 2 makes most other speakers sound mediocre in comparison?

 

[blueone]

I wonder sometimes if regularly listening to the Salon 2 makes most other speakers sound mediocre in comparison?

I don't think so. My music system has Salon2s, and my HT system and my desktop system have lesser speakers. The Salon2 system does sound better than the other two, but that doesn't prevent me from enjoying them. Video sound is so contrived that accuracy seems to mean less than, say, dialog intelligibility, and I'm not convinced any speaker can sound truly great on my large hard-surfaced desktop closely flanking a large monitor.

 

[amirm]

Agreed, I don't think you should do away with the subjective portion of the test. I think the arguments come out every time you don't like a speaker that seems to measure well. What do you think about instead of just saying a speaker is "good or bad", you compare it to a reference speaker so that every speaker is compared to a standard? I think the Revel M16 would be a good candidate but any decent speaker would do. I wonder sometimes if regularly listening to the Salon 2 makes most other speakers sound mediocre in comparison?

All good speakers regardless of brand sound similar to me. They are balanced tonally, clean and enjoyable to listen to across my large suite of music tracks. With poor speakers, one track sounds OK but not the next one.

The Salon 2 has impressive low bass which none of these bookshelf speakers can reproduce. But I don't judge them this way.

 

[Rusty Shackleford]

You really missed the point of that detailed post by such a mile?

I was explaining that showing spinorama measurements is not prescriptive enough and indeed, can be misleading given that blind tests did not agree with them. My sighted tests will give us a shot at hopefully determining what the elaborate, time consuming blind test produced.

No. It doesn’t follow that because blind tests allow us to discern a difference that sighted tests must, too. You’ve also chastised people who cite the complexity and expense of blind testing speakers for making “excuses.”

Regardless, the takeaway here and elsewhere from your comments is that you think sighted listening is acceptable, particularly when the listener trained. Indeed, you’ve said that trained listeners can be seen as “golden ears” who can discern things that others can’t. (Regardless of what you say, however, this cannot be limited to one category of equipment unless the trained listener who claims to hear a difference between, say, DACs later fails a level-matched blind test. After all, how else could you prove that the listener isn’t simply better trained or naturally gifted with superior hearing?)

Given that How to Listen was created by Harman to train it’s employees and is freely available (not to mention SoundGym and countless other sites and apps), it’s trivially easy for anyone to become a trained listener.

Let 1,000 sighted listening reviews bloom!

 

[Rusty Shackleford]

I think you should have a new ASR posting rule. You have to be a Forum Donor at the $100 level or higher to have a running argument with you about review efficacy. I've noticed there is a negative correlation between donors and complainers.

[Only kidding. A little...]

No, your are right on point. Usually the most obnoxious and argumentative people I see on the forum are the ones who have neither contributed to the forum financially or with loaned gear. I notice it all the time. It tells me that they can't stand me and what we are doing here and the argument they are having, is mostly sourced due to that.

Or perhaps many forum members are of modest means and have more pressing financial worries than donating to a retired executive from one of the biggest companies in the world?

The notion that one needs to send you money to be a legitimate member of this site is offensive. If you want to have a one-way discussion where you cannot be challenged by those whom you call “Joe Blow,” maybe just turn ASR into a comment-disabled website, rather than a forum.

 

[Rusty Shackleford]

I’m done with this thread and the SVS one. (The irony is that I don’t even own SVS speakers. I own Revels. But fairness is fairness.)

As with the MQA thread, when Amir is challenged, he resorts to ad hominem attacks and specious arguments from authority. It’s sad.

 

[whazzup]

No. It doesn’t follow that because blind tests allow us to discern a difference that sighted tests must, too. You’ve also chastised people who cite the complexity and expense of blind testing speakers for making “excuses.”

The way I understand Amir's words, he's saying a difference is already identified.... why then do you need to continue blind testings? The work should then be on figuring out WHY there's a difference, not MORE blind testings....

 

[Rusty Shackleford]

The way I understand Amir's words, he's saying a difference is already identified.... why then do you need to continue blind testings? The work should then be on figuring out WHY there's a difference, not MORE blind testings....

Yes, that is obviously true. But that doesn’t hold in the case in question in this thread. Whether there is, in fact, a clear difference is what’s up for debate.

 

[whazzup]

Yes, that is obviously true. But that doesn’t hold in the case in question in this thread. Whether there is, in fact, a clear difference is what’s up for debate.

Noted. My reply was in reference to your reply to Amir '...blind tests allow us to discern a difference that sighted tests must too...'

It seems then your beef should be with Harman, on the details / certification of their listening training and who gets to graduate?

 

[Blumlein 88]

I'm very much in agreement with Rusty Shackleford here. I also am not trying to chastise Amir. I know with speakers you have no choice except at some point to listen sighted. And that listening sighted at least has the potential to add useful information though I'd say at a lower level of reliability.

I think Amir you are a little too sensitive about this sometimes. For instance, nothing I've seen indicates trained listeners can substantially avoid bias just like untrained listeners. It makes sense that it probably is so though at what level and how much difference is not defined. I've asked if you have info about this, and unless I've missed it you mainly have told us how you used trained listeners working sighted at MS, and that Olive has indicated they do the same.

I go back to the optometrist situation. What he does is like asking audiophiles to pick better or worse or maybe louder or softer when differences are quite substantial and one can do that sighted. Even then it looks like blinded tests are more sensitive. However, as long as differences are huge relatively then sighted for disciplined trained people works well enough.

To jump sideways on this a bit: I believe the whole wrong turn subjective high end audiophilia took was the following. They believed and acted upon sighted listening impressions in ways that made sense for loudspeakers, but did so for devices that have no substantial difference. As stated someone says two speakers sounds different.......well duh? Someone says two wires sound different........now hold on this is ridiculous. In one case the differences are large, and in the other basically being none, the only thing that sways a person with belief one way or the other is the sighted bias. In the case of loudspeakers even with bias, the real differences are big enough people who care are not likely to tell us one sounds too bright if it doesn't really sound too bright. Maybe they'll be forgiving of a tiny brightness for a brand they think highly of, but likely not if the brightness is quite substantial.

So I'm not asking you to stop giving up your listening impressions. They do have value. But you come off sounding not a lot different than reviewers at Stereophile in the process. It gives me pause not because of your listening impressions. But when I see how Zu speakers measure, and have heard Zu loudspeakers and then reviewers write an article that indicates they are worthy of consideration I have to do a full stop, take a deep breath and wonder about a lot of things. That is one reason if you know of some criteria for trained listeners, and tests showing they can really substantially lower the portion of sighted listening which is biased I'd like to know more about it. As an untrained, but somewhat experienced audiophile I don't see a place in the world where Zu should be able to stay in business.

 

[MattHooper]

I think you should have a new ASR posting rule. You have to be a Forum Donor at the $100 level or higher to have a running argument with you about review efficacy. I've noticed there is a negative correlation between donors and complainers.

[Only kidding. A little...]

Careful. This makes you vulnerable to the reply: Maybe it's the ones without skin in the game, sunk cost bias, who can be more objective about the site.

[Only kidding. A little...]

 

[whazzup]

Where does one go to apply and be trained and certified as a listener? I think what people are questioning is this certification program and its validity.

Well, except the unannouced trolls / people with vested interests, that is.

 

[amirm]

Or perhaps many forum members are of modest means and have more pressing financial worries than donating to a retired executive from one of the biggest companies in the world?

I observed a correlation between those rude, obnoxious and aggressive posts on the forum who pick fights with me. Almost universally they are not supporting the forum in any form or fashion. They are big on complaining and being victims, but ask them to lift a finger to help figure out answers and all of a sudden, they are full of excuses.

Other members who don't fall in your type, can be free to donate or not donate. They don't fall in the category of picking fights with me and what the forum is about so your response is orthogonal to what we are discussing.

The notion that one needs to send you money to be a legitimate member of this site is offensive.

You have a serious problem following discussions. Legitimate members are here because they appreciate what we are doing. My only dealings with you is protests, complaints, insults, etc. So you are not a legitimate member of this forum. Still, out of extreme deference, we tolerate you like any other member even though you don't deserve such.

If you want to have a one-way discussion where you cannot be challenged by those whom you call “Joe Blow,” maybe just turn ASR into a comment-disabled website, rather than a forum.

Spoken like a bully that you are. If you want to know one thing about me is that I don't reward bullies by doing what they demand. You have no case in this argument so that is two reasons I don't listen to you.

Many other members have influenced me causing me to hugely change the way we do things. I test speakers because members pointed out that it is more important to test them than electronics. So I made a huge investment to get there. What did I get in return from a few like you? The most belligerent posts possible as a spoiled, self-entitled, angry mob leader that you are.

Learn some common sense. You are an invited guest. If you want to influence your host to make you a different meal because you don't like steak, learn to make a proper argument cordially and constructively. Anything else will get you nothing.

 

[Racheski]

So I'm not asking you to stop giving up your listening impressions. They do have value. But you come off sounding not a lot different than reviewers at Stereophile in the process.

Stereophile:
"Their ability to cast a wide, deep soundstage with super imaging was as good as any bookshelf speaker I've had in house. Their serious jump factor and meaty bass frequencies were a consistent treat. The speaker's slightly dry upper frequencies demand careful amplifier matching, but otherwise, for a sub-$500 speaker, there's not a lot to criticize. Coherent to their core, they never made me focus on a single aspect of their performance but led me deeply into the whole of the music."
Amir:
"Having looked at the measurements first before listening, I thought they would either sound good "out of the box" or do so with just a bit of EQ. That did not happen. Try as I did, I could not like the sound with or without EQ. Yes, I could see the appeal of boosted upper bass in giving "warmth" to the speaker. But too much of that was well, too much of a good thing. I am puzzled where the sharpness was coming from seeing how I could not tame it. Perhaps directivity error was causing this.

I can't recommend a speaker that doesn't sound good to me and that is where I stand with SVS Ultra. I suspect measurement score will be good, making me look bad. So be it!"

You can't be serious...

 

[MattHooper]

I'm very much in agreement with Rusty Shackleford here. I also am not trying to chastise Amir.

Likewise. I agree that, without even for a moment dissing any part of Amir's reviews: as I mentioned, Rusty Shackleford has found some reasonable pressure points to push here. (And using "not contributing to the site or not sending gear" as further grounds to undermine an interlocutor does seem a form of ad hominem).

I think Amir you are a little too sensitive about this sometimes. For instance, nothing I've seen indicates trained listeners can substantially avoid bias just like untrained listeners.

Yes this was the subject that was having me scrunch my nose somewhat in puzzlement.

I guess I've spent so much time trying to explain to the subjectivists on other sites that, no, you can't simply make bias go away by trying to be objective, or not having expectations. And no you can't avoid bias effects because you think you are such a sensitive listener. Biases affect every person. It's the reason why even scientists who are more knowledgeable about bias don't trust even themselves, and submit to controls for bias.

And to turn around and accept that the owner of this site somehow is able to evade the bias problems of sighted testing by being a "trained listener" - that is a really good, perceptive listener! - would feel rather hypocritical of me to accept. (Although I may well have misunderstood what Amir was saying...this does seem to be the tenor of his argument).

This is why I simply enjoy Amir's subjective reports on the grounds he is indeed a skilled listener, but I downgrade the confidence level suitably given it's sighted conditions. But any credence I give Amir's subjective reports has to do simply with his experience in listening to speakers and identifying various issues. But it's not with the further assumption that this allows him to largely avoid sighted bias. I also get good info from other subjective reports from other people too.

 

[Rusty Shackleford]

I observed a correlation between those rude, obnoxious and aggressive posts on the forum who pick fights with me. Almost universally they are not supporting the forum in any form or fashion. They are big on complaining and being victims, but ask them to lift a finger to help figure out answers and all of a sudden, they are full of excuses.

Other members who don't fall in your type, can be free to donate or not donate. They don't fall in the category of picking fights with me and what the forum is about so your response is orthogonal to what we are discussing.


You have a serious problem following discussions. Legitimate members are here because they appreciate what we are doing. My only dealings with you is protests, complaints, insults, etc. So you are not a legitimate member of this forum. Still, out of extreme deference, we tolerate you like any other member even though you don't deserve such.


Spoken like a bully that you are. If you want to know one thing about me is that I don't reward bullies by doing what they demand. You have no case in this argument so that is two reasons I don't listen to you.

Many other members have influenced me causing me to hugely change the way we do things. I test speakers because members pointed out that it is more important to test them than electronics. So I made a huge investment to get there. What did I get in return from a few like you? The most belligerent posts possible as a spoiled, self-entitled, angry mob leader that you are.

Learn some common sense. You are an invited guest. If you want to influence your host to make you a different meal because you don't like steak, learn to make a proper argument cordially and constructively. Anything else will get you nothing.

I invite everyone to compare my posts to yours. Nothing I have said is “bullying.” You, on the other hand, have referred to other reviewers as “Joe Blow” and flatly stated “I know more than you,” even though you don’t know the identities of everyone on this forum, their backgrounds, or (apropos of the debate here) their hearing training.

As with the MQA thread, when you’re challenged with contradictory research, you become abusive and simply assert your superiority, rather than demonstrate it with evidence or clear reasoning.

 

[amirm]

Where does one go to apply and be trained and certified as a listener? I think what people are questioning is this certification program and its validity.

Well, except the unannouced trolls / people with vested interests, that is.

I only know of trained listeners in the few companies that have them (Dolby, Microsoft, Harman, and a few others I don't know).

AES used to have a training CD you could get to learn to hear compression artifacts. I had it but it was after I was self-trained so don't know how good it is/was. See: If you want to have a one-way discussion where you cannot be challenged by those whom you call “Joe Blow,” maybe just turn ASR into a comment-disabled website, rather than a forum.

For speaker training, Harman provides their training software for free although I don't know if they still maintain it. See: https://www.audiosciencereview.com/forum/index.php?threads/harmans-how-to-listen-software.8262/

I highly suggest you get a software EQ and make small EQ changes, varying Q and see what impact it has on tonality.