The frailty of Sighted Listening Tests

[whazzup]

Not to belabor this, but isn't the whole point of this debate that Amir was claiming that his subjective impression were correct because he's a "trained listener"? That's how we got into what counts as training, if trained listeners are actually more valid or simply more reliable, etc. If he were simply saying "I like this better, but those are just my personal tastes" I agree it wouldn't be an issue.

I also don't think subjective reviews have no standards, insofar as that means "best practices." Certainly, things like level matching, having the equipment compared in possession at the same time, real-time switching, using consistent music, specifying what you are or aren't hearing in clear and precise language, etc. can make a subjective review more or less informative.

Then the debate should be about trained listeners as a profession? and how far their sighted testing ability goes.

 

[amirm]

Let us note: this is the approach already used by Stereophile for decades. In almost all reviews, the reviewer gives the subjective report and the measurements by JA are made afterwards.

And a lot of good it does for fidelity of their subjective tests! Of course if you think the measurements we do don't correlate with preference, then sure, not looking at them beforehand is wise. Is this what you are saying?

If the reason you say this is because you think my subjective views will deviate even more from measurements that way, I can see the riots in streets getting much worse. Again, is this what you want to happen?

 

[MattHooper]

And a lot of good it does for fidelity of their subjective tests! Of course if you think the measurements we do don't correlate with preference, then sure, not looking at them beforehand is wise. Is this what you are saying?

If the reason you say this is because you think my subjective views will deviate even more from measurements that way, I can see the riots in streets getting much worse. Again, is this what you want to happen?

Wasn't there feedback on your speaker reviews that suggested it would be a good idea to make the subjective listening report occur before doing the measurements, one main reason being knowing the measurements could set some level of expectation bias in what you hear?

And didn't you then adopt that mode so now you do the subjective review before you measure? Or am I mistaken?

I'd simply pointed out that stereophile has been doing that - keeping reviewers in the dark about measurements until after the subjective review - for years.

Geeze, I'm not leading some assault on the castle here. I've mentioned several times I'm happy with you combining the subjective sighted reviews with the measurements!

 

[amirm]

If by “help the garden” and “buy tools,” you mean donate money or gear, I think that’s silly. Not everyone has the funds to donate or would choose to send to a retired MS executive rather than, say, a food bank.

You have a knack for twisting things. Once again, the original comment was that people who complain bitterly about what we do and furthermore grab on my pant legs and won't let go, are also the ones won't contribute in any way to the forum positively. You back that data set perfectly.

That aside, people donate not because they think I am poor. They donate because it makes them feel good that they are helping a good cause. Consumers badly need more information about audio products they purchase. Despite all the money and time I have put in toward this, we are way, way away from the goal of having useful information we need for anything we want to purchase. In the budget speaker thread for example, there are 10X more suggestions than what I can reasonably buy to test. Yet the need is firmly there.

Having more people donate money, equipment to test, do their own reviews/tests is what we need. Not some kind of self justification that your money should go to food banks but mine should b spent buying measurement gear, sample to test, and pay for shipping to send things back.

Finally dealing with obnoxious people on this forum needs a counteracting form. I have very thick skin but at times it gets quite excessive and seeing a donation or two brightens my day. Not for the monetary front necessarily but the thought that goes behind it. I have people who are in financial bind but still make an effort to donate a few dollars.

But if that’s the case, just be transparent and make posting ability contingent on donation.

I rather see donations be genuine than creating classes of memberships as others have done with Patreon content/privileges, etc. I rather think that any audiophile sees the value of what we are doing and in their way, they help. Help can be positive, constructive posts. It can be words of encouragement. It can be loaning gear even if it is a $20 dongle as someone just donated today. Or as a minimum, not come here, eat the food for free, but bitterly complain that he doesn't like this kind of food.

 

[amirm]

2) This forum is focused on informing consumers, thereby impacting purchasing decisions. Companies viewed to have performed poorly have been mercilessly criticized and mocked. The defense of that was that such conclusions were based on measurements, not opinions. That may or may not have been a valid defense, but it was a clear one. However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO.

How do you know SVS is negatively impacted? They received objective measurements that would cost them thousands of dollars to produce themselves. Said objective data painted a positive view of their speaker by Olive score which you are trumpeting later in your post. So folks like you are more inclined to buy their gear than not.

As for my subjective view, that was not "poor" either. Here is what I said:

I gave the speaker "I don't know panther," not the headless one that I reserve for poor products.

The notion that subjective remarks are of no value in speaker evaluation is completely non-sequitur. This is completely standard practice: to listen to a speaker and post what you thought about it in a review. Of course, I am most alone in positive direct and negative remarks when the sound of a speaker is not pleasing to me. But nothing about that is out of line in what a reviewer is supposed to do.

Of course, if you want to protect the manufacturer than inform consumers, you would cry when negative reviews are posted about your favorite brand. That, I can't help. I am 100% on the side of consumers. You need to decide where you want to be. If it is not where we are, then you will be at odds on the mission of this site.

 

[Rusty Shackleford]

You have a knack for twisting things. Once again, the original comment was that people who complain bitterly about what we do and furthermore grab on my pant legs and won't let go, are also the ones won't contribute in any way to the forum positively. You back that data set perfectly.

That aside, people donate not because they think I am poor. They donate because it makes them feel good that they are helping a good cause. Consumers badly need more information about audio products they purchase. Despite all the money and time I have put in toward this, we are way, way away from the goal of having useful information we need for anything we want to purchase. In the budget speaker thread for example, there are 10X more suggestions than what I can reasonably buy to test. Yet the need is firmly there.

Having more people donate money, equipment to test, do their own reviews/tests is what we need. Not some kind of self justification that your money should go to food banks but mine should b spent buying measurement gear, sample to test, and pay for shipping to send things back.

Finally dealing with obnoxious people on this forum needs a counteracting form. I have very thick skin but at times it gets quite excessive and seeing a donation or two brightens my day. Not for the monetary front necessarily but the thought that goes behind it. I have people who are in financial bind but still make an effort to donate a few dollars.


I rather see donations be genuine than creating classes of memberships as others have done with Patreon content/privileges, etc. I rather think that any audiophile sees the value of what we are doing and in their way, they help. Help can be positive, constructive posts. It can be words of encouragement. It can be loaning gear even if it is a $20 dongle as someone just donated today. Or as a minimum, not come here, eat the food for free, but bitterly complain that he doesn't like this kind of food.

Constructive posts are not always positive ones, and vice versa.

 

[Rusty Shackleford]

How do you know SVS is negatively impacted? They received objective measurements that would cost them thousands of dollars to produce themselves. Said objective data painted a positive view of their speaker by Olive score which you are trumpeting later in your post. So folks like you are more inclined to buy their gear than not.

As for my subjective view, that was not "poor" either. Here is what I said:

View attachment 77090

View attachment 77091

I gave the speaker "I don't know panther," not the headless one that I reserve for poor products.

The notion that subjective remarks are of no value in speaker evaluation is completely non-sequitur. This is completely standard practice: to listen to a speaker and post what you thought about it in a review. Of course, I am most alone in positive direct and negative remarks when the sound of a speaker is not pleasing to me. But nothing about that is out of line in what a reviewer is supposed to do.

Of course, if you want to protect the manufacturer than inform consumers, you would cry when negative reviews are posted about your favorite brand. That, I can't help. I am 100% on the side of consumers. You need to decide where you want to be. If it is not where we are, then you will be at odds on the mission of this site.

You simply highlighted the positive words, not the negative ones and the “I cannot recommend a speaker that doesn’t sound good to me” conclusion. With selective editing, we can turn a breakup letter into a love note. What does that prove?

Speaking of twisting words, where have I ever said I want to “protect the manufacturer [rather] than inform consumers”? I simply said that it’s important to be clear in review procedures to be fair to consumers and manufacturers. Nor is SVS my “favorite brand.” As I said earlier, I own Revel speakers!

 

[MattHooper]

Much of your post certainly makes sense, especially in regards to donations. I've been out of work due to COVID since February (actually, from well before...it was a weird year). But I'll scrape up a donation as I want to see this site flourish.

BTW: What donation is the entry price for having the right to question the review methods without being whacked with the Ungrateful Broom? ;-)

I kid, I kid..

Though, briefly...

You have a knack for twisting things.

........

Or as a minimum, not come here, eat the food for free, but bitterly complain that he doesn't like this kind of food.

Hmm, that also seems at least to me to be putting a twist on Dale's position. I asked him to explicate his position and, to me at least, I would not describe it as "bitterly complaining" that he doesn't like the reviewing or site content here. He seemed to be asking some reasonable questions to establish clarity and consistency.

 

[LDKTA]

Very seriously : He's a trained listener,.less subject to biaises than average people, and, as a former manager, he has a lot of experience on how things work. He knows a lot of people in the audio industry. He measured and listened to countless audio products, more than 60 or 70 speakers in the past two months. Etc.

"He's a trained listener,.less subject to biases than average people"

While this may be true, this is only more incentive to take his word for what it's worth -- make of it what you will. I don't even understand what is being debated at this point. Can Amir not have an opinion that isn't 100% correlative to the measurements of the loudspeaker that is being [or has been] measured? Again, this is not a black and white issue -- even as a trained listener, he is still human and isn't immune from fallibility or bias. As I'm sure you know, human beings are inherently biased. If he doesn't recommend a loudspeaker despite that loudspeaker measuring quite well, does this mean that YOU should not buy those loudspeakers? Does this mean that his recommendation is incorrect? Should he not be allowed to recommend products to the masses? If so, why? Again, the preference ratings/scores are NOT 100% accurate either. This is precisely why more people in the blind listening test over at AVS chose the Salon2 over the M2. Multiple times throughout the thread and having spoken with participants of that test, there were mentions of "splitting hairs" between the two, despite the M2 appearing to be more superior on paper. Yet, there are things the M2 can do [better] that the Salon2 cannot and vice versa. By the way, there've also been sighted comparisons done between the two and the Salon2 remained triumphant. Are those who preferred the Salon2 wrong for preferring the Salon2? In this case, I was pleasantly surprised to see how well the SVS Ultra Bookshelves measured... But I've listened to them extensively (in several different rooms) and have come to the same conclusion as Amir. Not to mention, I absolutely love the aesthetic of the Ultra Bookshelves -- a bias. Is my opinion invalid simply because I've only heard the loudspeakers in sighted listening?

Back to: Preference is inviolate. I think you should be as honest with yourself as humanly possible and determine why you feel the way you do towards the subjective portion of Amir's reviews. I reckon that your answer to all of your questions is there.

If anything, that is one aspect of the reviews [the preference ratings/scores] I wouldn't mind doing away with. I genuinely do not see the problem whatsoever though, I only see needless nitpicking and bickering because you want it YOUR way -- unfortunately, not everything is going to go your way in life. My recommendation would be to stop idealizing Amir and realize that he is only human like yourself... He just happens to be a trained listener that chooses to share his opinion with us. Does this make his objective and subjective evaluations any less scientific? Does this make his evaluations less valuable? To each their own. Whether they're unscientific or less valuable to YOU is irrelevant as the reviews aren't conducted specifically for YOU. He isn't looking to please you and he doesn't have to. If you were to put yourself in his shoes and did things your way, you'd encounter the same sort of commentary because no one is perfect and this is NOT a black and white issue. You cannot make something black and white that is actually shades of grey.

 

[LDKTA]

You simply highlighted the positive words, not the negative ones and the “I cannot recommend a speaker that doesn’t sound good to me” conclusion. With selective editing, we can turn a breakup letter into a love note. What does that prove?

Speaking of twisting words, where have I ever said I want to “protect the manufacturer [rather] than inform consumers”? I simply said that it’s important to be clear in review procedures to be fair to consumers and manufacturers. Nor is SVS my “favorite brand.” As I said earlier, I own Revel speakers!

"I cant recommend a speaker that doesn't sound good TO ME."

Should he recommend a loudspeaker that doesn't sound good TO HIM simply because it measured well? Why is Amir not allowed to have an opinion? I understand that this may be contrary to the research done at Harman/NRC, however, the conclusion they came to is not 100% accurate. Anomalies exist and it is ok to ignore them. I don't see how you'd lose sleep over them. It is only indicative of what we'd be more likely to prefer and even then, there is more at play here. We're talking about two totally different loudspeakers that do not perform identically. The SVS Ultra doesn't even have a waveguide, while the M106 does... They have totally different transducers and totally different crossovers. Do you legitimately believe that someone (especially a trained listener) would be unable to reliably tell the two apart via unsighted or sighted listening? That was not a rhetorical question. Considering that you own Revel loudspeakers, I'd imagine your loudspeakers also feature a waveguide and I'd like to believe you know how that affects your perception of the sound emitted from the tweeters (regardless of how similar these two loudspeakers may measure to one another).

 

[amirm]

Speaking of twisting words, where have I ever said I want to “protect the manufacturer [rather] than inform consumers”?

Right here in what you said: "However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. "

What the company does financially has nothing to do with unbiased review of a product. The only reason to worry about it is because you care about the fortunes of the company than telling the readers the truth. It is not like we get paid a bonus when the review is positive and company gets more sales.

I am here to tell consumers what I think of an audio product with zero bias with respect to who built it. I have lost industry friends as a result of such reviews. But that is the price of unbiased reviewing. The moment you worry about a company making money, is the moment you have lost objectivity.

Of course we want to be fair and that is why the bulk of my reviews are measurements which they can repeat and dispute if they like. Every review also ends with welcoming questions, comments, etc. from them. To wit, they are welcome to show double blind tests of their speakers doing as well as Revel. To the extent they are silent, then that is that.

 

[Rusty Shackleford]

Right here in what you said: "However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. "

What the company does financially has nothing to do with unbiased review of a product. The only reason to worry about it is because you care about the fortunes of the company than telling the readers the truth. It is not like we get paid a bonus when the review is positive and company gets more sales.

I am here to tell consumers what I think of an audio product with zero bias with respect to who built it. I have lost industry friends as a result of such reviews. But that is the price of unbiased reviewing. The moment you worry about a company making money, is the moment you have lost objectivity.

Of course we want to be fair and that is why the bulk of my reviews are measurements which they can repeat and dispute if they like. Every review also ends with welcoming questions, comments, etc. from them. To wit, they are welcome to show double blind tests of their speakers doing as well as Revel. To the extent they are silent, then that is that.

Once again, talk about twisting. If your goal is to inform consumers, that will necessarily impact businesses, given that they make their money from consumers.

 

[LDKTA]

Once again, talk about twisting. If your goal is to inform consumers, that will necessarily impact businesses, given that they make their money from consumers.

No, you're twisting. "If your goal is to inform consumers, that will necessarily impact businesses, given that they make their money from consumers." Consumers aren't prohibited from making their own informed or uninformed decisions. That is the joy in having a preference of your own.

 

[amirm]

Despite having remarkably similar preference scores, one would come away from reading ASR thinking that the the M106 (5.79) is a good, recommended speaker and the SVS (5.70) is a poor, not recommended speaker.

That data is given so for people who believe that score, SVS has been given a huge gift, equating their designs with Revel.

The research however, was published as a conference paper so not peer reviewed. No published report has duplicated it. Nor is Harman using it itself.

I think directionally it is a good measure seeing how my subjective listening tests agree with it most of the time even though I don't know that score when I am testing the speaker.

BTW, it is the height of silliness to think a preference score for a speaker can have two decimal places! The tool that gathers preference score only allows single digits:

Just because computationally you arrive at fractions, doesn't mean you use them to rate speakers.

No way is a speaker preference score accurate to 1/10th let alone 1/100th as you are quoting. The error bars themselves easily invalidate such:

You have to understand the limits of what the measurements are showing in a difficult field of predicting listener preference for tonality of a speaker. Don't be blindly wedded to them.

Of course, you could prove us all wrong. Get those two speakers and perform a blind test and show that preference is identical between them, to two decimal places. Until then, Harman research shows that poor directivity is one of the top reasons for lesser preference in listening tests and SVS violates that key research. If it turns out this is indeed the case, then there is a lot that researchers need to answer for.

Fortunately I think the research is good and proper and the issue is limitations of the scoring system which according to its authors, was going to continue to assess other factors: A Multiple Regression Model for Predicting Loudspeaker Preference Using Objective Measurements: Part II - Development of the Model Sean E. Olive, AES Fellow

It is possible all of this was done and hence the reason the model published is not used. Perhaps they have a better model that they don't want to share with their competitors. Or they lost interest in developing this further, either because they found issues or didn't think it was important.

Either way, it is great that we have what we have. But don't trust it like a bible. We need to confirm what it is telling us and that is what I do as a listener.

 

[Rusty Shackleford]

"I cant recommend a speaker that doesn't sound good TO ME."

Should he recommend a loudspeaker that doesn't sound good TO HIM simply because it measured well? Why is Amir not allowed to have an opinion? I understand that this may be contrary to the research done at Harman/NRC, however, the conclusion they came to is not 100% accurate. Anomalies exist and it is ok to ignore them. I don't see how you'd lose sleep over them. It is only indicative of what we'd be more likely to prefer and even then, there is more at play here. We're talking about two totally different loudspeakers that do not perform identically. The SVS Ultra doesn't even have a waveguide, while the M106 does... They have totally different transducers and totally different crossovers. Do you legitimately believe that someone (especially a trained listener) would be unable to reliably tell the two apart via unsighted or sighted listening? That was not a rhetorical question. Considering that you own Revel loudspeakers, I'd imagine your loudspeakers also feature a waveguide and I'd like to believe you know how that affects your perception of the sound emitted from the tweeters (regardless of how similar these two loudspeakers may measure to one another).

For what feels like the millionth time, what followed were assertions that, as a trained listener, he could come to valid sighted, subjective opinions that others can’t.

This thread started with the OP asking if that is true, given that Olive said that speaker evaluation “must be done blind.” Since then, Amir has said that Olive’s study wasn’t up to a high enough standard, and has challenged the “must be done blind” conclusion.

This new hypothesis, that trained listeners can come to valid sighted conclusions, necessarily requires evidence. It also prompts the question as to what training counts. (We seem to have come down on How to Listen, since that’s what Amir used.)

This is all important for consistency and for evaluating claims. What’s the proper way to respond if someone says “I’ve excelled in my How to Listen training, and ___ sounds better than ___”? Is that now a claim that should not be met with the “your listening wasn’t blind” rejoinder? These are honest, important questions.

 

[Rusty Shackleford]

That data is given so for people who believe that score, SVS has been given a huge gift, equating their designs with Revel.

The research however, was published as a conference paper so not peer reviewed. No published report has duplicated it. Nor is Harman using it itself.

I think directionally it is a good measure seeing how my subjective listening tests agree with it most of the time even though I don't know that score when I am testing the speaker.

BTW, it is the height of silliness to think a preference score for a speaker can have two decimal places! The tool that gathers preference score only allows single digits:
View attachment 77096

Just because computationally you arrive at fractions, doesn't mean you use them to rate speakers.

No way is a speaker preference score accurate to 1/10th let alone 1/100th as you are quoting. The error bars themselves easily invalidate such:
View attachment 77097

You have to understand the limits of what the measurements are showing in a difficult field of predicting listener preference for tonality of a speaker. Don't be blindly wedded to them.

Of course, you could prove us all wrong. Get those two speakers and perform a blind test and show that preference is identical between them, to two decimal places. Until then, Harman research shows that poor directivity is one of the top reasons for lesser preference in listening tests and SVS violates that key research. If it turns out this is indeed the case, then there is a lot that researchers need to answer for.

Fortunately I think the research is good and proper and the issue is limitations of the scoring system which according to its authors, was going to continue to assess other factors: A Multiple Regression Model for Predicting Loudspeaker Preference Using Objective Measurements: Part II - Development of the Model Sean E. Olive, AES Fellow

View attachment 77098

It is possible all of this was done and hence the reason the model published is not used. Perhaps they have a better model that they don't want to share with their competitors. Or they lost interest in developing this further, either because they found issues or didn't think it was important.

Either way, it is great that we have what we have. But don't trust it like a bible. We need to confirm what it is telling us and that is what I do as a listener.

To be clear, I am only citing the decimal places because they are in the ASR spreadsheet. Obviously, if we round, the two speakers would have even more similarity.

As I’ve said many times, I’m not necessarily insisting listening must be done blind. But given that you’ve cited Olive many times and that he has said “must be done blind,” it was a fair issue for the OP to raise. You yourself have previously stated the importance of blind testing, level marching, short time delays between each piece of equipment due to brief audio memories, etc.

If it is true that trained listeners can accurately evaluate speakers weeks apart and sighted, that’s a bold claim. It may be true! But if it’s the case, it’s worth specifying the parameters of that, in terms of the training and making sure that it applies equally to all. If the Audiophiliac excels at How to Listen, does that make his sighted subjective reviews now valid?

 

[amirm]

Much of your post certainly makes sense, especially in regards to donations. I've been out of work due to COVID since February (actually, from well before...it was a weird year). But I'll scrape up a donation as I want to see this site flourish.

Please, please don't feel obligated to donate. It is not expected and certainly not in your situation. I hope the core message of what I wrote is not misunderstood.

Many of you contribute to the forum with greats posts which often have more value than financial contribution.

It is when the contributions are negative that I find the equation wrong.

 

[Rusty Shackleford]

No, you're twisting. "If your goal is to inform consumers, that will necessarily impact businesses, given that they make their money from consumers." Consumers aren't prohibited from making their own informed or uninformed decisions. That is the joy in having a preference of your own.

What? The goal of providing information about gear is to inform consumer decisions, which necessarily impact businesses. This isn’t a value judgment. It’s a simple fact of how the marketplace works.

 

[Rusty Shackleford]

Please, please don't feel obligated to donate. It is not expected and certainly not in your situation. I hope the core message of what I wrote is not misunderstood.

Many of you contribute to the forum with greats posts which often have more value than financial contribution.

It is when the contributions are negative that I find the equation wrong.

Thank you for clarifying that. Sincerely.

 

[Rusty Shackleford]

Much of your post certainly makes sense, especially in regards to donations. I've been out of work due to COVID since February (actually, from well before...it was a weird year). But I'll scrape up a donation as I want to see this site flourish.

Here’s sincerely hoping your situation improves soon, Matt. COVID is hurting a lot of people, physically and financially, due to no fault of their own.