The frailty of Sighted Listening Tests

[patate91]

Bu the way the wine experiment is a fun one.

Some friend, good food. You arrange a degustation : sighted and blind. Laugh garantied.

 

[patate91]

Cool!

https://daily.sevenfifty.com/unraveling-the-master-sommelier-exam-cheating-scandal/

 

[Dale Gribble]

There's many examples in this thread of ' the frailty of forums as communication platforms ' especially in complex subject matter . Much is lost much is added that should not be and serves as a distraction.

i think of ASR as amirm's garden , he's opened it up to the public, the locals enjoy it , some plant and help weed , some help with the buying of gardening tools , there's some open discussion on planting but the landscaping is mostly set . Some come and don't like it , maybe they mention why , maybe they leave and find another or make their own .

its plain bad manors to enter the garden , find Amirm and bash him about the head over and over again in some vain attempt to vanquish your displeasure , now if one had planted some boarders , helped weed or otherwise helped the garden be enjoyed by others you'd accept some critique, BUT just random strangers who wonder in ? Why's that acceptable? And to do so persistently?

Reading though this thread I find @Rusty Shackleford had been unnecessarily persistent and adversarial in his pursuit of Amirm , it's become nothing more that a argument to win and whatever the original point of contention that may of held some merit well that has been totally undermined. I'd like to think this is a example of ' the frailty of forums as communication mediums for complex subject matter' , things become adversarial and combative, rather than any personal false on his behalf.

Rather like subjective listening bias I think we are all vulnerable to the limits of this medium, yes @amirm included . There's so many things that underpin social interactions that's are missing on these forums , it would be amazing and rather unnatural if it didn't effect us .

Let's all keep that in mind before we go after eachother, create wild conspiracy theories or otherwise condemn and harass one another. Let's try and respect the garden and the guy who made it as well as those that help maintain it for the benefit of everyone.

Debates about facts are, by their nature, “adversarial.” I’ve said nothing vulgar or personal in this entire exchange.

My first apparent sin is speculating about why you changed the title of this thread, which was copied directly from the Olive article. If it was simply that you didn’t like the term, not that you were sparing Amir, I withdraw that speculation and apologize. However, I stand by my criticism of the decision to change the title, whatever the motivation. My second apparent sin is citing Amir’s posts from 2009. However, it’s absurd to think there’s something wrong with quoting an adult’s public writing! Amir has had these debates with many people, including me, across many years.

In terms of tone, Amir has referred to others as “Joe Blow” and repeatedly condescended to other posters. As I’ve said before, I’m confident any disinterested party wandering in wouldn’t find anything wrong with my posts.

If your view is that this is Amir’s “garden,” so he is above being challenged on his claims, then this site really should be, as someone above noted, Amir’s Subjective Opinions, not Audio Science Review.

If by “help the garden” and “buy tools,” you mean donate money or gear, I think that’s silly. Not everyone has the funds to donate or would choose to send to a retired MS executive rather than, say, a food bank. But if that’s the case, just be transparent and make posting ability contingent on donation.

 

[whazzup]

Debates about facts are, by their nature, “adversarial.” I’ve said nothing vulgar or personal in this entire exchange.

My first apparent sin is speculating about why you changed the title of this thread, which was copied directly from the Olive article. If it was simply that you didn’t like the term, not that you were sparing Amir, I withdraw that speculation and apologize. However, I stand by my criticism of the decision to change the title, whatever the motivation. My second apparent sin is citing Amir’s posts from 2009. However, it’s absurd to think there’s something wrong with quoting an adult’s public writing! Amir has had these debates with many people, including me, across many years.

In terms of tone, Amir has referred to others as “Joe Blow” and repeatedly condescended to other posters. As I’ve said before, I’m confident any disinterested party wandering in wouldn’t find anything wrong with my posts.

If your view is that this is Amir’s “garden,” so he is above being challenged on his claims, then this site really should be, as someone above noted, Amir’s Subjective Opinions, not Audio Science Review.

If by “help the garden” and “buy tools,” you mean donate money or gear, I think that’s silly. Not everyone has the funds to donate or would choose to send to a retired MS executive rather than, say, a food bank. But if that’s the case, just be transparent and make posting ability contingent on donation.

FWIW, I do agree that your comments and citation of Amir didn’t seem out of line.

With reference to the thread topic, I don’t think anyone disagrees that sighted testing will bring some bias to the table. That said, anyone arguing that every test must be done blind is just being naive.

Anyone citing research papers seem to forget that in order to carry out said research, requires research funding. No one is providing those resources to Amir to do constant blind testing.

I suggest that the next person to throw shade on Amir's sighted testing, should also drive or fly down to his place and help him move speakers and connect cables so that he can sit and just do blind testing comfortably.

 

[MattHooper]

Debates about facts are, by their nature, “adversarial.” I’ve said nothing vulgar or personal in this entire exchange.

My first apparent sin is speculating about why you changed the title of this thread, which was copied directly from the Olive article. If it was simply that you didn’t like the term, not that you were sparing Amir, I withdraw that speculation and apologize. However, I stand by my criticism of the decision to change the title, whatever the motivation. My second apparent sin is citing Amir’s posts from 2009. However, it’s absurd to think there’s something wrong with quoting an adult’s public writing! Amir has had these debates with many people, including me, across many years.

In terms of tone, Amir has referred to others as “Joe Blow” and repeatedly condescended to other posters. As I’ve said before, I’m confident any disinterested party wandering in wouldn’t find anything wrong with my posts.

If your view is that this is Amir’s “garden,” so he is above being challenged on his claims, then this site really should be, as someone above noted, Amir’s Subjective Opinions, not Audio Science Review.

If by “help the garden” and “buy tools,” you mean donate money or gear, I think that’s silly. Not everyone has the funds to donate or would choose to send to a retired MS executive rather than, say, a food bank. But if that’s the case, just be transparent and make posting ability contingent on donation.

From my point of view, your initial posts did seem fairly antagonistic, with a bit of score-settling tone. However, you went on to make what I believe are reasonable and fair points within this thread.

At this point though I'm unsure of the ultimate point you wish to make. I can see why you've criticized, for instance, some of Amir's statements about being a trained listener in relation to sighted bias. But ultimately, can you summarize what this means for your view of the sighted report portion of Amir's speaker tests?

Earlier in the thread I summarized how I synthesize some of the criticisms in to a view of accepting Amir's subjective speaker reports, in a way that does not seem hypocritical. But I'm not sure of where you come down on this. Are you saying to be strict the subjective reports ought to go?
Or that if they stay, it can only be undermining the remit of Amir's stance on audioscience? Too inconsistent to let stand?

Thanks for any clarification.

 

[Thomas savage]

Debates about facts are, by their nature, “adversarial.” I’ve said nothing vulgar or personal in this entire exchange.

My first apparent sin is speculating about why you changed the title of this thread, which was copied directly from the Olive article. If it was simply that you didn’t like the term, not that you were sparing Amir, I withdraw that speculation and apologize. However, I stand by my criticism of the decision to change the title, whatever the motivation. My second apparent sin is citing Amir’s posts from 2009. However, it’s absurd to think there’s something wrong with quoting an adult’s public writing! Amir has had these debates with many people, including me, across many years.

In terms of tone, Amir has referred to others as “Joe Blow” and repeatedly condescended to other posters. As I’ve said before, I’m confident any disinterested party wandering in wouldn’t find anything wrong with my posts.

If your view is that this is Amir’s “garden,” so he is above being challenged on his claims, then this site really should be, as someone above noted, Amir’s Subjective Opinions, not Audio Science Review.

If by “help the garden” and “buy tools,” you mean donate money or gear, I think that’s silly. Not everyone has the funds to donate or would choose to send to a retired MS executive rather than, say, a food bank. But if that’s the case, just be transparent and make posting ability contingent on donation.

This is disappointing but I will let others judge as is my privilege.

You must know you are here at my discretion , you're allowed participation is also at my discretion. It's nothing to do with Amirm , all moderating and punitive measures are born on me and are discharge by me and me alone.

 

[Dale Gribble]

This is disappointing but I will let others judge as is my privilege.

You must know you are here at my discretion , you're allowed participation is also at my discretion. It's nothing to do with Amirm , all moderating and punitive measures are born on me and are discharge by me and me alone.

Very sincerely, can you point to any stated ASR policy I’ve violated?

 

[HooStat]

Very sincerely, can you point to any stated ASR policy I’ve violated?

I think you missed the point. He is just asking you to give it a rest. Your points have been made and have been read by anyone who cares.

 

[Thomas savage]

Very sincerely, can you point to any stated ASR policy I’ve violated?

You have violated your own good time !

And btw I make the policy, it's nothing to do with Amirm, if anyone has a problem them must pay that on my door .

Cheers

And
.
I agree with your actual point, before you went too far and harassed a guy that has better things to do than indulge you. We are trying to go beyond ourselves here!

Thank you for your salient points .

 

[Dale Gribble]

From my point of view, your initial posts did seem fairly antagonistic, with a bit of score-settling tone. However, you went on to make what I believe are reasonable and fair points within this thread.

At this point though I'm unsure of the ultimate point you wish to make. I can see why you've criticized, for instance, some of Amir's statements about being a trained listener in relation to sighted bias. But ultimately, can you summarize what this means for your view of the sighted report portion of Amir's speaker tests?

Earlier in the thread I summarized how I synthesize some of the criticisms in to a view of accepting Amir's subjective speaker reports, in a way that does not seem hypocritical. But I'm not sure of where you come down on this. Are you saying to be strict the subjective reports ought to go?
Or that if they stay, it can only be undermining the remit of Amir's stance on audioscience? Too inconsistent to let stand?

Thanks for any clarification.

I’ll try to state my position simply:

1) This forum has been very critical of subjective reviewing, especially with sighted listening, and even more especially with sighted comparisons done after a long time lag. Given that, I think it’s fair to know what the accepted standards are now. Is it that it’s acceptable if the listener is trained? If so, what’s the training? I’m not trying to push for or against any particular position. I’m just trying to understand the position and make sure that it’s applied equally to all at ASR and beyond.

2) This forum is focused on informing consumers, thereby impacting purchasing decisions. Companies viewed to have performed poorly have been mercilessly criticized and mocked. The defense of that was that such conclusions were based on measurements, not opinions. That may or may not have been a valid defense, but it was a clear one. However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. Despite having remarkably similar preference scores, one would come away from reading ASR thinking that the the M106 (5.79) is a good, recommended speaker and the SVS (5.70) is a poor, not recommended speaker. Now, that very well might be the case! Maybe the preference scoring model is invalid, despite its good lineage. Maybe listening, even sighted listening weeks apart, is superior to the preference score at evaluating speakers. I just think we should be clear about the implications of that. Do we think blind tests or, at least, level-matched listening with both speakers being present at the same time, playing the same music, is necessary to come to a critical comparison, or not? I do think that, if they’re going to trump the data, the subjective reviewers should be longer, level-matched, and done against a consistent “good” reference speaker. But that’s just my view.

With influence — either over the epistemological contours of acceptable reviewing or over readers’ purchasing decisions and, thereby, companies’ business — comes responsibility. That’s my point, as simply put as I can make it.

 

[Thomas savage]

I’ll try to state my position simply:

1) This forum has been very critical of subjective reviewing, especially with sighted listening, and even more especially with sighted comparisons done after a long time lag. Given that, I think it’s fair to know what the accepted standards are now. Is it that it’s acceptable if the listener is trained? If so, what’s the training? I’m not trying to push for or against any particular position. I’m just trying to understand the position and make sure that it’s applied equally to all at ASR and beyond.

2) This forum is focused on informing consumers, thereby impacting purchasing decisions. Companies viewed to have performed poorly have been mercilessly criticized and mocked. The defense of that was that such conclusions were based on measurements, not opinions. That may or may not have been a valid defense, but it was a clear one. However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. Despite having remarkably similar preference scores, one would come away from reading ASR thinking that the the M106 (5.79) and SVS (5.70) is a good, recommended speaker and the latter is a poor, not recommended speaker. Now, that very well might be the case! Maybe the preference scoring model is poor, despite its good lineage. Maybe listening, even sighted listening weeks apart, is superior to the preference score at evaluating speakers. I just think we should be clear about the implications of that. Do we think blind tests or, at least, level-matched listening with both speakers being present at the same time, playing the same music, is necessary to come to a critical comparison, or not? I do think that, if they’re going to trump the data, the subjective reviewers should be longer, level-matched, and done against a consistent “good” reference speaker. But that’s just my view.

With influence — either over the epistemological contours of acceptable reviewing or over readers’ purchasing decisions and, thereby, companies’ business — comes responsibility. That’s my point, as simply put as I can make it.

You are not banned so why sign-up under a different profile.

 

[bluefuzz]

Debates about facts are, by their nature, “adversarial.”

Exactly. Facts are facts. Continuing to try and 'debate' them as you seem to want to is simply trolling.

Either you are putting your hand up to organise and conduct randomised, blinded trials to an arbitrary degree of scientific rigor of all the speakers Amir tests or you are just here to make trouble ... Which is it?

 

[Dale Gribble]

We're not banned so why sign-up under a different profile.

.

Huh? Where did I say I was banned?

 

[Thomas savage]

Huh? Where did I say I was banned?

So why the difference profile?

 

[MattHooper]

Exactly. Facts are facts. Continuing to try and 'debate' them as you seem to want to is simply trolling.

This is called: Begging The Question.

Either you are putting your hand up to organise and conduct randomised, blinded trials to an arbitrary degree of scientific rigor of all the speakers Amir tests or you are just here to make trouble ... Which is it?

This is called: a False Dichotomy.

 

[Dale Gribble]

So why the difference profile?

As I posted in the first post in this account earlier in this thread, I couldn’t log into my original account. I’d enter my password, and it would load indefinitely. I’m not sure why. I tried clearing my browser data, using a different browser, even a different device. I will try again to log into my original account at some point to see if the issue is resolved. But I never stated I was banned. Nor do I want a new account. I like my old one! Haha.

 

[MattHooper]

I’ll try to state my position simply:

1) This forum has been very critical of subjective reviewing, especially with sighted listening, and even more especially with sighted comparisons done after a long time lag. Given that, I think it’s fair to know what the accepted standards are now. Is it that it’s acceptable if the listener is trained? If so, what’s the training? I’m not trying to push for or against any particular position. I’m just trying to understand the position and make sure that it’s applied equally to all at ASR and beyond.

2) This forum is focused on informing consumers, thereby impacting purchasing decisions. Companies viewed to have performed poorly have been mercilessly criticized and mocked. The defense of that was that such conclusions were based on measurements, not opinions. That may or may not have been a valid defense, but it was a clear one. However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. Despite having remarkably similar preference scores, one would come away from reading ASR thinking that the the M106 (5.79) is a good, recommended speaker and the SVS (5.70) is a poor, not recommended speaker. Now, that very well might be the case! Maybe the preference scoring model is poor, despite its good lineage. Maybe listening, even sighted listening weeks apart, is superior to the preference score at evaluating speakers. I just think we should be clear about the implications of that. Do we think blind tests or, at least, level-matched listening with both speakers being present at the same time, playing the same music, is necessary to come to a critical comparison, or not? I do think that, if they’re going to trump the data, the subjective reviewers should be longer, level-matched, and done against a consistent “good” reference speaker. But that’s just my view.

With influence — either over the epistemological contours of acceptable reviewing or over readers’ purchasing decisions and, thereby, companies’ business — comes responsibility. That’s my point, as simply put as I can make it.

Ok thanks. With #1 you are pushing at similar issues I've raised here too regarding the status of sighted listening on ASR. It's not in an antagonistic way, it's just trying to hash through what are some messy borderlands which tend to occur when you start with a very stringent stance for making claims, but which inevitably bleed and blur in to the practicalities of real life. (E.g. strictly speaking all talk of how a speaker sounds from any individual's subjective impression would be stricken, given almost none of our listening and evaluations are done under blind conditions. On the other hand, that just seems draconian and impractical, and too severe given at least with speakers there are plausible differences to apprehend and describe. Which is why I settle on my Extraordinary Claims Require Extraordinary Evidence heuristic in provisionally accepting sighted evaluations).

As to #2, they seem to be valid (and non-hostile) issues to raise.

 

[richard12511]

I’ll try to state my position simply:

1) This forum has been very critical of subjective reviewing, especially with sighted listening, and even more especially with sighted comparisons done after a long time lag. Given that, I think it’s fair to know what the accepted standards are now. Is it that it’s acceptable if the listener is trained? If so, what’s the training? I’m not trying to push for or against any particular position. I’m just trying to understand the position and make sure that it’s applied equally to all at ASR and beyond.

2) This forum is focused on informing consumers, thereby impacting purchasing decisions. Companies viewed to have performed poorly have been mercilessly criticized and mocked. The defense of that was that such conclusions were based on measurements, not opinions. That may or may not have been a valid defense, but it was a clear one. However, if companies, particularly small ones like SVS, are now being criticized (and thereby having their business impacted) based on subjective evaluations, there should be some careful thought about that, IMHO. Despite having remarkably similar preference scores, one would come away from reading ASR thinking that the the M106 (5.79) is a good, recommended speaker and the SVS (5.70) is a poor, not recommended speaker. Now, that very well might be the case! Maybe the preference scoring model is invalid, despite its good lineage. Maybe listening, even sighted listening weeks apart, is superior to the preference score at evaluating speakers. I just think we should be clear about the implications of that. Do we think blind tests or, at least, level-matched listening with both speakers being present at the same time, playing the same music, is necessary to come to a critical comparison, or not? I do think that, if they’re going to trump the data, the subjective reviewers should be longer, level-matched, and done against a consistent “good” reference speaker. But that’s just my view.

With influence — either over the epistemological contours of acceptable reviewing or over readers’ purchasing decisions and, thereby, companies’ business — comes responsibility. That’s my point, as simply put as I can make it.

I think you make a good point here.

 

[LDKTA]

It’s simply untenable IMHO to dismiss such tests while at the same time allowing Amir to claim that, in sighted listening weeks apart, he can reliably differentiate between a 5.70 and 5.79 rated speaker.

Refrain from putting so much faith into the preference ratings/scores. They are not perfect by any means and they can easily be ignored -- I ignore them myself.

*Nearly through reading this thread, I noticed another member pointed this out as well.*

 

[samysound]

Refrain from putting so much faith into the preference ratings/scores. They are not perfect by any means and they can easily be ignored -- I ignore them myself.

*Nearly through reading this thread, I noticed another member pointed this out as well.*

This has actually been mentioned about the preference rating across multiple threads. I believe it was also mentioned that Harman themselves do not use this rating in their own speaker design/test process.