Rusty/Dale, I agree with others that these are fair discussion points. I know a lot of us have similar questions, and I think it would be nice to take another step closer to coming to a more consensus understanding of them. In order to do that, my feeling is that we would need to establish a little more clarity and precision in the questions that are being proposed. For instance:
When you say "this forum has been critical of ____," what exactly is this referring to? Is it Amir being critical? Is it SOME people? Is it MOST people? Is it the forum software itself?
I THINK what you mean to say is: "SOME people on this forum have been critical of _____." Because I certainly am part of this forum, and I don't think I've been particularly critical of subjective reviewing, and I've observed others who take that position as well.
When you ask for "accepted standards," what term does that mean to you? Would these be written standards determined and posted by Amir? Would they be consensus standards formed by a group of independent experts who reviewed and summarized what we know from the literature? And how would you know they were accepted? Accepted on ASR? Would that be determined by a ratification vote?
What I THINK you want is for Amir to establish what HIS standard is regarding the validity of subjective listening impressions and to be consistent with it.
When you say "necessary," what does that mean? "Necessary" for you, specifically, to accept the results? "Necessary" for the results to be usable at all, based on published evidence? "Necessary" to know, without a shadow of a doubt, that the results are true?
I think a BETTER question might be: How reliable and valid are the results of a listening test when performed under specific conditions (i.e. blinded/sighted, trained/untrained, influenced by $$$, level matched, specific music tracks, etc.)?
I think that's a great question. In other words, are qualitative listening tests that report subjective impressions of the sound (under specific conditions) more reliable than the typical ASR member's interpretation of a set of spin graphs (or the calculated preference score, either one).
1) This forum has been very critical of subjective reviewing, especially with sighted listening, and even more especially with sighted comparisons done after a long time lag.
When you say "this forum has been critical of ____," what exactly is this referring to? Is it Amir being critical? Is it SOME people? Is it MOST people? Is it the forum software itself?
I THINK what you mean to say is: "SOME people on this forum have been critical of _____." Because I certainly am part of this forum, and I don't think I've been particularly critical of subjective reviewing, and I've observed others who take that position as well.
Given that, I think it’s fair to know what the accepted standards are now. Is it that it’s acceptable if the listener is trained? If so, what’s the training?
When you ask for "accepted standards," what term does that mean to you? Would these be written standards determined and posted by Amir? Would they be consensus standards formed by a group of independent experts who reviewed and summarized what we know from the literature? And how would you know they were accepted? Accepted on ASR? Would that be determined by a ratification vote?
What I THINK you want is for Amir to establish what HIS standard is regarding the validity of subjective listening impressions and to be consistent with it.
Do we think blind tests or, at least, level-matched listening with both speakers being present at the same time, playing the same music, is necessary to come to a critical comparison, or not?
When you say "necessary," what does that mean? "Necessary" for you, specifically, to accept the results? "Necessary" for the results to be usable at all, based on published evidence? "Necessary" to know, without a shadow of a doubt, that the results are true?
I think a BETTER question might be: How reliable and valid are the results of a listening test when performed under specific conditions (i.e. blinded/sighted, trained/untrained, influenced by $$$, level matched, specific music tracks, etc.)?
I do think that, if they’re going to trump the data, the subjective reviewers should be longer, level-matched, and done against a consistent “good” reference speaker. But that’s just my view.
I think that's a great question. In other words, are qualitative listening tests that report subjective impressions of the sound (under specific conditions) more reliable than the typical ASR member's interpretation of a set of spin graphs (or the calculated preference score, either one).
Last edited: