@MattHooper I think you're wrong. Completely discarding and disregarding sighted listening test has nothing to do with the fact that they always lead to error. It's just that since reliability can't be established you throw them all out the window, yes, the proverbial baby with the birth water.
Some of your sighted listening tests may very well confirm the exact same thing that the blind test would tell you. But it's random and thus unreliable. It's like Amir said on YouTube on the subject of DBT; if you toss a coin and guess it's head, you can't conclude that you can predict tosses. So the fact that some of your sighted tests might be right is, sadly perhaps, not a reason not to throw them all out the window. Since you can't discern which sighted tests were the "correct" ones, the unbiased ones, you can't use any of them. That's why they're useless even when they're accurate. You can't make reliable statements and judgements based on them so you really have no use of them.
The right judgements in sighted test are, if nothing else, arbitrary.
Thanks killdozzer.
See my reply to qc3pma. I understand that. Please keep in mind I was specifically responding to what I saw as an obvious overreach by Newman, though, as an inference from that study.
Remember that we can't reason in a bubble, and that we ought to be able to see how the implications of a train of thought stretches in to the rest of what we do and believe, to make sure we remain coherent and practical. And that's what I'm talking about.
Everything you wrote could be applied to almost anything we do. For instance exchanging notes on a recipe we have been trying "
I found it worked adding salt; it helped dial down the sweetness of the X in the recipe" or whatever. There are all sorts of biases and confounding factors involved in such exchanges, same as in audio. But we don't therefore throw all such exchanges out as "unreliable" because "you never know if it was just an expectation effect.". It would actually be irrational, make life untenable, so it's reasonable to make these exchanges of "information" provisionally on the basis they are plausible. (E.g. adding salt really can alter the taste of food). And given we can seem to come to some agreement on the effects of altering the recipe. Strictly speaking we won't know we aren't fooling ourselves unless we started doing double blinded studies on the recipes. But as I said, that's untenable for much of life, which makes it still reasonable to use the "
extraordinary claims require extraordinary evidence" heuristic and provisionally accept subjectively-exchanged "information" provisionally.
So, again, it helps to stretch out the logic as far as possible to see if/where it breaks.
If "sighted" listening is unreliable, do we throw out all intersubjective exchanges of information that are not scientifically controlled?
Do we stop exchanging notes about how different albums are mastered? Do we stop exchanging notes on the very production sounds of albums? I work in film post sound - virtually all our communication involves conveying sound subjectively to one another, to say what we want and if we've achieved it or what needs to be fixed. If I'm to throw all that out as "UNRELIABLE" do we just fold the business...or how would you replace it? And how would you actually account for how well it actually seems to work to achieve the goals of a production?
See, unless you can stop this slippery slope somewhere you end up leaving much of life both incoherent, and actually unexplained.
The issue then is if you can say "
Well, ok, yeah lets not go too far, it's at least reasonable to use intersubjective reports to convey information in these cases..." My challenge will be then
"Ok, give me the basis on where you draw this line, when it becomes reasonable to accept information relayed by subjective descriptions." Could you provide an account that ISN'T some version of the basis I'm already claiming - e.g. that we are justified in provisionally accepting plausible claims. And if my account is generally correct in "how we get off this slippery slope" then it applies to speaker evaluation too. Like everything else, yeah there are variables that aren't being weeded out. But that doesn't mean we can't actually come to provisional "knowledge" claims based on intersubjective agreement. Especially if there is some level of experience that induces a level of trust in someone's reports.
For instance, I hear a lot of speakers at my friend's place who is an audio reviewer. Numerous times when he's sat me down and asked what I'm hearing, e.g. compared to the last speakers or to his own speakers, I'll describe the character and he will find I've described essentially the characteristics he hears as well. It often works in the reverse too (if I ask him to describe the sound, before telling him what I think). So I do find I have some justified level of confidence in his subjective inferences, as he has in mine. All without controls or measurements.
Is this a scientific level of confidence? Of course not. But it's the type of intersubjective confidence-building exchange (e.g. like food recipes) that we all accept as rational and warranted. (Even when, strictly speaking, we could of course be in error). Oh...but isn't this just another slippery slope? After all, people use intersubjective exchanges like this to come to believe in anything, from cults, to crazy religions, to new age nostrums etc. No. This is where the heuristic "
extraordinary claims require extraordinary evidence" cuts that slippery slope. So long as the claims are not in conflict with known empirical facts, and are plausible, we can accept them provisionally in this manner. Not if you are doing science! The "extraodinary claims require extraordinary evidence" isn't good enough if you are studying a new vaccine, or Floyd Toole looking for a higher level of confidence and predictability. But for much of life outside the lab, lots of empirical conclusions and life itself would be untenable without allowing a lower level of evidence for plausible claims.
(BTW, even if we are strictly talking about science, I also pointed out that Newman's leap of inference from a scientific study was not, itself, very scientific).
Cheers.