So, different soundtracks vary in terms of their correlation with ... something. What is the something? Potentially, the something is the objective sound quality of the speaker. But the objective sound quality of a speaker is knowable only by way of subjective listening tests using listeners who have been "trained" to hear the qualities that are uh, believed to matter. When we identify a particular speaker as being of exceptional objective quality, what we truly mean is that listeners prefer that speaker with greater consistency than other speakers. It follows that the soundtracks that are deemed superior for doing speaker evaluation are the soundtracks that produce more consistent results in user preference testing.
It seems entirely legitimate to me with one possible caveat, which I doubt is a legitimate concern. If the listeners were trained to prefer a particular flavor of sound, then the soundtracks selected would be the soundtracks that best enable the listeners to detect that particular flavor of sound. While this is a theoretical concern, I am inclined to doubt that this sort of thing is taking place, because I don't think it likely that the trained listeners would have been trained to prefer a particular flavor of sound other than sound that is truly, objectively more accurate. I just wanted to point out that if it were true that listeners were trained to prefer a flavor of sound that did not equate entirely to accuracy, then the soundtracks that were selected through statistical means for conducting speaker evaluations would be soundtracks that best permit the listeners to identify the flavor of sound that they had been trained to prefer. For example, if the listeners had been trained to prefer speakers with exaggerated treble in the on-axis or first-arrival response, then the soundtracks that would be most strongly correlated with consistency in speaker preference would be soundtracks that best enable the trained listeners to identify speakers with exaggerated treble in the on-axis or first-arrival response.
To say it bluntly, Harman should not be using "trained" listeners in their speaker evaluations, and should not be endeavoring to train listeners. The same goes for anyone else doing this. While they perceive that there are good reasons for doing this, they are deceiving themselves. While there may be certain advantages of doing this, it utterly destroys the scientific legitimacy of the process. And it concerns me that they do not understand this. If they really were the true scientists that they represent themselves to be, they would be fully cognizant of this problem and that would have forbidden the use of listeners that had undergone any sort of training. What they should be using, and should have been using all along, is professional musicians, especially musicians that have been classically trained. There are very many thousands of them all over everywhere, so it isn't like there is any shortage that would have forced them to use trained listeners.
Consider the possibility that there is a subtle quality in speaker accuracy that the people who train the trained listeners have overlooked. Since the trained listeners would not have been trained to listen for it, the process by which soundtracks are selected would have the effect of eliminating soundtracks that are good for identifying that subtle quality.
This is my serious post on this topic. The next one will be my humorous post on this topic.
