@dominikz, thank you for having this discussion with me. I think we have avoided getting into a squabble despite disagreeing, which in and of itself is something of an accomplishment. But beyond that you have challenged my thinking (in a good way!) and I appreciate that.
If I'm not mistaken, in both experiments all loudspeakers and all rooms were evaluated, it is just the context that was changed - in one case the experiment is designed so that the loudspeakers are compared and rooms kept constant per trial (this is the one where room-insensitivity of loudspeaker evaluations is suggested), and in the other the rooms are compared and loudspeakers kept constant per trial (this is where loudspeaker-insensitivity of room evaluations is suggested). The first experiment was done both live and with binaural recordings, while the second one was only done (and only practical) with binaural recordings.
Yes this is my understanding too. My impression is that listening in the actual room was done mainly to confirm the validity of using the binaural recordings for evaluations.
Of course, and I fully agree that people will prefer some rooms to others when listening to the same set of loudspeakers in each. But in my mind this is a separate question compared to whether relative loudspeaker preference evaluations are impacted by the room being used, all else being the same.
I understand that the paradigm of "the room doesn't change speaker preference rankings" has a very strong following here on ASR, but imo this study does not justify that conclusion because it did not evaluate a sufficient variety of speakers types (the evaluation did not include ANY variety at all in that regard!). In order for "the room doesn't change speaker preference rankings" to be a generally applicable principle, we would need a study which includes competent representation from a wide variety of loudspeaker types as well as a wide variety of room acoustic situations. (Note that in this context "the room" includes "how the speakers are positioned within the room", which is another arguably significant variable.) That study will never happen unless it can be done in the virtual domain because it would be prohibitively expensive, and it may still never happen because the two-channel stereo market is shrinking.
(What the available research DOES reliably demonstrate includes these basic principles: The first-arrival sound is the most important; the spectral balance of the reflections matters a great deal; the arrival times and intensities of the earliest reflections matter a great deal, the earliest in-room reflections involve tradeoffs between clarity and spaciousness; the arrival directions, particularly of the earliest reflections, matter; the in-room reflection field should be neither too strong nor too weak, and should decay evenly across the spectrum and neither too fast nor too slow; and the bass region modal behavior generally improves as room size increases.)
My paradigm is "a speaker should get two things right: the direct sound, and the reflected sound." And the reflected sound is dominated by the room interaction.
So here's a reasonably common real-world situation: Suppose a person is limited to placing speakers up against a wall. Would a speaker deliberately and competently designed for up-against-the-wall placement, but which did not score well when positioned out in the middle of the Harman Shuffler room, be a possible contender? If our paradigm is "the room doesn't change speaker preference rankings", we would dismiss the wall-placement-optimized speaker because it has already failed to score well in a room. If our paradigm is "a speaker must get two things right: the direct sound and the reflected sound, the latter being dominated by room interaction", then we would consider a competently-designed wall-placement-optimized speaker.
That indeed sounds like a worthy design objective! Unless I'm mistaken similar motivation was behind at least some of the NRC and Harman research.
Arguably the most all-out assault on "transplanting some of the most desirable loudspeaker/room interaction characteristics into an ordinary room" was the
Acoustic Research MGC-1 "Magic" loudspeaker of yesteryear. It failed in the marketplace, but imo for reasons unrelated to its basic concept (for one thing, it was too expensive for the performance it offered).
On a related note - I believe I remember reading that they also use(d) the same room for stereo blind evaluations (in which the side walls would be much closer to each loudspeaker), with similar results to the mono evaluations.
I agree that mono listening is better for evaluating loudspeaker
sound quality (and have been using mono listening for that purpose in my speaker design efforts for about forty years), but I do not agree that mono listening is better for evaluating loudspeaker
spatial quality.
And according to a study done by Wolfgang Klippel (yes,
THAT Wolfgang Klippel!) and cited by Floyd Toole starting on Page 185 of the 3rd edition of his book, sound quality and the "sense of space" both contribute equally to the perception of "naturalness", while the "sense of space" slightly
dominates the perception of "pleasantness".
By way of example, consider the Polk Audio
Stereo Dimensional Array (SDA) speakers, whose extraordinary spatial characteristics only exist in stereo operation. Imo the "rules of the game" should not be applied in a way that works against even ONE outlier speaker which advances some important aspect the art in a worthwhile way. If our priority is exceptional results, we should be inclusive towards exceptional solutions.
However I do see your intention - though I have to admit that personally I probably wouldn't make the same inference.
Thank you.
