My intention was mainly to provide a counterpoint to the implied idea that the configuration of the room used for the evaluations makes the existing loudspeaker preference research invalid, and to provide what I thought were appropriate references (which are also quoted in dr. Toole's book).
Well if I came across like I was saying the existing research is "invalid", then I communicated poorly.
Perhaps you and I view the research from different angles - I assume you're looking mainly at the preference scores and rankings, while I'm looking for clues about how to build a better loudspeaker. So you and I can look at the same study and arrive at VERY different conclusions! For instance, you cited a particular study as indicating that "the room has an insignificant effect on relative loudspeaker preference", and I get something very different from the SAME study.
In the study binaural recordings were made of each of the three loudspeakers in each of the four rooms so that listeners didn't need to actually be in the rooms. The three loudspeakers were SIMILAR to one another regarding directivity, so significantly different loudspeaker topologies were NOT being compared. When listeners were comparing speakers recorded in a given room, their loudspeaker preference rankings were consistent from room to room. BUT when the different rooms were randomly mixed into the evaluations (easy to do with binaural recordings), the ROOM is what dominated preference!
So, what is the contribution of "the room"? It is the reflections - their arrival times, arrival directions, net power contribution, spectral content, and decay characteristics. Obvously THOSE THINGS (or at least SOME of them) matter enormously as far as preference goes, because when the four different rooms were included in the evaluations, THE ROOM, not THE SPEAKER, is what dominated preference. Or to put a finer point on it, it was the LOUDSPEAKER/ROOM INTERACTION which dominated preference when the room was one of the variables.
So here is my take-away from that study, peering through my "looking for clues" lens: I see a theoretical "window of opportunity" IF we can figure out a way to get a loudspeaker to interact with whatever room it's in MORE LIKE the interaction which takes place in a really "good" room. In other words if we can transplant some of the most desirable loudspeaker/room interaction characteristics into an ordinary room, we probably will have made a worthwhile improvement.
Now let's return to the Harman Shuffler room: Presumably this is a VERY GOOD room, and presumably that LONG time delay before the sidewall reflections arrive tends towards REVEALING a speaker's qualities (otherwise they would have shuffled the speakers up against a side wall). Looking again through my little lens, I'm getting a clearer picture of what the aforementioned "window of opportunity"
might look like: It
might look like a speaker whose radiation pattern minimizes early sidewall reflections when used in stereo in a "normal" listening room!
But if I were to design such a speaker and sneak it into the Harman Shuffler room, I would not expect it to score well. It would have been deliberately optimized for a very different acoustic environment, and its loudspeaker/room interaction targets would probably NOT be compatible with its placement in the Shuffler room.