I agree.
But we'd be in the dark with many of the issues that have been surfacing were it not for the THD measurements, as well as CSD and individual driver and port FR plots. And we are still "missing" step-response, IMD and Iin-room FR plots. There are many aspects to speaker performance and the more comprehensive set of measurements will produce the most accurate "picture".
The Spinorama alone is insuficient to characterise performance.
In absolute terms I agree too that the spinorama is insufficient to give a
complete picture. But I do think on and off axis performance - I've complained several times about the lack of horizontal directivity information, for instance -- explains almost everything barring very audible distortion. Which in this case, I guess Amir heard, but I'd still look towards the frequency response and directivity before concluding it was distortion for sure (even though Amir is probably one of the most trained people at hearing distortions).
I still expect either EQ or extended testing with more listeners would show this speaker to be a good one. I mean, the speaker is a well-reviewed one otherwise.
On a separate note, you mentioned Genelec's slow listening paper earlier, and I do think there's merit to that. That said, in my personal experience, it mostly tends to support the frequency response and directivity stuff.
Other than Amir, I probably have some of the most experience correlating extensive on and off-axis measurements of various speakers to listening impressions around these parts. But unlike Amir I spend
at least a week listening to a speaker and taking notes before I measure it, in both mono and stereo,. Usually several weeks, sometimes over a month. I also capture and look at CSDs, step response, and driver distortion, but I've yet to find a speaker where any of those things clearly describes something I'm hearing that's not present in already in the spinorama+ detailed off-axis measurements.
(Note I'm not talking about the preference score here, but rather the raw data. There are certain qualities I know I like, such as a bit of extra energy around 2kHz as wide directivity).
Of course, I'm sure others would disagree. Plenty of speaker designers focus on minimizing distortion or improving time domain response, after all.
And granted, I don't have Amir's distortion training, and I suspect at the very high end smaller differences matter not present in the spin matter proportionally more. Kevin Voecks had this to say in a
sound and vision article where the Revel F208 (slightly) beat the Ultima Studio2 in a blind test at the company's speaker switching room:
As you know, many people argue about double-blind tests. Most of their arguments are without merit, but not all. One of the most important is that in my opinion and observation, it does indeed take extended listening sessions to hear the more subtle differences. The important thing is that these more subtle differences can indeed become more evident over time. Having listened to the Performa3 series and Ultima2 series both for very long periods of time, the difference at high frequencies especially is dramatic. The Ultima2 tweeter is so much "cleaner," with vastly lower distortion (even though the Performa3 distortion is far below most speakers) that it is much easier to listen to without fatigue. Combined with the advantages of low diffraction, it is the high frequency range that causes the Ultima2 series to win in long-term listening tests. Getting back to the blind testing, that kind of difference is best heard with longer sessions. There must be breaks between long sessions, as fatigue sets in, but that is where differences that audiophiles live for become apparent.
It's possible
@amirm is kind of 'fast-tracked' into hearing those differences in distortion that show up in extended listening that Voecks describes.