So... the thing is, there are so so so many measurements with speakers that are often not measured. On-axis response and directivity are the lion's share, sure - but you also have to consider things like port noise, thermal compression, distortion (both harmonic and more importantly multitone distortion/IMD), phase rotation/time alignment, group delay, and so on and so forth.
So yeah, you can characterize speakers with measurements, but doing so exhaustively is a ton of work. On the surface two speakers may measure extremely similarly, but sound quite different from one another. Or, through a combination of various factors interacting, sound quite similar in spite of their differences.
As an example - people here love to crap on brands like ATC that have less controlled directivity - and in spite of their wildly differing measurements, the similarly sized ATCs (SCM50 ASLs) don't sound that different from my Neumanns. They both sound pretty much "neutral".