Could you please upload or link to the corresponding measurements?
You may want to lookup Erin's driver tests. While SB Acoustics or Purify perform so and so--see down below the IM graphs, the well renown Dynaudio generates the desired IM at moderate levels, despite being an 8" mid/woof. Especially the lower mids are enriched by IM at way lower levels of bass content.
In another table Doppler is considered, theshold at 20% or so, reached at 22mm of excursion, hence way beyond mechanical destruction. But, as far as I know, the Doppler is linear. To keep IM of Doppler type below 1%, which I would consider close to not bearable anymore, the excursion should not exceed 1mm. That's not that much.
Why is it, that people accept so much IM? Because the concept is a bit more complicated than HD, maybe.
Not all serious listening requires 96db levels . Serious stereo listeners ...
Sure! I not only recently got to listen to quite unusual musical content, though. Standard music, as often used for tests and then after to entertain guests isn't as sensitive to a flattening out of short spikes in amplitude with howling noises. Overprocessed as it is anyway. I don't speak the word of highend-ish blahblahblah. It is the feel of a vanishing loudspeaker not only in stereo-imagery, but in very aspect from white sheet tonality to rock solid dynamics, again all in every frequency range without any exception. Just no flaw to be unconciously excused. Of course it adjusts the focus on inherent draw-backs in the priciples of stereo recording/mixing itself ;-)
In a nutshell: remember non-linear distortion as a relevant workfield
Add.: an ethusiast should have a measuring microphone; measure the IM of a 80Hz tone with a simultanously played 1kHz tone; repeat with a 2kHz tone; comments are appreciated
The IM of specifically a coax, namely woofer cone moving, tweet radiating through it, as was mentioned here several times, isn't the thing, it's the IM itself everywhere ;-)