The decay is measure after the signal has stopped, correct ??
The main thing to understand is this:
If a cone can stop fast, it can also produce a high frequency. This is because frequency and acceleration (i.e. stopping fast) describe the same thing, but in different ways.
A steep line on the oscilloscope (i.e. "fast") has high frequency components. The steeper the line, the more high frequencies.
This is why people say there is no such thing as a "fast" woofer apart from its frequency response, since the frequency response tells you how fast the cone can (or can't) accelerate/decelerate. There are definitely fast or slow drivers, but this isn't a quality that ought to be measurable apart from FR.
What else might produce a longer-than-wanted decay time?
Resonance: This should show up in the frequency response plot, but to your point, is easier to see in the CSD plot.
Group delay: This shows up in the phase response plot.
If you're thinking about transient response, you're thinking about the impulse response or step response plot, right? These are actually just different ways to visualize the frequency + phase response plots, same with CSD - they actually all show the same thing, but presented in different ways.
Another way to think about it: How can you have a pure 20hz signal that lasts for less than 1/20 of a second? (hint: you can't, that's what 20hz means)
I think high THD might also create the impression of a slow woofer, because if you have resonances in the same system, you might end up exciting a resonance with a harmonic of the bass note you're actually trying to play, causing the note to ring longer than it should. All this stuff put together probably creates the impression of "slowness" in cheap / badly designed subs.