As many have said, the idea of of driver 'speed' is a huge set of misconceptions, imnsho.
The audible quality often described as speed, is best described as transient response.
And
transient response is a function of full-range frequency response. (phase too, if cutting edge)
A subwoofer driver has very little discernable transient response on its own, it must be combined with the rest of the spectrum before its transient response can be evaluated.
Listen to your sub alone, particularly with its low pass filter in place if it is being crossed over to main speakers.
Download and play the ultimate transient impulse, a single sample full spectrum Dirac pulse
https://www.audiocheck.net/testtones_impulse.php
Not much to hear, huh?....
(make sure it is sub alone, or volume is low for full speaker...lots of HF energy.)
Ironically, the subs with the highest amount of THD will sound like they have greater transient response...
simply because
transient response (repeat after me Lol)
is a function or frequency response.
The mass of the driver's diaphragm, cone, ribbon, etc, has to be considered in combination with the strength of the motor driving it.
Alone, mass is immaterial. Power to weight rules.
Better subs do not strictly attempt to lower cone & coil mass....they optimize those materials with the motor strength for the intended SPL and low frequency extension..
Best 'speed'..... best transient response possible, ........comes from a complete full-range response, having both flat magnitude (freq response) and flat phase, where all contributing drivers are operating in their pistonic range. (no breakups producing higher freq response)
Sure a room can muck things up, but any speaker that meets the above 'best transient response possible criteria' will sound "fastest haha" and better in any room, than a speaker that doesn't meet it. (all extraneous variables being the same, e.g. radiation patterns, linear SPL capability, etc)