Not really, no. Keep in mind that a larger driver also means a bigger magnet and voice coil. F=ma. In theory you could probably build a large driver that was "slow" but it would have terrible frequency response and so it would be unusable. There have been some
previous threads on this topic that you might find useful.
But what it boils down is that most of the important time domain characteristics of a speaker/subwoofer are in fact contained in a frequency response measurement. If there are significant impulse response issues then they will show up there as well. You can't simplify down to "driver size".
There are some edge cases where "speed"(typically measured as group delay for subs) can be an audible problem but this doesn't have anything to do with driver size, rather it's a result of the cabinet design, port tuning, and sometimes use of DSP. Smaller ported subs(eg 12" or less) with tuning in the audible range(>20hz) can have excessive group delay near their tuning frequency. But even then there's very little evidence that this is easy to hear or a big problem.
What causes "slow bass" is almost always overly excited room modes. There is a myth that smaller subwoofers are "faster" because they just don't play as low and thus don't excite those low frequency modes as much. The real way to fix this problem is with room correction EQ and multi-sub systems.