It does take some time for the movement to stop, but this duration does not exceed the time corresponding to the frequency being played. If it does, it reflects itself as a roll off in the frequency response.
The driver's response time doesn't affect its distortion directly, as long as the...