Note a constant group delay means every frequency is delayed equally so the effect is the same as moving the speaker. A constant 1 ms delay is about the same as moving a speaker about 1 foot (13.524 inches using 1127 ft/s for the speed of sound). What matters is the difference in group delay over frequency, which results from the phase changing non-linearly over frequency. That is, the phase is not a straight line with constant slope. The yellow highlighted section in the post above by @NTK makes that clear (compare low to high frequency delay to check audibility). How audible the delay difference depends upon frequency since longer wavelengths (lower frequencies) mean more delay can be tolerated without significantly changing the waveform we hear. Also note that, if the delay results in some multiple of the wavelength, it will not be heard for a steady-state signal, since one wave is the same as another when delayed by a full cycle. The speaker's crossover design, or DSP (room correction), can help correct nonlinear phase and achieve (near) constant group delay.