Ok, let's get few things straight. Pretty much the only thing that introduces phase deviation in a passive speaker system is coming from the passive XO phase shift. Once you correct that phase shift with the FIR filter you will fix the looks of IR, but, as I already showed, the GD will remain pretty much the same.
You should be aware that, when EQ-ing the phase od the speaker, the target is not to flatten GD to zero but to flatten the excess phase graph to zero. Now, you could flatten the GD to zero as well, but that would introduce so much pre-ringing and that would, by all means be audible in a nasty way, while that typical rise in GD at low frequencies usually is not problematic at all.