I think you will find +20dB at 20hz to be out of the ordinary.
GD in small sealed subwoofers can be kept at inaudible levels even with a HPF.
Discussions of the audibility of phase shift are a grey area still with inconclusive conclusions from the studies that have been carried out so far. Anecdotally and subjectively it's very audible in my experience.
Regardless of the audibility of phase shift in itself, the group delay from a HPF can still be having an affect around the crossover point making it hard to integrate around the crossover point.