When you say a linear phase speaker, what does that mean? If you're talking about multiway speakers with noncoincident drivers (like your designs, as far as I know), phase can only ever be linear relative to one position with a speaker. A speaker designed for linear response on axis will not be linear elsewhere. If I'm wrong please correct me.
From what I know, under anechoic conditions, 60 degrees of phase shift within one critical band is considered audible. In rooms, I don't think your working hypothesis holds when you start adding the effects of reflections. The phase error in the loudspeaker would have to be extremely gross. John Vanderkooy did a study that showed little audibility for shifts adding up to several cycles. If you also account for the fusion interval for localization, which is measured in tens of milliseconds, and then phase locking for bass taking upwards of 200ms, phase doesn't really enter the picture other than when comparing an ideal situation to a typically measured one.
Regarding the topic at hand generally: a paper by Earl Vickers:
https://www.sfxmachine.com/docs/FixingThePhantomCenter.pdf