So what is the subwoofer supposed to be in phase with, the active drivers, or the ports/PRs? It can't be in phase with both!
You've raised an interesting point. For starters, the subwoofer's low-pass acoustic response needs to blend in with the high-pass acoustic response of the main speaker system. The latter is made up of the driver's acoustic response as well as the acoustic response of the port or passive radiator.
As the wavelengths of sound at typical port/passive-radiator resonance frequencies are very long, the combined driver and port/passive-radiator acoustic output can be treated to a good approximation as a single source of acoustic output (a monopole). Of course, there are more complex low-frequency systems (e.g. dipoles), but for this example, we won't delve into those.
If the filtered acoustic outputs of the subwoofer and the main speakers are designed to follow the 4th-order Linkwitz–Riley response shapes, then the subwoofer and main speakers will be in-phase with each other. This is particularly important through the crossover transition region, probably for about an octave above and below the –6dB crossover frequency. The high-pass and low-pass shapes are what is called complementary, as they sum to unity magnitude.
The phase shift of the subwoofer at the crossover point is also affected by the inherent phase shift of its own low-frequency roll-off characteristics. Hence, the response of a typical subwoofer can only be approximately complementary to that of the high-pass filtered main speakers. However, the Linkwitz–Riley filter topology is relatively tolerant of phase shift errors, so any blending issues are usually well and truly hidden by the typical effects of room modes. Note that the phase shift difference between the subwoofer and the main speakers can usually be adjusted using the phase shift control (not the polarity switch) that some subwoofers provide.
It is of course possible to create a well-blended subwoofer and main speaker response using other types of filters. For example, the subwoofer could have a 3rd-order Butterworth low-pass filter applied to it, and the main speakers could have a 3rd-order Butterworth high-pass filter applied. The resulting acoustic outputs will be approximately complementary. However, 3rd-order Butterworth filters are sensitive to phase errors around their crossover point. Hence, achieving a smooth blended result, even theoretically and without including the effects room modes, is not entirely straightforward. There are interactions occurring that can be difficult to minimize, and that means that the summed response through the crossover region and either side of it can be a bit lumpy (not entirely flat).