What’s your evidence that it’s audible?
Seriously? You have to ask about matching between channels?
Go read Toole's work. Just do it. Then learn about direct vs. indirect sensation, how that interacts with rooms, the effects of rapid phase shift (as opposed to slow phase shift), and how the ear works. Learn about how directional cues and room responses affect sensation.
Toole's work is limited as to mechanism, but the measurements are pretty much immaculate. That, alone, is absolute proof of my comments in regard to time/phase response and radiation pattern.
I'd think matching between speakers was a no-brainer, but apparently not. Just set your balance control off a bit and tell me that doesn't bother you, ok?
I'll also add various forms of distortion, from driver distortion to port noise to cone breakup to sudden variations in radiation pattern due to cone modes as other things that are testably audible, that you also discount.
No, magnitude response is not the only thing. Even with magnitude response (which is what you're calling frequency response) are you evaluating first onset, total soundfield response, or what? You haven't even specified what you think matters.
Is it direct nearfield, direct farfield, overall power response, or what that you believe in.