In simple terms, consider, what we would need to do to eliminate a room reflection at the listener's ear. We would send out an impulse from the speaker, and it would be picked up by the ear, followed by a delayed reflection of the impulse. We could eliminate the reflection by sending out an inverted version of the impulse from the speaker at the right moment. This would cancel the reflected impulse at the listener's ear. The listener would then receive an unwanted reflection of the inverted impulse. We would need to eliminate that with a further inverted inverted impulse, and so on, decaying to zero. BUT what if the reflection was louder at the ear than the direct sound? The inversion of the transfer function wouldn't work, because the result would be unstable and would not decay to zero, instead tending to infinite output amplitude.
BACCH has to do something like this with a ping-pong between the speakers. It couldn't work if the crosstalk at the wrong ear wasn't slightly attenuated compared to the direct.
Consider the output of the port as a 'reflection' that is louder than the direct signal from the cone, and you see that you can't invert the transfer function.