I'll try one more time. Read the notes on the Piston Excursion calculator. Try a free software like WinISD, take a smaller driver and a larger one and model a simple closed box. Take a look at drivers TS parameters just to see how many of them there are and that drivers are different, period. Observe how different will a small and large driver behave in different plots that the software has to offer. Then imagine a real situation where you put them in a box and in a room. They will sound different. Maybe similar in that 80 Hz, 80 db sinewave, but lower in frequency and higher in SPL, for the smaller driver you'll need a ported box, and/or DSP, if it can handle additional power. Ported box is another can of worms in terms of driver behavior and frequency response. DSP will basically demand more excursion and more power for the given frequency, so if the smaller driver can handle it, ok, but if not, distortion would sky rocket.
Yes, you could probably level match smaller and larger driver at a single frequency in the same room and hear no difference. Then lower the frequency just a bit and already the smaller driver will be more quiet, so you'd have to give it more power to level match again. What's the point?