My old Quad ELS57s were inefficient and really needed the 2x45 watt of the only amplifier that you could safely use with them (the Quad 303). My current Quad 2805s are even less efficient, and really need the 2x140 watt of my Quad 606-2 (and cannot cope with much more). My old 11 Ohm LS3/5a desktop speakers were inefficient and could not handle much power. My modern Harbeth P3ESRs are also inefficient, but at least they can handle quite a bit of power (about 100 watt), even though they are still not suitable for largish rooms. So the dilemma still exists, but fortunately to a lesser extent.
As for listening levels, I think that at least in the Netherlands we often have much larger living rooms in which we listen, so that alone demands far more power. The sound proofing of our mostly modern housing stock has also improved enormously, so we can actually use the power.