I believe this is a myth efficiently debunked in the Audioholic series on "Loudspeakers & Power Ratings", namely in
part III
Notice: The myth is that it is
easier to destroy a speaker that it is the
most common reason and that
it will kill your speaker really fast. It's not a myth that it can be done since it can, the myth refers only to the underline parts. We might add a rather new part of this myth that it is
barely audible clipping.
If you don't have time to read the entire series on Audioholics web site (and that would be a shame if you don't mind me saying), here's the gist of it that only concerns the myth above:
It is not easier as you'll have to withstand a prolonged unpleasant distortion and not turn the volume down. Also it will not destroy the speaker, only the tweeter.
It is not the most common reason for the same reasons as above - upper mids sound horrible, almost everyone turns the volume down. It is far easier to kill the speaker with too much power and this is a more common cause.
As I said in the first sentence; it is not killing your speaker, only the tweeter and it needs some time (and even more in modern ferrofluid cooled).
It is not a "silent killer" of your speakers as it has been made to look. A huge amount of power going to the tweeter as it does when clipping will really be audible.
So, it can be done, at the end of the day, but it has been really made into something that it is not. Another thing, underpowered is really circumstantial. Expecting a speaker rated 50w - 250w to give you 100dB with a 20w amp is kinda your fault. But if you want it to play up to 90dB, even with dips to 4 ohms, a 20w amp will do it for years since it need first few watts for that anyway.