When the amplifier clips, whether due to a large bass signal or at some other frequency, harmonics "spray" across the entire audible bandwidth. That creates a more "harsh" sound that can be fatiguing across the audio spectrum (not just the bass).
When a speaker is overdriven several things can happen. The worst case is the cone or spider hitting its mechanical limits and buzzing or rattling. Before that distortion will rise due to several factors, including cone breakup or flexing that generates standing waves and thus frequency tones not in the music, thermal compression that limits the excursion and again causing distortion similar to an amplifier clipping (though usually "softer"), etc. Different speaker drivers react differently, so cones and horn diagraphams may sound different than ESL or planer-dynamic and ribbon drivers. Port noise can also occur, a "chuffing" or airy "woofing" sound from the port rather than the drivers themselves.
I am not sure I have a good way to distinguish speaker from amplifier clipping without instruments. Generally the harsher sounds are from amplifier clipping IME while speaker clipping tends to be not as obnoxious. The catch is that means upping the amplifier power may simply destroy the speaker if it cannot handle the extra power, often before the listener realizes it.
It may be worthwhile to remember that 1 dB is a very minor change in volume, about what most of us might do when asked to bump it "just a hair", and requires about 25% more power. A noticeable increase ("Turn it up a little, would you?") is about 3 dB and that takes twice the power. Doubling the perceived volume (in the midrange) takes about a 10 dB increase and that is ten times the power (!) If you are clipping your amplifier, chances are you need to more than double the power to prevent it. Many years ago studies showed peak-to-average power in music was around 17 dB, meaning you nee about 50 times the average power to handle the loudest peaks. If you use about 1 W at typical volume then you need 50 W on tap. 10 W, get a 500 W amplifier, and so forth. Modern recordings, at least some of them, exceed 20~25 dB, and some movies are pushing 30 dB (1000 times the average power). Chances are the lowest levels are below average, so hopefully you do not need a 1 kW (or 10 kW) amplifier.
Moving closer to the speakers (or moving them closer to you) helps a lot, and of course you can buy more sensitive speakers that do not require as much power to play loudly.
HTH - Don