It doesn't take more power to drive per se since the power sensitivity are usually comparable.
The answer gets long but here's a shortcut answer: Consider the Redbook 2Vrms output voltage coming out of your CD player or computer or whatever. This voltage is sufficient to drive most speakers to 80dB, which is plenty loud. Many preamps can output higher than 2Vrms too. So why aren't we directly connecting speakers straight to CD players or preamps? Because these equipment have a low current output limit, so the moment you connect the speakers and go above like 0.1V the current limit gets reached and the signal clips.
So speaker amplifiers, aside from generating around 10x to 100x the voltage, they also need to provide the current which is really their main thing, compared to preamps which only provides higher voltages. Thus, things that require higher currents out from speaker amps are usually called "harder to drive", since a speaker amp has no problem driving loads with higher impedances.
On the flipside, in the headphone realm there are also people who call high impedance headphones as "harder to drive", because typically headphone amps don't run into current limits while the voltage provided is usually plenty, however some headphones have high impedances which require higher voltages that typical headphone amps cannot output.
So in short, whether we call something "harder to drive" is based on whether it presents more work for the amplifier with respect to the amplifier's main job, all other things being equal.