Generally the high impedance headphones are low sensitivity and low impedance headphones are higher sensitivity.
Sensitivity is how loud it plays when a certain voltage is applied (dB/V)
High sensitivity headphones are usually very easy to drive and require little voltage.
High sensitivity/high impedance headphones draw very little current, high sensitivity/low imp headphones draw a bit more current at the same output voltage level.
Most gear (incl. portable) can easily drive high sensitivity headphones, regardless of the impedance.
Low impedance, low sensitivity requires a higher voltage and higher current and thus requires a beefy amplifier.
Power rating determines how easily you can fry them (combined with efficiency = dB/mW)
How much power you need (in which impedance) is basically determined how loud you want the headphone to go (shortest peak power in music when momentarily listening to the song at an unusual high level.
For this you need to know the SPL you need to reach, how much EQ you need in the bass (depends on frequency response and target), sensitivity of the headphone and impedance of the headphone.
So even when one would normally only draw 1mW on average you may need 100mW, or even 1W to satisfy the above 'peak power' conditions.
There are no 'hard to drive' headphones. There are low sensitivity and low impedance headphones that require a LOT of power to reach loud peaks. But even these headphones, at normal listening levels require just 30mW or so.