Anyway, you need less voltage swing but more current for a low impedance load (headphone, in this case) and more voltage swing but less current for a high impedance load. It's Ohm's law in action: Power = current^2 multiplied by resistance (impedance), but also power = voltage^2 divided by resistance (impedance).
High impedance headphones need more voltage swing for the same power output, so higher voltage power supply rails in the amplifier are required to avoid clipping. Low impedance headphones need more current, so the headphone amp needs to have a lower output impedance and a robust enough power supply to sufficiently power the headphones (you can sort of think of output impedance as how good a headphone amp is at supplying current - lower output impedance = better current supply).
The other thing to factor in here is sensitivity. This is measured with a given power (or voltage) input. For example: Sennheiser HD650s are rated at 101dB with 1 milliwatt input at 1KHz. This means that at 1KHz, a 1 volt input will produce 103dB of sound pressure. This is the more determinant factor for how easy or hard headphones are to drive than the impedance is.
When people say headphones are "power hungry", really what they mean is that they're not sensitive. For example, HifiMan HE400i's have a sensitivity of 93dB - that means that at 1 milliwatt of power, they produce 93dB of sound pressure. Because the decibel is a logarithmic unit, that means that they need a great deal more power to reach the same sound pressure level as the HD650. Interestingly, at the same voltage input, the HE400i's are actually slightly more sensitive (105dB for the HD650s vs 107dB for the HE400i's). Once again, this is Ohm's law in action.
But, because the HE400i's are considerably lower impedance than the HD650s (40R vs 300R), they need much more current to reach that same power level.