That's the simple answer. If you want me to elaborate, I'll do too.
If you are talking about distortion per voltage, there is no upper limit. Higher the impedance, lower the distortion. If you directly connecte the output to audio analyzer, that's what you get, usually tens to hundreds of kohms.
If you are talking about distortion per mW, that's a bit of different story. But firstly, none of the headphones on the market has high enough impedance and low enough sensitivity to trigger the problem. However, the maximum power output is limited by the voltage times the current. If you increase the impedance too much, there isn't much current, hence not enough power is generated. So if you makes the amplifier clipping, sure you will get much distortion. Except that, distortion will be very very low.
One more thing, lower impedance = heavier load, higher impedance = lighter load.