In theory, there's no difference between a headphone amplifier and a loudspeaker amplifier.
Both have to drive a certain number of volts into a defined impedance with low distortion.
However, loudspeakers have an impedance in the range of 3-16 ohms (with a few outliers) and with power requirements from a couple of watts to several hundred. Headphone amps tend to drive higher impedances, and with far greater sensitivity so need fewer volts. That makes the task of the headphone amp much easier, albeit with more emphasis on low noise.
A typical headphone amp consists of a 5532 op-amp, which provides typically somewhere around 8v (+20dBu) at vanishingly low distortion and inaudible noise into typical headphones. I know there's a lot of 'typical' in that statement, but I think it's justified for the majority of headphones and amplifiers. Where it perhaps falls down is for the very sensitive, where noise may become an issue, and the headbangers and specs merchants for whom loud is good and VERY LOUD is better.
I have several pairs of headphones, from AKG (75 ohm), KOSS (270 ohm) and a good few 'no-name' in ear types (20 ohm or so) that I use when travelling, and I've never found any of them a problem, so assume that the spread of output voltage, headphone impedance and sensitivity are all such that any headphone amplifier can cope adequately. I've never seen the need for a dedicated headphone amplifier, what comes out of my tablet pc or pre-amp has worked fine for me.
S.