If it works for you,
great! But you are potentially "stressing" the headphone output so
it's not something I'd recommend to others...
You don't want to "match" the impedance. The output impedance should be low relative to the load and a 1-Ohm output impedance into 8-Ohms isn't too bad. 4-Ohm speakers is "pushing it" but the impedance might still be OK,
The real (potential) problem is - Impedance and resistance are
"the resistance to current flow." At 8-Ohms you are "pulling"4 times as much current as with 32-Ohms, or 8 times as much at 4-Ohms, and more than it's designed for.
Of course it sounds "clean"! It should sound clean with the amplifier too (as long as the amp isn't defective or over-driven) but
when you lower the volume you also lower the distortion so any distortion becomes harder (or impossible) to hear. You also won't hear all of the little-subtle musical details. An amplifier will amplify any existing noise, plus it will add SOME noise and distortion of its own. It doesn't necessarily audibly degrade the sound but the mythical "straight wire with gain" doesn't exist.
An Analogy - If you have a 16-bit file and you attenuate it by 48dB, you are only using 8-bits so you only have 8-bits of resolution. (This is why people argue against digital volume control.) But you don't hear any loss of resolution. You also lose resolution during quiet passages or during a fade-out. At the end of a fade-out you could be down to 1 or 2 bits but it's so quiet you can't hear ANYTHING so you can't hear the terrible resolution!
If you have 8-bits of usable resolution and you re-amplify it (digitally or electronically) you WILL hear the loss of resolution as quantization noise.