While damage is unlikely, things would potentially get distorted at a lower volume than you'd like. Taking the 3rd-gen 2i2 as an example, distortion gets concerning high at only 2-3 mW into 33 ohms:
This is a review and detailed measurements of the Focusrite Scarlett 2i2 Audio Interface (DAC/ADC). I was talked into buying it recently. The 2i2 costs US $140 including free Prime shipping so very reasonable for amount of functionality. For a budge device, the Scarlett 2i2 looks pretty good...
www.audiosciencereview.com
Mind you, around 1 mW would still be good for about 100 dB SPL on the X2HR, or if we're more conservative and going with 300 µW, 95 dB SPL. That would be adequate for
me being a quieter than average listener, but in the grand scheme of things - with headphone amplifier performance being what it is today - it's still
muy potato. You may well have an onboard output that does a lot better than that. Besides, headphone output common ground return resistance appears to be lackluster as Julian Krause found crosstalk not even reaching 40 dB with a 32 ohm load. This would make itself felt as kind of a stereo widening effect which tends to reduce bass impact (with bass generally being center-panned).
Also see edit above.