I think you should have tested this unit whit sensitive IEM's. I was quite dissapointed to discover this much noise. These devices are not commonly used with full size headphones.
I agree. The change will be small. I would get one of those Topping DAC's to go with the amp. For example https://www.audiosciencereview.com/forum/index.php?threads/topping-d10s-usb-dac-and-bridge-review.14859/