Just discovered a handy free* alternative. My Dragonfly Red + headphone produce a serious amount of electrical noise on the PC USB, but it all goes away if I plug the USB DAC into the monitor USB hub... should have thought of trying it sooner.
* - requires monitor with built in USB hub
That indicates issues with the PC's actual USB power itself, as your issue couldn't be a ground loop if it occurs with headphones. Well either that or perhaps RF getting into the Red's analog circuitry, possibly from built-in WiFi or even USB 3.x peripherals. You can try using a USB extension cable assuming you have one - if it does nothing or makes matters even worse it's power, if it helps it's RF.
If you can narrow things down to USB power, I would check everything that could give you a bad ground connection, from an ATX connector that's not seated properly (or the same on the opposite side of a modular PSU) to poor contact from PSU to case to motherboard (with everything being painted and case panels getting thinner and thinner these days, I wouldn't necessarily trust that... with my current build, I whipped out my multimeter and checked what it had to say on the matter). The board itself can be problematic if it's just a cheap 4-layer job. A complete rundown of components starting from board, case and PSU may give more clues. PC issues can be hard to troubleshoot as they may only occur in a specific
combination of components, with none of them being at fault individually per se.