My desktops and servers have been running (Debian) Linux since 2007. I also have a few embedded devices (orangepi & raspberrypi) running armbian, and I used to run routers on openwrt (no need to anymore as my ISP actually supplies decent kit & updates it...it runs Linux). I've been running USB DACs on Linux for about 5 years. I've used some synchronous and some asynchronous DACs & devices. Essentially everything worked out of the box as the Linux kernel includes all the necessary drivers for any USB standards compliant device. You just set up alsa so the default device is your hardware DAC & you have bitperfect audio. It was so easy! And recent versions of pulseaudio allow you to choose to avoid resampling so you can even run a desktop OS with a software mixer and *still* expect unmodified audio.
This is still the case if your audio source files are on disk. But something changed so that now if you stream audio and your audio is passed off to a USB DAC then you get crackles and pops. It is horrible. And it's the same on x86 hardware as on ARM. I have an OrangePi One and an OrangePi PC2 running as streamers. My raspberrypi is kind of retired but I try stuff out with it. It used to be so easy, then I upgraded from Debian Stretch and it all turned to crap. Same with my x86 server. My solution was to revert the orangepi devices to Debian Stretch with 4.19 kernels and the CPU frequency scaling set to a fixed frequency. These are devices dedicated to the one task of passing data to a DAC so it's acceptable to tweak them specifically to this. But it didn't used to be necessary. My x86 server performs multiple tasks and really needs the frequency scaling to work so I can no longer use it for audio.
I'll reiterate: everything is fine with playing back files, the problems arise when the device is required to handle a stream i.e. it's working as a UPnP renderer. It's not specific to raspberrypi but appears to be a more general problem. If this same problem exists outside the Debian & Debian derived environment I don't know but will be checking soon (by installing arch linux on a spare SSD and testing it).
Anyway, my advice to anyone wanting to run a raspberrypi or orangepi or even an x86 PC as an audio streamer is to go all the way back to Debian Stretch, disable the cpu frequency scaling and it will work fine. Luckily rendering audio requires almost no CPU power so you can set the CPU to a very low frequency, save power, keep it cool easily and everything is fine.