I thought it was odd that I don't seem to experience this issue on my Mac Studio so I upped the ram pressure and played some DSD128 and what do you know, problems there too.
I use Roon with 2 different endpoints and I can play DSD256 files upsampled to DSD512 without issues when using the endpoint that is running ALSA. While using the Mac for other stuff .
So the ram pressure issue has something to do with MacOS.
But Sonoma seems to have fixed that for the people that use DACs with usb from the Mac.
There are other things that help like increasing buffer size in soundsource , preventing the Mac from using powersaving mode (amphetamine) and I believe there are more issues with older xmos chips before the X316.