I’m using a Mac mini M1 with 16GB RAM and don’t have any playback issues. My current music player is Audirvana, but I’ve also used Roon in the past. The audio output device I’m using now is Dirac Virtual Audio Device, previously I’ve used Element with Blackhole and various DSP plugins (incl. Dirac Live Processor), or HLConvolver with Blackhole. Final output is to USB DACs.
My Mac is dedicated to music playback, and the CPU load is aways very low, as is the memory pressure. Even so, I have found a way of making the drop-out/stuttering issue appear, simply be reducing the buffer size in Dirac Live Processor. With all the devices mentioned above, when I set them up and configure sample rate and buffer size, I always set the buffer size to max, typically 2048 samples. This is probably why I never encountered the issue before. If I drop this to, say, 512 then the audio issues start popping up. They are always accompanied by “HALS_OverloadMessage” messages logged by coreaudiod, as seen in console.app.
There seems to be a low-level bug in this architecture, perhaps to do with interrupt scheduling, that causes some buffers to over- or underrun occasionally. I hope that Apple can identify and fix this. In the meantime, using 2048 sample buffer sizes seems to provide stability, at least it does for me. I suppose the Jitter Cleaner style devices do the same, by buffering USB data.