I haven’t tried the Denon without DSP. My room really needs correction. My filters were set up by Mitch Barnett. He’s amazing. But it becomes very expensive and complicated to implement in multichannel. So I’m exploring Dirac and top notch processor. Also, I think going from 5.0.2 to 5.2.2 with ART will make the bass much smoother. The TADs go way way deep and load, but the room causes problems
You should try a true like for like comparison without DSP in either system...
if you have two different DSP's comparisons become well nigh impossible.
The RME's are great DAC's (and ADC's) but whether that is actually an audible perceptible difference is quite a different matter.
Doing the comparison might make no difference (the heart wants what the heart wants) - but it would be helpful to you to know whether it is in fact a hardware platform issue (amenable to hardware platform upgrades) or in fact a software platform issue.
It will tell you where you should focus your efforts!
One of the Marantz afficionados on Audioholics went from the 7706 AVP to the new AV20 - according to hardware specs, there should have been nothing in it.... but he noted a substantial audible improvement.
Currently, his conclusion is not that the hardware is an improvement, but that the later AVP uses updated Dolby Atmos decoder... and the latest decoder updates have not been included in firmware updates for older hardware.... only the current generation is getting it.
So the improvements are down to software - and not even the DSP / RoomEQ (!) - but the decoder firmware.
It can be difficult to debug and work through - but we have a complex chain involving Source - Decoder - DSP - DAC - amp - speakers.
Having said that, there is something to be said for opting for premium solutions such as Audiocontrol or StormAudio, where there is a high level of confidence that all the updates will continue to be made available whether at decoder level or DSP/RoomEQ.
Even better is using PC software - but that gets tricky - eg: Netflix on the Smart TV provides a full Atmos feed.... Netflix on the PC nerfs it down to DD+ and 5.1 or 7.1... the Atmos decoder, doesn't do height channels - so you might purchase the Dolby Pro developer/mastering software suite... but Netflix won't feed it the full stream.... catch22... (and the limitation of the RME solution)
I can get a full Atmos feed using VLC player... but not using Netflix