When I say adjust the subs, I mean that I take a sub measurement, a Left main measurement, a Right main measurement in REW and I see if the subwoofer line is crossing the mains FR at the right crossover target. I'm using MultiEQ-X to tell the AVR what crossover I want, yes. If the sub measurement line is either higher or lower than the target crossover, 100hz for example, I just make small volume adjustments on each of the subs and re-measure until the subs are crossing the LR at the crossover target.
I'm green at all this so let's keep that in mind.
Well this complicates comparing the different crossover alternatives quite a bit, as you are introducing other changes at the same time. And neither we nor you will know if your preference for one setting is due to the crossover or due to the overall bass level (which is significantly different between the different measurements).
I would suggest testing different crossover settings in the Denon without adjusting the gain if you want to listen to and/or measure which crossover that works best. Then AFTER determining that, you can adjust the gain to your preference.