I have the Topping Pre90/Ext90 analog preamp. It has the same behavior.
I'm using the balanced inputs, and when I select the RCA or XLR + RCA modes, output is 6 dB
hotter than XLR-only mode. I was expecting the Pre90 to do a better job of matching levels between modes. Failing that, I had imagined that gain in XLR-only mode would be greater, not less than modes that activate the RCA output.
I guess what it's doing internally is routing the full balanced voltage swing from the balanced input to a single-ended signal path when the mode enables the RCA output. After the signal passes through the volume control, the signal is routed to RCA outputs as-is but for the XLR outputs, the voltage is doubled, resulting in 6 dB higher gain. For XLR in, XLR out, there are no conversions to/from a single-ended signal path.
In short, I don't think there's anything wrong with your D90SE. Although the volume knob on your amp is higher when you're using XLR-only, this mode is giving you better signal-to-noise ratio out of the DAC. Don't get too hung up about volume knob position. Actually, depending on how the SA-1 implements volume, the results may be better at 1/2 o'clock than 10/11.