Just bumping an older thread to share some info.
I was wondering about the XLR vs RCA output on a R-2R DAC from Denafrips. The Venus II.
I could clearly hear a difference between the two connections but wondered what exactly was going on so I emailed the distributer. It turns out RCA only uses the positive signal from pin 3 of the XLR connection so you are only getting half the processing power of this 4 ladder DAC.
BUT - the reason for the sound quality difference on my 300B SET tube amplifier is that it uses a balanced input board (2 opamps) to merge the XLR signal back to single ended and it does sound significantly better that the half signal from RCA,
So the question that needs to be asked is both how does the DAC create the single ended signal and how does your amp translate the XLR signal (unless it is a fully balanced amplifier). Both need to be known before deciding if RCA may be equivalent to XLR in sound quality. But many times it will not be.
The RCA signal is the same signal as the XLR. Assuming you do not have massive noise, the balanced amp still cancels common mode noise you pick up on the signal and shield.
The amp will take the difference of the + and - signals. This means you get 2x signal and 0 common noise. That is, if your signal XLR is +A and -A, then the amp will get (+A - -A) = +2A as the signal. Any common noise N will be (N - N) = 0.
One reason you might hear a difference is XLR is you get 2x the voltage. it's just louder. If you level match your speakers (i.e. with a volt meter) between the balanced and unbalanced, the difference will likely go away.
The XLR and RCA might expect difference input impedances. Some units, when they take the RCA from pin 3 use a separate driver op amp for it, some just take it straight from there.