Because a voltage is the difference of potential between 2 points. Ground and tip for RCA, hot and cold for XLR.
The gain of an amplifier is the factor applied to the input voltage to obtain the output voltage. Not to the potentials.
If you need different voltages depending of the type of the input to obtain the same power at the output, then different gains are applied depending of the type of the input.
So if gain RCA = gain XLR = 26dB (or so), which means I need 1.5V-2V or so output voltage from my source, then in theory the 4.1Vout RMS from the Oppo 105 XLR outputs is more than enough to drive a Hypex power amp, with the volume knob on the Oppo not needing to be more than half-way full, right? (wouldn't it be easier if manufacturers produced a Hypex-based integrated amp? )