5V will lower the noise floor compared to 4V, but if you find your volume control is very limited due to excess gain (like even on your quietest tracks you are only using like 1/3 of your volume range), you may consider 4V just to help improve volume control granularity. Generally speaking, better to use 5V if your system is fine with it.
If you're using amps with adjustable gain, set them for the lowest gain and use the highest voltage at the DAC (5V) to achieve the lowest noise floor. If the gain is limited or no adjustment on your amps and very high (+26dB or more), you may find it better to use the 4V setting just to get finer volume control.