Biggest issue to me is 16 bit output will have a lot of residual noise unless you have an analog volume control. Assuming you are using a DAC with 2 V output, residual noise at the DAC output will be 30 uV, which will then be multiplied by amplifier gain.
A big advantage of 24 or 32 bit output to the DAC is that at lower levels digital volume control will attenuate the noise from your source and push it below the noise floor of your DAC + amplifier.
For example, take a 16 bit source and 32 bit output to the DAC. If you have a DAC with 112 dB dynamic range at 2 V output, coupled with an amplifier that has 100 dB dynamic range at 5 W in to 4 ohm and 25.6 dB gain, at -20 dB digital volume control position residual noise at the speaker terminals is 121 uV which is pretty good.
If you only have 16 bit output to DAC with the same system residual noise at the speaker terminals will be 591 uV at digital all volume control levels. In this system a lower noise DAC or amplifier would gain you almost nothing.
Michael