So, then, any time you use a digital volume control that has steps other than 6.02dB steps (1 bit, exactly) you have irrevocably assured harm (which may be minimal in may cases, yes) to your original signal.
So I assume you propose here a volume control that does a bit shift..
But that means you have to throw away one or more LSBs, which can only be done properly by dithering.
So harm is done in both situations.
In that case I prefer to have finer digital steps and dithering, accepting a slightly higher noise floor.