The noise floor of the amplifier is fixed and depends on its gain.
Even with no input, the input referred noise will be multiplied by the amplifier's inherent gain.
Thus, an amplifier with higher gain will always have higher noise floor (more noise at output) than that with a lower gain. This is counting with zero signal input, i.e. attenuator at maximum attenuation.
There are ways to reduce input referred noise, for example by running higher bias currents (which reduces input stage transconductance and creates other issues), but only to an extent. You can't avoid the laws of physics. This is before we even consider the noise sources within an amplifier and yes, every component generates noise when current passes through it. This is unavoidable and they add up.
For those interested, Bob Cordell has a (very technical) explanation of noise sources in an amplifier in his design handbook. There are no known ways of removing noise once added to a signal. Noise can be divided into three basic types - voltage, current and thermal. This noise is inherent to the amplifier and power supply design, and cannot be removed even with your volume control at zero position.