But what we are trying to achieve here is to give the cone various properties. Weight, stiffness, break-up modes etc, depending on what the driver is to be used for.
Two drivers made of different materials but with similar properties, operating within their passband, will not sound "like paper" or "like aluminum".
The subwoofer/bass drivers we use are aluminium, the midbass / midrange drivers are paper, and the tweeters in one of our coaxials is a silk dome, while the other is a polymer compression driver.
The reason behind this isn't because I've selected drivers with these specific materials because of their musicality or inherent sound, but because they happened to be the materials the driver manufacturer found to be suitable for the parameters I needed for a particular driver.
So that's what you've got reversed in my opinion. It isn't inherently about the materials, it's about the properties we are trying to achieve with the drivers, and then selecting materials (treated or coated such and such) to achieve that. Again with your 4" midrange driver example, yes they sounded different, but not specifically (and certainly not exclusively) due to the material.