As evidenced by the various responses above, the obvious answer is that the material makes a difference how a driver behaves, but not whether that behaviour is good or bad.
I’ve owned and built speakers with drivers ranging from paper, to magnesium, to papyrus, to aluminium ribbons, to electrostatically charged plastic, to ceramic, to diamond, to the intestines of a ritually slaughtered goat.
In spite of their claims at the time, and my varying levels of enthusiasm, none are or were intrinsically better than another.
Berylium is the flavour of the month for tweeters, for example. Yet SB Acoustics lowly SB26AC measures better than their lofty SB29 Be.
KEF and Genelec aside, none of the top-rated speakers here have exotic or unusual drivers. The Revel M106uses quite ordinary SB Acoustics drivers. But it uses them well. The Ascend Sierra has exotic drivers yet doesn’t use them well.
There is a thread that is no doubt tripling as I type about the Purifi kit speaker. Whether it is because of shipping damage, poor design, poor execution, or wrath of the audio gods, the drivers come across as over-priced crap.
Yet if you read the equally well-conducted review from Erin of the Selah Purezza which uses the same driver and an alternative non-dome tweeter, the very same drivers come across as the second coming.
It is all about execution.