There are better and worse drivers, but ultimately they're just parts of a functional product.
Just like putting a spoiler on your car doesn't make it go faster (except in situations that spoilers are designed for), using a better driver in a bad speaker design won't make it sound better.
A better analogy - putting a bigger engine in the car should make it go faster. However, you also need the transmission, fuel system, brakes, wheels, and tires to keep up with the engine for the car to actually go faster.
In general speaker designers need to choose drivers for their strengths and work around their weaknesses to get the best overall result they can. The crossover and cabinet can all ruin the performance of a great driver if done incorrectly.
So ultimately, let's say that the quality of drivers defines the ceiling of performance ... there is no floor, if the implementation is bad enough.