So true. Using the best driver unit for the given "job", is definitely the best way to go. As you build your system, sit down and figure out what you really need, and if it ain't more expensive or troublesome for you, please overdo it a bit, since that leaves headroom for fun and EQ.
I changed my SB23NRX - which are not bad at all - to WO24P drivers, and they fit much better in my closed design and a bit higher cross-over point + lower distortion, is always going to clear things up - to a certain degree of-course.
Second was going from KEF R3 to R3 Meta - definitely easier to lower the cross-over for the tweeter, when distortion in the Meta is way lower.
When I change something now, I always conduct a small blind-test with one of my friends, and if it can't be pointed out after level-match and EQ, with the listening partner "blind" - then it's usually not worth it, and effort should be put elsewhere.
Both the woofer and coax change/upgrade - was easily chosen 10/10 of the times, when level matched and EQ'd within around 1/5dB.
Every time we did this same test with amplifiers (YBA, D'agostino, NAD, Groundsound, Behringer... list goes on...) we never reach that precision in choosing one amplifier being better than the other - which fits very well with the more scientific approach to HIFI here on ASR.