Everything is compromise and tradeoff. All of these are generalizations:
- Larger woofers are going to beam and have breakup modes at lower frequencies. necessitating lower crossover points. You are more likely to need a mid driver and/or HF waveguide
- Larger woofers and larger baffles generally make for a speaker harder to integrate into the decor
- Larger woofers can make for a larger speaker that costs more to store and ship
- Larger woofers tend to be more efficient, albeit optimization for lower frequency response eats at this
- Larger woofers do not necessarily negate the need for subs, especially for home theater work
- Smaller drivers can make up the difference by increasing xmax, but that has linearity and efficiency costs, as well as literally making the driver more expensive.
- You can run multiple smaller drivers to get the displacement, again more complex and expensive, but slimmer baffles may be worth it (see interior decor comment above.) It also looks more modern 'cool' to have a big wall of drivers, so this is a very popular design type for modern towers
- Use of multiple and/or xmax-optimized smaller-diameter drivers may require the same big cabinet volume as a bigger driver, albeit possibly in a more pleasing shape. Alternately phrased, cabinet size is largely a function of low-end extension and desired SPL, not driver diameter. Iron Law wins again
- Extremely generalized, surface area x wattage = Volume. Absolutely you can get desired frequency response from a smaller driver if you're willing to reduce the needed SPL (or get closer, see: headphones) You can also just absolutely dump wattage into smaller drivers if they're built for it, although the motor can end up getting so big that making the cone smaller becomes pointless. Stuff like 'punch' or 'chest thump' may require a minimum volume level, though.