It is much cheaper and easier to make a cabinet "quiet" if it is smaller. Any vibration in the walls is radiated as sound, good in a musical instrument, bad in a speaker.
I think it is educational to rest a finger gently on the main speaker cone whilst playing music then feel the cabinet then think about how much greater a radiating area the cabinet has than the cone area.
Big boxes give more bass but not necessarily a clean sound in the budget case. It is hard to separate the sound coming from the box from that coming from the drive units by measurement, though it can be calculated.
The speakers I like best all have a lot of effort put into the cabinet. (KEF LS50, Proac EBS, Yamaha NS1000, Goldmund Epilog)
My main speakers are modular with the midrange and tweeter in one cabinet and the bass in another, rather like bookshelf speakers and a sub.
I also, rather surprisingly, markedly preferred the ATC SCM50 on stands to its tower version SCM50T, both big speakers. No idea why but it was a clear preference, and their pro versions are all stand mount not tower type FWIW.