I switch back and forth a lot; the main problem is that I live in an apartment, so while I have competent speaker setups in multiple rooms, outside certain hours I just can't use them, and in general I'm a lot more aware of loudness with the speakers (I have a good relationship with my neighbors, and want to keep it that way). So for focused listening, or even just particularly dynamic/soundtrack-heavy movies, I wind up with headphones on at some point.
Plus, travel means headphones, although that's more about silencing the outside world than SQ; I'm basically married to my 1000XM3s when I leave the house.