That's what I did back when I was single and/or healthy.
My man cave/home theater/office/whatever is in the basement. Unfortunately, it has rather low ceilings, and a post in the middle...ergo, setting up a home theater was challenging because the post forces a highly asymmetrical arrangement and there are lots of reflections to deal with. No matter how much treatment I put up, or EQ I applied (a couple of different room correction systems) it never sounded good enough to justify the cost.
I finally accepted the fact that no matter what gear I put down here in the "big system" it simply would never satisfy me. Thus I bought a pair of HD600s in 2006 and never looked back. I've always found that rather inexpensive USB DACs sound nearly the same as expensive ones. I was an early adopter of the E-MU 0404 USB, then the ODAC, tried the MF VDACs but always returned them LOL and now run a Topping D30. Very capable headamps have always been pretty inexpensive. I still use my CKKiii, a DIY built by one of the respected guys on head-fi, with super low output impedance, purchased for under $200. Bought a Magni, didn't care for the cheap volume pot (the CKKiii has an Alps Blue Velvet.) I rarely buy any new cans, as one can find lightly used ones on head-fi so inexpensively that I almost always let someone else take the hit on the purchase of new gear.
After cataract surgery, I ended up with two detached retinas and something called anomalous correspondence (my eyes send incongruent images to my brain so I don't have normal binocular vision and depth perception.) Adaptation to that has been a long process, which forced me into early retirement--another reason for a tighter budget. I don't bother to watch the 60" screen down here any more because it's actually less of a (literal) headache for me to watch them on my 24" PC monitor, and I can't be more than 30" or so from it because I use special glasses even for that. That's why I need something rather small that fits easily on a desk and works within a tight triangle.