I see, this is truly interesting to me, that there seems to be people like me that once they hear the distortion they can't unhear it thus the enjoyment is limited by constant awareness of "unclean" signal, and yet there are people who find tube-amps/vinyl pleasing and contributing to enjoyment. It's a recurring theme from my observations.
I totally get that!
So if you are trying to understand why someone may have a different experience:
I actually gave a go at weaning myself off the tube amp colorations (at least of my preamp) by buying the Benchmark LA4. Whenever I A/B'd the sound (level matched) I could hear a bit of distortion stripped away by the more accurate LA4. I lived with just the LA4 for months, really enjoying it, and then I put the tube preamp back in wondering if I'd now be sort of allergic to the distortion (like you suggest). But, no, I just found "wow, no wonder I have enjoyed this preamp in my system for so long." It just added some things I really like.
I think that one reason I don't end up with the reaction you do is that I'm not seeking "pure accuracy" per se, but something that can at least remind me of the real thing, even if imperfectly. So for instance, I just came back today from doing what I often do: listening to some live street musicians playing - stand up bass, banjo, trumpet, tenor sax. As I always do I closed my eyes to question "what are the characteristics I'm noticing most, and like most about what I'm hearing in the live music." And what always strikes me is a sense of size, density, warmth of timbre, roundness and richness, clear, yet "relaxed."
Against that criteria, I find all systems colored to one degree or another. Even the most neutral systems depart from reality in a way that I find an overall coloration, and once I've heard a system for a little while I know how it will sound with practically everything.
So then it becomes for me a "pick the coloration you can live with" and it turns out I keep tending towards a slight tube coloration. Many of those characteristics I observe in live music are precisely the aspects I find added by some tube distortion (subtly). Example: Yesterday I listened to some of my favorite old R&B recordings with the LA4 and what I hear is a very pristine signal, it sounds like a really excellent recording. But with the tube pre-amp in the chain, I can hear a little "burr" of distortion added, but it's sonic effects, to my ear, are that the horns become more solid and dense sounding, a bit more tonally forward, more filled out and round, like real objects moving air, and there is a slight exaggeration of presence giving a sense of "it's right there" clarity, yet at the same time a bit more "relaxed" and "timbrally warm" sounding. All these aspects mimic what I tend to enjoy in real instruments and voices. (And this is all a *subtle* coloration, not some hit you over the head stuff, but for me even a subtle nudge towards what I like is subjectively significant).
Anyway, I hope that helps explain an alternative perspective. Back to the Audiophile Mansion. (And, boy I'd get a kick out of hearing his stuff. It's even just interesting to me to hear what people like, and why. I have friends who do reviews, especially one good pal, and we don't always like the same gear, but it's so interesting to hear it nonetheless).