My personal fix for this "problem" ; forget about it.
Only my opinion. IMO audiophiles spend way too much time and energy thinking about things which, in the overall cosmic scheme of things, just aren't important. I used to obsess about stuff like this myself until it occurred to me that it was ruining my enjoyment of my system and the music.
Once I read a story about a guy who was in bed with a woman who was so beautiful that all he could think of was how envious his friends would be if they could see him at that moment. As a result he couldn't just get into it. I'm not sure what this has to do with anything but it seems somehow related.