That’s been my experience and conclusion too.
I would qualify that to say that “audiophile nervosa” is a real thing. Many audiophiles have experienced going through periods of messing so much with the gear that they find it hard to just settle down and enjoy the music for its sake.
But I have never met anybody permanently in that state. I don’t remember meeting an audiophile who couldn’t enjoy music.