My analysis -
It's ALWAYS been somewhat snobbish/elitist, and audiophiles have always been a bit NUTS. In the early days it was an expensive luxury. The costs came down and the equipment got better the 1970's (mostly because of solid state electronics) and the studio recording equipment became very good. The "kids" were buying good stereos but the "audiophiles" were still nuts.
I once posted a question on HydrogenAudio to Arnold B. Krueger (inventor of the ABX box), asking him if he expected so much controversy. He said yes. So he knew all of those golden-ears were full-of-it and I guess he knew they would never admit it. I don't understand the controversy because ABX seems perfectly logical to me.
I think what really killed the hobby was the introduction of the CD which "solved" noise, distortion, and frequency response. Most of the normal people who just love good sound stopped buying audio magazines (and stopped constantly upgrading equipment) and they lost interest in the "hobby", while still enjoying good sound. That left the hobby dominated by the old vinyl & tube guys.
The same thing happened to photography as a hobby. An iPhone is better than most film cameras and you don't have to deal with the film (or the associated delay) and digital photos can be digitally edited, etc.
...I never considered myself to be a "full-audiophile" because I've always been cost-conscious. But I was a snob when it came to lossy compression. I'd heard some lousy-sounding MP3s (maybe first on a "cheap" DJ setup) and I didn't believe they could be "CD quality". But now, I realize that most of the time with a good MP3, there is no audible difference, or you have to listen very carefully to hear a difference. Plus, some of the BEST sounding music I own is on concert DVDs with 5.1 channel Dolby AC3 or DTS (both lossy formats).