I have a hunch that most people don't think much about audiophiles or what it is, and don't have really strong opinions on what audiophiles do. That's probably more something audiophiles think about other audiophiles, in group stuff. That's why the goofy audiophile joke cartoons were found in audio magazines, rarely in more popular magazines.
So when it comes up at all among friends or acquaintances I have no problem saying I'm an audiophile. There's no "yeaagh!" reaction; they just don't have strong feelings either way, and I can elaborate on what that means if they are interested. Often enough it can start when a guest spots my speakers or my rack full of equipment, tube amps, big assed shiny turntable etc, and they express some surprise "wow, what's this stuff?" Then I briefly explain. If I just said "I love music" that wouldn't really explain the gear. They love music too, but they don't buy this kind of stuff! So I say I love music and I'm an audiophile as well so I enjoy geeking out on audio gear too, and this is the type of gear I find interesting. I never get the sense they are like "wow, this guy is crazy" but rather "that's cool, you have a hobby, and it's sort of interesting because I never knew about this kind of thing." And if it turns out they are up for a listen to something they get it even more.
It's all about how you communicate, IMO. If someone happens to have some negative connotation about the term, you can just explain how you approach the hobby.