There's a lot of things one might reasonably argue we shouldn't do so often.
What interests me about this thread is the wide range of meanings people get from the word, some of which are entirely incongruent.
When I try to be fair and detached about it I think that in common usage the word means to someone who practices a consumer hobby, usually in their own home. So I don't accept the "connoisseur of sound" although that may also be true. Pedantic recourse to etymology doesn't rescue the word.
On the other hand, I don't accept the implications of foolishness, pseudo-scientific beliefs, guru worship, and so on.
To me an audiophile is someone who is into fancy or high performance home audio reproduction gear more or less for its own sake. For example, when Amir wrote
Now that's what I call audiophile!
I want to distance myself from certain values that seem to me to be associated with audiophile practice but not from what performant gear can do for me or what I can usefully learn from the bodies of knowledge associated with audiophile commerce.