"Most audiophiles" have been tested and their preferences very much match trained listeners. Here is a formal test of trained listeners versus many other groups of population including audio reviewers:
View attachment 51283
As you see, the ranking of the speakers (each colored graph) stays the same no matter which group is listening.
I have taken the test twice myself and both times I voted the same as the majority/trained listeners.
You just need to put aside your intuition on this front and go with science and research.
Of course this is for blind tests. Make it sighted and many factors impact preference from story behind the speaker, to its cost, shape, etc.