Yeah, "high end audio" only makes sense if we're talking about price and not performance.
After all, when we're talking about "high end fashion" or "high end real estate" we're talking about price, not sweaters or houses that materially outperform other goods in objective ways -- we don't expect a $500 designer sweater to perform better than a $50 sweatshirt from Carhartt.
As for "high fidelity", that's (sort of) simple to me. The pursuit of tonally accurate reproduction of the source material, at low distortion levels, at sufficient SPL.
The actual measurements for those values (is 0.1% THD high fidelity? or 0.000001%?) can be endlessly argued over but, but high fidelity is more about the pursuit of getting the best fidelity possible rather than hitting some magic number and declaring "aha! I've achieved it!"
(It's a pursuit that's separate from the enjoyment of music. However, they pair extraordinarily well so no problem there!)