Agree that there us no hard and fast line, but getting back to a point I made earlier, most true audiophiles have an interest in the underlying physics and are equipped with enough skepticism to not throw money at improvements for which no physical/pyschoacoustical basis exists. Granted its easy to be confused by the engineering vs advertising game, and who here among us has never chased a spec that for all intents and purposes is meaningless, for instance SINAD numbers that are probably one to two orders of magnitude.more than what is audible? But I would also hazard to say that true audiophiles recognize that the numbers are likely meaningless, but opt for the better kit for peace of mind (as opposed to screaming you must be bloody deaf if you can't hear the glaring differences (of course never having done a blind ABX on him or herself). That there is just sheer arrogance.
Edit: another difference is most audiophools have never measured their systems despite the inexpensive and not too technically demanding equipment.
Talking about psychoacustic, this is one area in which we currently have no means of measuring it. Although we could use a litening test for comparison, it is still extremely difficult to quantify it.