There are no barriers, or they could be easily overcome. The fact is that men and women are different, women (generally, on average) prefer to spend their time interacting with humans and not doing the 21st REW scan of a new speakers arrangements. These differences (men preference for dealing with things and women preference for human interactions) are natural, inborn, they can be detected with one week old babies - girls' eyes follow human face much more than say keys. The differences in brain activity can be detected even before birth, in fetuses. Female fetuses exposes to unnaturally high amount of testosterone (due to some mother problems) are much more likely to become engineers etc.This subjectivist and objectivist debate feels like an internecine war, which no-one outside the audiophile community cares about. We both sides have a lot more in common with each other than with non-audiophiles. There's an element like alcoholics anonymous when people change from one side to the other and become like tee-totallers in relation to their previous beliefs. Maybe there is also an emotional/religious element to the sides we gravitate towards as some of the most subjectivist people I've debated with are engineers with a formal education.
I don't think the audiophile community is intentionally sexist. It's just we have developed an extreme gender skew which creates additional barriers. There's a lot of similar communities like the knitting/sewing community which are almost completely female. That becomes a barrier itself for men who might be interested in knitting.