I've been wanting for a long time to unload all my issues with classical music, but didn't really find a judgment-free platform to do it on.
No, the first major obstacle of classical music, is the culture of classical music. That concept encapsulates so many practices that are very alienating to anyone not coming from inside that culture.
It starts with how a lot of classical music listeners and players perceive themselves and the music they play. It's no coincidence that it's common to categorize anything that is not classical music as pop-music. There is something about the attitude of the classical music people that is condescending and self-elevating. They commonly consider their music as more sophisticated, complex, deep, human, etc, than the "popular" genres of music. Some consider classical music "universal", as if it captures something more real or basic about humanity, and thus every human can relate to it. From my point of view, classical music is merely older, that's it. The virtue of age does not make it any truer, in the same way that old religions aren't truer because of their antiquity, and old science practices aren’t more accurate because they precede the current ones. It is true that a large part of today's musical understanding and composing practices derives from what was learned by classical composers, but in the same time there are a lot of aspects to modern music that have no root in classical music. And on top of that, classical music is merely a western concept after all, and it's hardly universal to people outside Europe and the US.
Another strange aspect of the culture, is the listening habits. Listening at home is indeed not a very different affair from listening to any other genre, apart from the average lengthiness of the pieces. But listening to a performance is a whole different beast. You need to dress up whether you are a spectator of a performer, go to a huge concert hall and sit totally quietly for hours at a time, clapping at the appropriate moments. The performers themselves are dressed like they are about to get married, and usually play with all these weird unnatural mannerisms. Of course these concerts are usually quite pricey, and to fully enjoy the pieces one should be deeply familiar with them and maintain focus for the duration of the performance. Although from my experiences, apart from the very lively conductor or soloist on stage, the audience usually resembles more of a funeral crowd than one that is engaged with the music they are listening to. It's such a different experience from any other modern music listening experience, that it's very hard to accustom yourself to it if you are used to go to 'normal' pop and rock concerts. And I don't see anything deeper of more profound in that way of experiencing live music. Sitting stiffly and quietly in a chair seems a lot less profound than standing, moving, dancing, and physically engaging with the music you are hearing.