You need a bit of knowledge to get the most out of a complex music, which sounds intimidating.
Or you can just go to the concert blind and take in all the emotions, but people don’t know that’s a perfectly valid way to enjoy classical music. (It is! go to the concerts and don’t be afraid to have an opinion)
It is tricky to get someone to recognise that they aren’t knowlegable enough. Even if you say it as gently as possible, some will still hear “you’re dumb” and no one likes that.
Also it’s a great tool to manipulate people : “I don’t need a scientist trying to explain me life from the depths of their lab !! I have commonsense !!”