I’m also a non-native speaker and I’ve also been taught to speak a certain way (“you and I are going” but “he saw you and me”; don’t split infinitives; don’t end sentences with prepositions, etc.), but then I read Steven Pinker’s The Language Instinct and - even more relevant here - The Sense of Style. We’ve been taught to use language a certain way, but our teachers were following the prescriptivist school of thought. You say these rules were written by native folk, but it’s often (if not usually) the native folk that say less when they “should” be saying fewer.

I know you said it’s only mildly infuriating to you, but if proper use of language is something dear to your heart (as it is to mine) - I really recommend the above books as I think this is something not worth to get even mildly infuriated about. The border between less and fewer is fuzzier than you think and - in the words of Pinker - once you really master the distinction - that’s one fewer thing for you to worry about.

Edit: typo