If something bothers you, worries you, frustrates you, angers you, and so on, then you can also say that it "gets to you".

"Don't let ___ get to you" is a common piece of advice. It means that you shouldn't get too upset. Tell your friends or family members this when they have some kind of problem and seem to be in a bad mood about it.