When people (or things) are really annoying, you can say that they're "testing the limits of your patience".

It's like you're imagining that the annoying things are trying to be as annoying as possible, so that they can find out what will finally make you angry.