Being "at your wits' end" means that you've tried really hard to solve a problem, but haven't been able to solve it. You say this when you're really frustrated and worried.

I'm at my wits' end with Gabby. She doesn't listen to me, doesn't do her homework, stays out late. What am I supposed to do with her?

This expression can be used in casual or formal situations.