"Dragging someone down" means making them unhappy, angry, frustrated, unsuccessful, or other bad things.

People who are in a bad mood or have bad personality traits themselves sometimes "drag down" others. For example:

His constant complaining is starting to drag down the entire team.

Another way to use this phrase is in the expression "drag ___ down to (your/his/her/my) level".

You can't let them drag you down to their level.