Talking to a person "directly" means that you talk to them, instead of to someone else.

You should tell him directly to stop calling you. I mean, he obviously doesn't know how to take a hint.

Saying things "directly" is usually a good thing in English. A more negative-sounding expression to describe speaking directly to someone is "to (his or her) face":

I'll tell him right to his face that he's being a complete ass.