You usually use this phrase when you're just about to give someone bad news. It's often used when you are breaking up with a boyfriend or girlfriend, when telling your boss that you're quitting your job, and so in.

If you say "I have to talk to you" to someone, the most likely response will be "What's wrong?"