The phrase "It's time for you to leave" sounds quite straightforward and aggressive. You would say this when you're angry at someone and want them to go away. The same goes for "It's time for me to leave."

If you don't want to sound angry, here are some other phrases to use:

It's time for me to get going.

It's about time for you to leave, isn't it?

Note that adding "I think" doesn't make it sound less angry. It sounds a little outwardly polite, but still quite angry if you say:

I think it's time for you to leave.