You can say this to end a telephone conversation. Say it when you've finished everything you wanted to say and when the other person isn't talking.

A: Well... I guess I should get back to work.

B: OK, I'll let you go then. It was great talking to you.

When you say "I'll let you go", it suggests that you feel like the other person is very busy, and that you're selfishly using up their time by talking to them for so long on the telephone. So "I'll let you go" is a polite-sounding phrase.

On the other hand, you wouldn't use this phrase with someone who's much higher in status than you, like someone who's interviewing you for a job.