This is a polite set phrase that's used at the end of a conversation. It suggests that the person you're talking to is busy and their time is valuable. You are using up their time by talking to them. So this phrase shows respect for that person, even though you may be the one who is actually busy and wants to leave.