When you say that someone "had to" do something, it means that someone made them do it or that they had a really strong reason to do it.

In the situation above, where Kazu cancelled his trip to visit you, everyone uses the phrase "had to" because it would seem rude to say that he cancelled the trip because he wanted to cancel it. So Kazu will usually say that he "had to" cancel, and when you're describing it, you'll also say that he "had to" cancel the trip.