There are two meanings for this phrase. One meaning is to allow a person, like a prisoner, to leave. The other meaning of "let someone go" is similar to letting an object go. It means to emotionally accept that something is gone. This is similar to "moving on".

This second meaning is the one that's used in the example above. You can imagine that, in the novel the speaker is talking about, the character did something like deleting his ex-girlfriend's phone number from his phone.