Your "old man" is your father:

My old man used to tell me, "It's not whether you win or lose; it's how you play the game."

This is a casual phrase. It's also kind of an older expression, so young people these days may not use it very often.

Sometimes we use "old man" to talk about someone's husband, but that's not as common.