The expression "Man!" shows strong emotions, like shock, fear, admiration, envy, surprise, annoyance, etc..

For example:

Man, I am so exhausted.

Man! This computer is taking forever to start up!

Man! What a gorgeous sunset!

You can use "Man!" when speaking to both men and women, because you're not saying it to anyone. You just say it to show an emotion.

"Man" is casual, but it's completely polite. It's a lot more polite than other words with similar meaning, like "Shit!"