Use this phrase when someone has done something that they shouldn't have done:

It was wrong of me to doubt you.

It was wrong of him to say that.

Most often, we use this phrase when someone has hurt another person's feelings.

You wouldn't use "It was wrong of ___" if someone broke a rule or law. For example, it would sound strange to use this phrase to talk about a man who hit his wife.