"Start something" is a slang phrase for starting an argument or a fight:

He came up to me and tried to start something. I was like, "Whatever!". I just ignored him.

If someone is acting in a way that seems like it might cause an argument, then you can say that that person is "trying to start something". For example, family members might have a conversation like this:

A: What are you doing? You're doing that all wrong.

B: Are you trying to start something?

A: No, I'm just saying, that's not the right way to do it.

B: I don't want to hear it!