"Branching out" means doing something new and different.

For example, if an actor has mostly made serious dramatic movies in the past, but now he's starring in comedies, you can say that he's "branching out". You can also say:

He's starting to branch out into comedy.

You can "branch out" in your studies, in your work, in the kind of food you eat, and in other ways:

I've always just done aerobics classes at the gym, but I thought, "Hey, I should branch out and do some other stuff." So I've been going to the pool and stuff like that.