This means "You don't really think that; you're only trying to be nice to me." Say this when someone compliments you, but you don't believe them or if you want to seem humble:

A: You're one of the smartest people I've ever worked with.

B: Whatever! You're just saying that.

A longer version of "You're just saying that" is:

You're just saying that to be nice.

You can also use this phrase in negative form and in questions:

A: You look great!

B: Really? You're not just saying that?