Use this phrase to talk about something that you don't like:

I've tried sushi, but... it just isn't for me.

"It isn't for me" is polite but clearly expresses that you don't like the thing that you're talking about.

When the thing you're talking about is plural, say "they're not for me":

A: Why don't you get a pair of sunglasses?

B: Sunglasses? Nah, they're not for me.