This is an expression that you use when you want to say that something (like a "person") doesn't have very much of a quality or characteristic (like "niceness"):

not that nice of a person

The meaning of this is similar to:

not a very nice person

But "not that ___ of a ___" is more casual.