This is an example of understatement. If you're talking about someone and say:

She's not exactly poor.

...it means that she actually has a lot of money. Instead of simply saying "She's rich", you might use this expression in order to sound witty and intelligent.

Here's another example: if your husband is overweight, but made fun of you for being a little heavy, you can say:

You're not exactly in the best shape yourself, you know.

"Not exactly" can be followed either by an adjective, or by a noun:

A: You're dumb.

B: You're not exactly a rocket scientist yourself.