have trouble (doing something)
Another way to say this is "It's hard to (do something)":
If I HAD to pick something, I'd say it's that I have trouble saying "no" to people's requests.
The version "have trouble ___ing" sounds a little bit more formal than "It's hard to ___".
Follow PhraseMix