When you work in an office, you can call your workplace "the office" in casual conversation. You call it "the office" even when the person you're talking with doesn't work with you.

For example, you can say this to a friend:

I have to stop by the office for a few minutes, so I'll meet you at the restaurant at 7:30.

"The office" sounds more natural than "my office" for talking about your workplace.

Within the workplace, "your office" is the specific room that you work in (if you have one; some people work in cubicles or open rooms with other people):

Frank, I'd like to show you something. Can you come by my office?