You can say this when you go somewhere, usually for a short time. 

It’s always closed when I go by.

I went by her office earlier to check on her.

Notice that this phrase is often used in the past tense.

Another meaning of “go by” means that you don’t stop at all. For example, while driving, you might “go by” an interesting sign on the road.

We went by a car dealership a few minutes ago.

Some guy went by blasting music on his bike.

For a very quick visit, you can also say “stop by,” which is similar.

I’ll stop by around four, if that works for you.