The job that members of the military do is called "service". We say that they "are serving their country" because "service" is a kind of work that you do in order to help other people in your society:

It's important that we honor those who serve their country.

To specify which part of the military a person works in, use the phrase "serve in ___":

I served in the Self Defense Force for twelve years.

We also use the word "serve" to describe other jobs that help society, but some use "on" instead of "in":

He served on the New York Police force for over 30 years.