When you see the phrase "the four of us", you should be able to tell that:

• There were 4 people in the group that went to the restaurant, including the writer.

• The writer already mentioned how many people were in the group in an earlier sentence.

You can tell #2 because this phrase uses "the". English speakers use "the" when they think their audience will know what they're talking about.