When you're talking about something bad that might happen, the phrase "come to that" means for that bad situation to happen. People usually say that they "hope it doesn't come to that":

A: Man, I'm so hungry! If they don't get back with the food soon, I'm going to start searching through the couch cushions for old crumbs to eat.

B: (Laughing) I hope it doesn't come to that!

Another common phrase is "I doubt it will come to that", which means that you don't think something will happen:

A: If they aren't able to cool down the reactor, there could be a nuclear meltdown which would force tens of millions of people to evacuate.

B: I really doubt it will come to that.

The sentence "I hope it doesn't come to that" is also common.