In this case, "tell" doesn't mean "say". It means "decide" or "figure out". The phrase "I can't tell ___" is used to talk about something you're not sure about:

I can't tell whether it's finished or not. I can't tell whether she's being shy or just doesn't want to talk to me.

You can use "can't tell" with any question word like "how", "what", "who", and so on:

I couldn't tell what they were saying.

For the life of me, I can not tell how he did it.