There are lots of clues. One of the simpler ones is counting seasons. I've only seen ice cores from the Arctic, which don't go as far back, but just looking at them you can tell summer ice from winter ice. Counting the layers give you years. There are also more accurate ways to measure the difference (you can look into what delta 18Oxygen means).

You will lose count if there was a period where summers got warm enough to melt more ice than was formed during the winter, but you can then use the other methods to try and correct for that.

Even so, being of by a 100 years isn't too bad when you are counting 100000.