You need to differentiate between independent variables and dependent variables. Your reasoning would be correct if our existence relied upon the existence of earlier civilizations, but it doesn't.

Imagine a bunch of civilizations that all formed independently, across a large span of time. If they all assumed that they evolved at the time in the age of the universe where a civilization is most likely to evolve, they would have a better chance of being correct than any other assumption.

This is why we should assume we are at the peak of the distribution.

You might find that your reasoning is more relevant when it comes to the doomsday paradox.