I think this answer, whilst partially correct, kind of misses the bigger picture. Medieval and Modern societies did have the resources for this kind of monumental building. maybe not as many but they certainly did. They just used those resources building things that were useful to those societies, namely churches and cathedrals. The later Middle Ages and Early Modern period saw a massive expansion in the construction of monumental structures like Notre Dame or Cologne Cathedral. As admired as the Romans were the were admired by Medieval and Early Modern people for their vast empire and Christian piety. Pagan temples and arenas for blood sports had largely gone out of fashion before Rome even fell, what purpose would later people have to maintain those ruins?

So whilst you are correct that later societies couldn't marshal the same resources for monumental constructions as the Romans could, it is not really the most significant factor in my estimation.