Three things really.

First, these buildings just happened to be built in areas where it is both easy and desirable to build things, which means if you want to build something better or more efficient, you have to remove the old buildings. That decrepit temple dedicated to a god you don't worship anymore is fair game when you need to build administrative offices or a new palace.

Second, maintenance becomes a problem. It might cost more to maintain a building than than the people who surround the area can afford. Moreover, natural disasters happen, large earthquakes tended to be hell on these large stone and brick buildings. Once the roof caves in or the walls fall, the cost of rebuilding would be absolutely not worth it, doubly if you're interested in rebuilding it to the original specifications.

Third, the building material were valuable. If an earthquake knocked the walls down there was little interest on just holding on to those pieces of what is now rubble, the owner might sell it off or people might just come and take the stones and bricks for use in other construction. Sometimes it happens because of normal decay of society, war might come and the government might flee, people will start picking apart the buildings to rebuild their houses because no one is there to stop them. Speaking of war, the material used also happens to be useful for things like walls and fortifications. Military leaders found it easier to use locally sourced material from existing and often decaying buildings than to get their own through a quarry or making their own bricks.

Last, as culturally valuable objects, invaders often would raze these sites to punish a population for resisting or to eliminate their culture. In the 19th and 20th century as artillery and later aerial bombing became more powerful though somewhat inaccurate, things just happened to get hit, or defenders used them for storage of war supplies which made them a target (the Parthenon). Related to that some culturally significant objects were military fortifications and buildings that were destroyed in the normal events of war. Large extravagant gates and walls would have been legitimate targets for destruction, and once captured it was sometimes prudent to eliminate these walls and structures so they couldn't be reused as to deny the enemy their future use. In many cases they were rebuilt but they were often rebuilt in ways that didn't resemble the original, because warfare changes and the design of walls and buildings also have to change. So it wouldn't make sense to restore it and it's original way.

Furthermore sometimes buildings are used and modified throughout their history, which begs the question what point of history do you restore that building to? It's absolute original? Or maybe the way it looked at a key point in history? Or maybe restoring it to a point where we absolutely know what it looked like? Any one of those choices destroys one history to preserve another. It's not something we can decide.