In Rome the old buildings served as quarries for later buildings- especially Imperial buildings were demolished in the resources-starved medieval period. From the earliest period of Christianity in the Roman empire, the collapse of support for pagan religions meant the temples were neglected and started being pillaged for their decorative stone especially - the bulk of the coloured marble in Roman churches dates from imperial Rome (in the older churches in Rome, most of their often mismatched columns have been taken from older pagan temples) Ever noticed a lot of churches have large disks of coloured stone set into the floor or walls - sections sawn from pagan Roman columns. Marble was burnt to make lime for mortar. We know this was an on-going problem as the authorities from the 4th century onwards kept writing new laws to stop it from happening, as previous laws were clearly not working. So the ravages of time which brought the buildings of imperial Rome low had two legs & hammers, chisels and wheel barrows.

A few buildings were protected as holy sites - the Colosseum survived reasonably intact for a long time as it was a martyrs' shrine, but in the 15th century special permission was given by a Pope to his nephew to take a lot of stone from it to build the ginormous Palazzo della Cancelleria.

Moving large blocks of stone, especially on land, is really difficult in a pre-modern period. Most of the best marbles in Rome came from Asia Minor by boat in the Roman period. Once there was no longer the infrastructure - or the slaves - to move stone, people just grabbed it from the nearest empty building, the temple of whatever. Lots of the older, medieval buildings in the ghetto in Rome have architraves, bits of columns or statues embedded in them, and are probably built with stone & lime & rubble hacked from older imperial Roman structures.

Even Bernini's Four Rivers fountain in Piazza Novanta "re-uses" (that's the correct technical term) travertine blocks from the Roman Circus that used to occupy the site (you can see an excavated space at the North end of the piazza that has a section of the original Roman structure about 4m below current ground level)

It's a fascinating & complicated subject. Some buildings in the center of Rome incorporate stone from all major periods of history: medieval base with inserted Roman decoration, next floor a Renaissance logia, then Baroque window frames, all topped by an illegal 1970s cement terrace.