The whole launch stack and rocket are hooked up to various plumbing and electrical systems on the pad. Disconnecting those properly and safely takes many hours.

An example of some data from LC39 https://www.nasa.gov/pdf/168440main_LC39-06.pdf

Then the whole tower launch platform has to be readied for moving.

The launch tower doesn't move that fast either, about 1mph max but with something like SLS they will stay lower for the sake of safety. It takes about 24 hours to move it from the VAB to the Launch pad. That distance from LC39B to the VAB is about 4 miles.

Another thing to consider is that rockets are basically massive water towers in that their supporting structure is on the outside and then inside is basically empty space. They are built to withstand vertical forces from launch first, and everything else second because the chances of them facing those other forces are muuuch lower. When its already costing billions to build it as is, it'd be insane to build it to withstand things that are lower on the chances.

I was trying to find a document on the whole process, which should be available to the public but finding it is another story.

I found the stuff below, which is mostly for marketing purposes so its really dumbed down which is unfortunate because it simplifies a lot of things.

https://www.nasa.gov/pdf/168440main_LC39-06.pdf

https://www.nasa.gov/sites/default/files/atoms/files/combined_crawler-transporters_fact_sheet_final.pdf

The two handbooks below have a ton of juicy technical details. https://www.nasa.gov/sites/default/files/atoms/files/nasa_systems_engineering_handbook_0.pdf

https://ntrs.nasa.gov/api/citations/19660024575/downloads/19660024575.pdf