The immune system attempts to clear asbestos from the lungs (but isn't able to very well) and in doing so creates a lot of damage. The process the body uses releases a lot of reactive oxygen species (which damage DNA).

Asbestos is present in the natural environment and in industrial uses and as a result we all have asbestos fibers in our lungs which cause damage, but a little damage is not enough to cause problems. It's those who have regular high exposure over many years in the occupational setting who will develop problems.