To add on to it, same reason why being hit in the hand with a baseball line drive will probably break your hand, but you're fine swinging the bat.

The force is distributed over a larger area - less force is applied on any given area, so it does less damage

The force is spread over a longer period of time so the amount of force at any given time is going to be lower.

Similarly with a gun, the recoil is going to be in the neighborhood of 0.45m/s. The bullet is going to be traveling at 370m/s. The bullet is going to apply all of its force hundreds of times faster than the gun, on a dramatically smaller surface area.