It’s not about force being spread over a larger area.

There are two things here:

1. Momentum is conserved. Momentum is the mass of an object times it’s velocity. Velocity has a direction. The total momentum of gun+bullet before you shoot is zero, so it has to be zero after you shoot. So when you shoot the bullet speed times bullet mass in one direction has to equal the gun speed times the gun mass in the other direction. But the gun is much much heavier. To keep math easy we’ll say 100x heavier. So for the mass of the bullet times the velocity of the bullet(plus propellant) to have the same magnitude of momentum as the gun the bullet has to go one way 100x faster than the gun goes the other.

2. The kinetic energy is proportional to the mass times the velocity squared (1/2 mv^2). The bullet weighs 1/100 times that of the gun, but is going 100x faster (to keep numbers easy). This means that the bullet is carrying 100x as much kinetic energy as the gun.

So for every unit of energy your hand/shoulder has to absorb the bullet proof vest (combined with your flesh/bones behind it) need to absorb 100 units, which is gonna take a lot more to absorb. With something like a rifle, much heavier than the bullets it fires, that could go to 200x, 300x or more.