Violet has the shortest wavelength, so the greatest chance to interact with a more dense medium. This slows it more than blue, which is more than green... you get the idea.

Hitting straight on you wouldn't notice, but at an angle, it causes the light to bend (think of driving a car, then slowing the left wheels more than the right, the car will turn).

Because the Violet portion bends the most, it ends up on one end, and the red (which bends the least) ends up on the other.