Nozzle under-expansion: There is energy left in the exhaust that could have been converted into momentum with a longer/more expanded nozzle. This energy goes into expanding the plume outward into the air aft of the vehicle that could have been used to push the vehicle forward. You get a certain amount of "pressure thrust", which is the pressure of the exhaust relative to the ambient air pressure pushing against the vehicle, but this is small compared to "momentum thrust".

Nozzle over-expansion: You have expanded the exhaust gases past the point where they are at the same pressure as the ambient air, and the atmosphere is actively pushing back against your thrust stream. Extreme over-expansion will cause the exhaust plume to creep back into the nozzle, detaching from the nozzle wall and eventually leading to combustion instabilities.

A vacuum nozzle can never be over-expanded, since the static pressure of vacuum is essentially zero and the exhaust plume will always have positive total pressure. Vacuum nozzles' sizes are limited by other considerations such as mass and structural integrity.