There are three possibilities for irrelevant structures, as far as evolution is concerned:

• they’re useless, but completely harmless.
• they’re useless, but passively harmful - they use energy or resources that could be used elsewhere.
• they’re useless, and actively harmful.

The effect of the third possibility is obvious. Individuals that disrupt that harmful thing do better, reproductively. Mutations that prevent the harmful thing from forming are selected, just likely any other positive selection, and the disrupted version spreads through the population.

The second option may be harder to visualize, but it leads to exactly the same outcome. Forming an organ you don’t need uses resources; maintaining a non-helpful organ uses energy. If you can divert those resources and that energy to, say, earlier reproduction, you’ve got a positive selection trigger that will spread through the population.

If something is useless, but harmless, then there won’t be any positive selection in disrupting it. (Also, it’s possible in the second case that the positive selection is so weak that it won’t spread particularly quickly, and that comes down to this case.).

But mutations happen all the time. We are all born with dozens of mutations, almost all harmless or nearly so. Those who are born with harmful mutations have them negatively selected. Harmless mutations just hang around, because they aren’t selected against.

If a mutation hits our now useless structure, there’s no selection against that mutation - neither negative nor positive. If there’s no negative selection against mutations, then very gradually, just through genetic drift, the structure will slowly accumulate mutations and gradually, just through drift, become slowly and randomly inactivated.

Nails are a bad example here because they do have positive value to us - not ripping apart food, but supporting touch and gripping, for example. A better, classic example is the vitamin C production system in fruit-eating primates. Because fruit is rich in vitamin C, there was no value to those primates in keeping the system, but it was also pretty much harmless, not using significant resources. Over time, genetic drift allowed mutations to accumulate in the enzymes involved and the system stopped working - causing no harm, but no benefit either.

(Some people argue that inactivating the vitamin C pathway does have a small positive value, but the principle is there anyway.)