The short answer is, it appears that baleen evolved from the system that grew "adult" teeth in baleen whale ancestors, in combination with existing genes used elsewhere in whales for producing keratin as hair or claw sheaths.

The ancestors of whales, like us, had two sets of teeth—what we call "baby teeth" and "adult teeth." As fetuses, baleen whales actually begin to grow their baby teeth! The teeth bud and can even begin to mineralize, but eventually tooth development is halted and the tooth buds reabsorbed, after which baleen forms in the upper gums.

So baleen whales never begin to grow their "adult" set of teeth, but the genetic/protein signals that trigger the development of baleen are very similar to ones that trigger development of teeth in all other mammals, even though the baleen itself is nothing like teeth (instead, baleen is thick sheets of keratin, very similar to hair or fingernails). So what seems to be happening is:

• The whales jaws/gums signal the production of "baby" teeth.
• A different signal then aborts their development and they are reabsorbed rather than rupturing out of the gums.
• The gums then signal the production of "adult" teeth, but with a twist, where hair-like baleen grows out of the gums instead of enameled teeth.