For babies the biggest thing is getting them used to being in the water and at the pool. You can do that for free at a public swim at a DC Parks and Rec pool. Of course, a class would be helpful for some folks, for a variety of reasons, but it's not really necessary until the kid gets to 3-ish years old.

Source - I taught YMCA/American Red Cross swimming for 15 years, including baby classes