The last sentence you wrote is essentially correct. The b cell that gets activated first, regardless of affinity, is the one that gets clonally replicated. Those clones have some randomization of their antigen matching site -> the ones that bind better get selected for future cloning -> ad infinitum until infection is clear.

Also, there are memory b cells of previous infections. Infections tend to be geographic; standard respiratory infections, hand infections, foot infections, etc. So those cells are already there for clonal proliferation.