I am not much of a mathmetician, but have published theoretical ML papers. To answer your second question, I had no idea that I would be able to solve the problem when I started out.

The process was a lot of trial and error, and iterative refinement. I started with the simplest form of the problem I could and made every simplifying assumption that I could (usually that various terms are insignificant). Once I got closer to solving the equations, I got a better idea of what form the initial problem must take for the final solution to be solvable. Then working backwards, I determined which assumptions are necessary, solved the problem, then checked the assumptions.