In the post I said I unfroze the CNN layers, I meant after the transfer learning part. I run it untill it early stops with all CNN layers frozen, then run it with unfreezing the top 200 layers or so.

I'm obliged to work on Keras K don't know if it has an LR sched method, I'll check the API great advice.