Here's a paper on habitability around brown dwarf stars:

https://iopscience.iop.org/article/10.3847/1538-4357/ab5b13

It does say that brown dwarfs cool much faster than our sun, so I think your story's premise is plausible. The paper assumes you need at least 1 billion years for life to evolve so the mass of the star must be at least 20 Jupiter masses or it'll cool too fast to remain in the habitable zone. It further limits the minimum mass based on requirements for photosynthesis and UV output.