That's an interesting claim, certainly believable if you look at direct deaths (which globally is probably under 100 - almost entirely from Chernobyl, especially if you count Kyshtym as military).

I'd love to see the source

Estimating, and attributing things like cancer correctly (reduction in life span and life quality) from things like construction and dismantling, uranium mining, etc is trickier.

On the other hand so is the mining for the metals needed for solar, and the construction risk per kWh I would guess are orders of magnitude higher for solar (especially rooftop solar)

I'm sure both are dwarfed by the impact from oil, gas and coal though.