The issue with cooling in open space is that there's nothing you can dump your waste heat into.

Things like ISS use special radiators to radiate away waste heat - while small things that can generate a lot of heat for their size, like space suits, use water evaporation to mount a compact heat removal system.

On the surface of the Moon, you have the entire Moon that you could dump your waste heat into.

Of course, actually dumping the heat into the ground is an engineering and construction challenge - so early Moon and Mars reactors might opt for radiators, like the ones ISS currently uses. They would need a lot of them, but it's still doable for small reactors.