As somebody that made poor financial decisions in my younger years and did this exact same thing for my first house I say do not touch your retirement funds. That was a few decades ago and I still regret touching retirement funds

I could have got the house without tapping my retirement

And using an online stock market calculator that 10k then would be worth 50k today if I had kept it invested