You'll need to specify what you mean by 'worse'. There is CONSIDERABLY more mining and extraction of fossil fuels across the entire globe, both in developed and undeveloped countries. This leads to considerably more emissions and environmental damage than rare earth materials. There is also all the pipeline leaks, ocean platform leaks, etc to weigh in.

I am also led to believe that rare earth material mining is more polluting on an individual mining/operation scale, but because there is farrrrrr less of these mines in operation, it does considerably less global damage. It does more damage locally than globally.

This is why, depending on which side of the fence you are on, there are ways to twist a discussion to make one side sound so much better than the other. Ultimately, both have negative impacts on environment and in many cases the employees and communities they exist in or near. Recycling can be an answer to try and prevent the impacts of either scenario above. We just need the infrastructure and political willpower to make it happen.