It just depends. When you get a cut on your skin, that area is almost certainly going to get bacteria trying to grow in it. Your body will, mostly successfully, fight it off.

But... Sometimes it gets red and infected and then your body will really work at it... And then fight it off. Other times you lose a finger if left untreated.

Same with viruses. Mostly you can fight them off, but eventually one can put you in the hospital.

Then there are the scary members of both viral and bacterial families. Ebola and rabies viruses are always going to do serious damage. Likewise, there are a decent number of flesh eating bacteria that, once they take hold, are very difficult to stop.

Just to throw some other pathogens in there, if a fungal infection manages to get into your spinal fluid or lungs, that can be impossible for your immune system to fight off. Similarly, if an amoeba gets up into your brain you are hosed.

But, for every deadly virus bacteria fungus and amoeba, there are countless numbers of harmless ones that your immune system just deals with without breaking a sweat