So yes, that's another good point. Part of the answer is a gravitational force can affect a system without affecting the motions within that system - and that does come down to the smoothness of the potential. I also wanted to clarify that dark matter isn't a magic thing that just speeds up orbits - in terms of gravity, it's just a distribution of matter, that acts like any other distribution matter. But yes, the other part is that there isn't an additional local effect, as the local density of dark matter really is very low.

Though I would want to clarify for other readers (as I assume you know this of course) that while there is locally not much dark matter (it has a very low density), but because it's smoothly spread out over such a huge 3D structure, its total mass is much larger than the mass of the stars and gas in the Galaxy.