> I also randomly slow down and speed up without noticing.

Again, without hearing I could be way off, but what I'm guessing is that you're counting the "rhythm" as opposed to the beat. Maybe counting a down beat as an upbeat because it "feels" right. Would definitely try counting your favorite songs & just getting that down ad nauseam. Another thing would be if you're counting, clap or tap something to get that physical feeling as well of doing so. It will help it translate when you're not focusing on counting but still have the beat going on somewhere in your extremities.