Quite a few of these mutations impact mitochondria in a negative way. For example the LHON mutation often causes blindness, but only in about 30% of carriers (so there's probably some environmental or non-mitochondrial DNA factor that "activates" it).

Since cells have multiple mitochondria, and since the non-mitochondrial DNA rules most of the cell, it can be tricky to determine what any mutation does and having it may not cause a change for all carriers. So it won't have as strong a selection pressure as it may intuitively seem, and any fitness improvement may still take a long time to spread thru a population.