"Stay right except to pass" is mostly only applicable on the freeway. If you're on a road where someone may need to make a left or right turn at any time, there should be no expectation of them moving to the right lane just because you want to go faster than them. On the freeway though... it's "stay right except to pass. Not "stay right unless you think you're going a reasonable speed".