Height matters more than anything. As long as the trim doesn't interfere with your switch plate, it's good. You can always leave extra wire to the box and see how it feels in one spot, and then move it if you have to.

If you're getting it inspected, put it where you want and see what the inspector says. If you left extra wire it should be trivial to move.