Are you positive that the hinges are screwed into a stud? Asking because I had the same issue, but it was because the top pivot hinge was attached to drywall using a heavy-duty toggle bolt, and the weight of the 1/2”-thick glass door was slowly pulling the drywall out of square. I fixed it by having a shower glass company come remove the door, then I cut the wall open and installed a stud right where the hinge went. Then had the shower glass guys come back to reinstall it. This was 6 years ago, door’s been nice & straight since then.