Try this. Drill a hole with a bit that is several sizes smaller than the final size. This will be your pilot hole.

Using final size bit, slowly drill from "back" side (side that isn't seen as much) to a depth of half the thickness of the wall.

Then drill from the front side and finish the hole. Presumably, there will be less material being pushed out compared to fully drilling from only one side.