How are the Japanese going to supply an invasion force in Hawaii? Admiral Nagumo was worried that if he didn't start back after the 2nd wave he would have to scuttle some destroyers due to lack of fuel.
If Nagumo does a third wave his pilots have to land at night, of which they are not trained to do so. If he loses a quarter of his pilots he can't replace then with equal quality. He basically loses a carrier or two due to not having pilots. If its a disaster he might not have enough power to project meaningful air power at sea. Several planned invasions would have to be scrapped. Destroying the fuel depots or sinking the carriers buys time, but neither are going to be long term game changers. The US can replace the fuel, and the US built (many) more ships from Pear Harbor to 1945 then Japan built plus its pre war navy.