4 feet should be good for all but the very north of Wiscowsin. I built a shed the same way, gravel with concrete over.

If it's too wet the holes will have water in them. That will be a problem. I had to move all my poles over 3 feet after drilling about 10 of them and letting them sit overnight. The next morning we were screwed. Even tried pumping the water out. They just refilled.

Don't just auger your holes down. Let the auger fill with dirt. pull it out and dump it off to the side. Otherwise the dirt will be nothing but in the way when you set your poles.

The Harbor Freight cement mixer was good enough for the 25 holes I filled. I still use it from time to time. I did have to blow the motor out with air to getting going again once.