Humans still had to tell alphazero what it was supposed to optimize for. We defined what the rules were and told it what outcomes we wanted.

If we want an AI moral system, we'll similarly have to define the rules and what outcomes we want.