Fun idea, ruined by humans.

You think "oh lawyers will be fair" but we thought that about cops too and we know how that worked out. Oh well use people "on both sides" but it's awfully easy to fool that.

It's a hard problem. I don't have the answer either, but it's hard to be worse than the current system without going straight up evil.