Amazon's hiring algorithm once had to be turned off becsuse it became racist and sexist. It was automatically excluding non white and female sounding names.

Iirc, it was because the majority of applicants were white males. As a result, most of the hires were white males. The AI recognized this as meaning that white males were the ideal candidate and would hire them more, which caused a feedback loop.