In one of our recent work https://arxiv.org/abs/2012.14956, we used Genetic Algorithm to attack NLP models in a hard label black box setting, where we do not have access to the confidence scores of the model.