> (...) a new state of the art performance of 50.3% accuracy on the MedQA biomedical question answering task.

Oof, the fact that the accuracy is only 50% does not inspire confidence.