Hi all I came across a new failure case, I have added it in the original post. There is a chance that the watermark is causing the issue but still, with 71.7% confidence the model has detected it as a face.

Facial attribute analysis, that is identifying the gender (one of the attributes) of the face detected. That's why face detection, the preliminary step, has to be done in a way that only human faces are detected.

This is a great insight. Thanks for the help.

Due to lack of details of cartoon faces, the model should not detect these cartoon faces as faces with a high confidence score (about 86.8 %). If this is the case then these models are as bad as haar cascade based face detectors.

I think so these models were trained with real human faces (eg.- http://shuoyang1213.me/WIDERFACE/) and not cartoon faces. So, the example that I have shown would be a false positive.

Retraining the models might be my last resort to solve the issue as I still want the high performance of these models. And retraining them would definitely come at the cost of their performance.

I understand that, but these are state of the art face detection models (as per papers with codes), one would assume that they would have taken care of these kinds of false positives. This has been a common issue across many face detection models and I hoped that someone could suggest a model that has been trained against it. Fine-tuning the detector would be my last resort.

These are pretrained models, which we can access by their respective libraries and if we check at papers with code rankings, they are one of the best for face detection. I had hoped that someone would have used these libraries in their application and would have solved the issue.