Nothing bothers me, per se, it's just the way it stands out. The colour is unlike the rest of the piece but not in a way that's glaringly out of place. It's also the only clothing he's wearing, somewhat contradicting the title and leading one to wonder at your reasoning or perhaps to infer something about the unseen context of the picture. You could also look at it as being a kind of balance to and distraction from his genitals which otherwise would be a clear contender for the focus of the piece. Maybe you've found a clever way to make us consider his whole body rather than just the fact that it's a male nude. In any case, I think your colours are well-chosen and I like the crayony texture of the lines.