I agree with your perspective, but disagree with the assumption that the use is derogatory or a gotcha. I think it's usually used to indicate that an irreligious celebration is fine, because after all the traditions don't have their root in the religion anyway. At least, that is my viewpoint. I'm an atheist but I like celebrating Christmas as a secular holiday. Things like a Christmas tree don't feel weird because they're not Christian in origin anyway; there's no overt Christian symbolism going on. I would not ever set up a nativity scene, though; that would feel extremely weird to me because it's obviously an expression of a faith I do not have.

I believe the "gotcha" aspects are, in fact, a reaction to the "keep Christ in Christmas" people. There are people who believe that the secularization of the holiday is wrong and offensive. It's at that point that people are ready to come out swinging about how so many aspects of the celebration did not emerge from the religion itself. It's not so much a "gotcha" as an Uno reverse card. People aren't slagging on a holiday they probably enjoy themselves for being inauthentic, they're defending themselves against people who insist that the non-Christian aspects don't belong. And from what you've stated, I'm sure you agree that they do belong, because this is a cultural thing more than it is a religious one.