I think it’s not only not wanting to be labeled, but recategorizing the meaning of the behavior in general. Even if you live in a liberal area and nobody would think less of you for it, you maybe don’t want to give a false impression about yourself/your friendships—least of all to your friends who now might worry you’re coming on to them or crossing boundaries if you try to hold hands or something!