It actually happened very often, especially if both men were similarly armed and armored. I'm gonna assume you're referring to pre-modern battles since you mention Samurais and swords.

Usually, the majority of an army was infantrymen wielding melee weapons, and in fact the word "melee" refers to the confusing, scary part of pre-modern battles where two infantry contingents finally clash and don't have much room to maneuver.

Medieval European martial arts manuals have sections on hand to hand combat, including grappling an enemy with a sword or spear with and without your own weapons.

Indian martial arts under the broad label of Kalaripayattu include armed and unarmed instruction. Same with Chinese martial arts.

I don't really know much about other parts of the world, but wrestling and grappling seems to be a fairly universal past-time. You even see Mongol and Turkic wrestling traditions, and those dudes were most famous for horseback fighting. Those skills would probably be necessary for lance-wielding cavalry if they get dismounted.

(Eastern Asiatic) martial arts were generally developed as a substitute to combat with weapons where weapons were restricted. For example when some classes of the society were forbidden to own weapons (slaves, peasants) or geographical restrictions of weapons (ban of weapons in towns). This actually includes not just bare hand fighting but also substituting weapons with some other objects and tools, like farming tools (nunchagu might be the most famous example).

Therefore it's safe to say that it occurred often. But that does not mean that it was a good or proficient way of fighting. More like something out of necessity.