> your freedom to punch the air end where my face begins

That standard falls apart when you realize that very few actions (certainly none of the ones we care about) have no effect on others. Whether it is talking with someone, playing multiplayer games, or even buying groceries at the store -> all of those can have meaningful effects (positive and negative) on others.

Saying that you are free to do things that have no effect on anything or anyone is irrelevant, since no one cares about those actions. In most cases, freedom refers to the right to do things that might be harmful/objectionable to someone else.