"What if there were no hypothetical questions?"

Yes it's an old joke, but relevant. Reasoning is inherently based on analogy and abstraction. I would claim "what ifs" are necessary to form any sort of world view or philosophical position. The fact that context can change our view of the correct answer is interesting but does not defeat the purpose of the question. Arguing about which context is most relevant is just where it starts to get interesting.

That's why the option to leave is important. If they chose to stay they are subject to the rules. That is their choice.

> not being able to apply those rules and restrictions to people who explicitly do not consent to the social contract.

If someone I don't have a "social contract" agreement with attacks me I will certainly defend myself and perhaps kill them if necessary, and that would be morally justified. I don't need their agreement to do that. Similarly if someone who doesn't agree to my town's social contract regarding property rights breaks into my home and steals my stuff why should I need their agreement to apply the law? If you don't agree with the town rules you should not enter the town.

You can argue the "social contract" is the willing acceptance of a group's rules in order to obtain in-group or social benefits. It is a valid "contract" to the extent that you can leave the group and it's rules and join another (or live as Robinson Crusoe).

There is no a-priori reason to suppose a group's rules would be fair or maximize benefits over restrictions, however as long as there are multiple groups and individuals or families can migrate between them or influence the rules, then there is a selection effect towards the most beneficial, fair and effective rules. The fact that other primates also have strong concepts of "fairness" suggests that this group selection on rights and duties has been affecting our ancestors for millions of years. Thus you can argue that modern societies have reached a near-optimal and thus pretty "moral" set of rules for maximum personal net benefit with minimal restrictions and unfairness. It is therefore probably unreasonable for someone to reject modern social norms. I don't know if this is considered a Lockean view.