I'll start off with a personal story, and then offer a few things to consider.

Story:

When I was a kid, there was a long trail that ran behind a bunch of neighbors houses. I used to walk this trail for years. When I grew up, I left and came back on a visit at one point years later. I was walking the trail one day and a guy came running out of his house, waving his arms, and explained very firmly to me that this was no longer allowed. In a similar vein, this was because a different neighbor had been getting drunk and riding his quad around this guys property late at night. I was incredibly angry at this rebuke, but told him I'd stop. I wasn't angry at him because I felt like the trail was "mine", but instead because I had this kind of connection to it. You know? It was a very special feeling of being able to walk the same path for my entire life. It was his choice, and his call...it was his land after all. I just felt this terrible anger about it because I felt like it was kind of like taking back a statue that had been put up in the center of town. It wasn't owned by me or the town, but the loss of that thing hurt in a very personal, sentimental way.

Considerations:

1. If you'd like to institute some sort of compromise, perhaps you could put up signs explicitly saying what people should do, rather than assume they know what not to do.

   "Before entering, call this number... Unwelcome guests and trespassers will ... Do not damage the land in the following ways ..." Something like that. If you cut out the excuses people dish up, then hopefully fewer problems would occur. I suppose depending on how egregious things got, you could escalate things from there. Personally, I would have demanded the person word that they would pay for the damage, if they had purposefully destroyed my fence.

2. On the flip side, it might not hurt to try and cut some of the more clueless people a tiny bit of slack, too. I would think that if I, say, asked the same fellow once a week for a full year, "Do you mind if I go through here?" And he says "Sure, yeah!" each time, would it make sense to keep asking? It wouldn't necessarily be me thinking "Oh, this is mine" but instead something more like, "Why am I going to keep bothering this guy with asking?" I suppose you might see things differently, but it might not hurt to think on it.

3. Also, do keep in mind that, unless these are people that know you pretty well, they might feel some apprehension about banging on your door. "The locals told me they go through this guys land all day, every day. Why am I going to check with him first? What if he isn't even home? What if he's in a rotten mood and doesn't want visitors?" Etc. I've noticed some of the younger generations can get very skittish about talking to strangers - even if those "strangers" happen to be only be a few roads down, and have lived there for years. There's a kind of assumption of "extreme privacy", people tend to assume they're "bugging" someone by coming up unannounced. I can't quite explain it.