You’re technically not supposed to swim outside of the beach but I do it sometimes in the summer after work, in the opposite end of the pond. Only once did a teenage park ranger row by and nicely tell me that swimming wasn’t permitted outside of the beach area (my friend had a sparkly inner tube so I imagine that made us more visible from afar) so we got out and that was that.

So basically, no it’s not allowed but if you’re discreet about it you should be fine.