Save the day trip for the summer. For someone already from the Midcoast area, you won't be as awe-struck with the place as someone from another state to make going in the winter worth your while. (Not to mention, ferry cancellation is far more likely in the winter than summer.)

The summer is much more vibrant. Swimming in the quarries, hiking lanes island (either beach or trails), or visiting the flea market (Saturdays). Not to mention, your food options are far more varied in the summer.

I'd recommend bringing a car. The island is not to big, but it will help you fit the most into a single day.