If you're really trying to get away from the vibe of NJ/Fairfield County, and you have this much flexibility, I would definitely go Madison or Guilford. There is just an entirely different vibe there, with lower population and a more beachy feel.

Personally I find the Farmington Valley area just as complicated and crowded as any NYC suburb. Think "the one road with every big box store on it that backs up every Saturday by 10 AM." Which, is not a bad thing at all, but it sounds like your goal is to find somewhere different.