There are downsides - and the weather is the biggest one - but there are downsides everywhere! I don't think Vermont is perfect but I don't really want to live anywhere else, either.