if you are trying to get out of new haven, the only cheaper options along that stretch of the shoreline are west haven (which is still pretty affordable) and bridgeport (which, like new haven, varies wildly depending on which neighborhood you are looking in)

as others have said, the naugatuck river valley has affordable rents, too. it's served by the waterbury branch line of metro north so it's possible to live a car-light existence up there, although it's much easier along the shore.