You will be absolutely fine. There is plenty of foot traffic in that area at all times of day and night. You are just about across the street from the Yale Security offices. There are plenty of good restaurants within a block or two and the rest of downtown and Yale is just beyond that. Stop & Shop is only a short walk, though I might only do that in the daytime as the Elm St side can get a little squirrely at night.

Use common sense and take the same precautions you would in any city and you'll be fine.

In general though, New Haven is a great little city that punches way above it's weight in terms of culture, restaurants, bars, clubs, arts, music, etc.

Edit: I live a couple of blocks away from there and have never had any problems.