I like New Haven. There is a lot to going on for a small city. However, it's not New York or Chicago. It's not big enough to just wander in a random direction and be guarantee to stumble upon interesting places. Unfortunately for you, the Popeyes you stopped in is at the unofficial demarcation between the tonier part of New Haven and the poorer. That was some bad luck. I'm sorry you didn't find what you were looking for, but with a bit more research I think you could have put together a worthwhile day trip.

Yale has some interesting architecture. There are some cool gothic-style buildings about a block away from the Shops at Yale. If you walked down Wall St you would have seen law school, the Beneicke Library and Woolsey Hall.

Most of the restaurants and nightlife are concentrated in the area bounded by Orange St, Howe St, George St and Elm St. You can also find some small pockets of restaurants along Whitney Ave, Wooster St. and Whalley Ave in Westville.

If you want culture, there are plenty of plays and classical music concerts because of the Yale's drama and music schools. Yale also has an art gallery and a center for British art. There are some great festivals on the green when the weather is nice. You can hike up East Rock Park to see views of the city and Long Island Sound. Light House Point is also a nice walk. They have a festival of Christmas lights in December.