Keep your eyes open, your phone in your pocket, and you should be fine, regardless. That said, the area near Lexington market can sometimes be a bit dicey late at night but there’s usually people around. Baltimore to Charles is probably the safest bet, however.