That a building was used by someone who had slaves should not disqualify it from having a marker. Indeed, I would argue the opposite. We need to know these structures and people existed. Ugly or not, it is part of our history. Let's not erase it.