I believe that there comes a point where a minority can be out-voted, by public opinion, to the point where its possible to direct intolerance on them with little more than a protracted campaign of hate speech.

Put another way, when there aren't enough XXX to defend themselves, public opinion can be swayed with little more than 'freedom of speech'.

At that point, the public become willing to vote for INtolerance in that specific case. It's happened many times historically, and is a reason I can get behind both aspects of European hate-speech laws, and aspects of American sue culture.