> I understand when a vendor passes along the costs of using a credit card and don't really have a problem with that.

I don't, it's the cost of doing business. If you don't want to accept cards, you don't have to. But you will have an unhappy customer base as a result.