Break up, you cant be blamed if you're not attracted to a person. Now I wouldnt call him ugly or say that why because thats just a horrible thing to do, but you're both better off not being together if one of you isnt happy.