The actual problem here is the lack of commitment.

For example, I physically "prefer" women that don't have the body type of my wife and I always have. But that doesn't make it okay to invalidate the commitments I made to her and emphasize my fantasies over her.

Because then I'm no longer happy with the person that I have.