Olive does not mean white tho. When read ASOIAF, the Dornish were described as having Olive skin, and Dorne is based on the Middle East.

Edit: lol some knob downvoted me.