Charles doesn't actually look like the portrait. He has a much slenderer, oblong face shape with a prominent nose and ears, slender jaw, on top of much more wrinkles and much less hair.

Even when he was young he didn't have that face shape.

They didn't want to make the portrait actually look like him because he is so unconventionally handsome, so they made a fictional Charles.