So the concern is this: A patient may become dependent upon a doctor for his/her medical expertise and care, and this creates a power imbalance between the two. If the doctor then starts looking for sex, they may feel obligated to comply in order to keep being looked after for whatever medical issues they have. This is why it is unethical, unprofessional, and not permitted by medical regulatory bodies.

There are still always grey areas, and certainly if a doctor and patient agreed to terminate the medical relationship, and a certain period of time passed, then it would be okay in some circumstances.