Infant mortality was high (one estimate is that the average Roman woman had to have 5 children just to keep the population constant), and higher still in urban areas. A large urban area like Rome would have been particularly unhealthy. Throw in a very competitive political environment, the tendency of younger emperors to be killed before they could marry and that emperor remained always an office (open to anyone who had enough support among the military 'electorate'), rather than a hereditary position, and 'dynasties' rarely lasted more than 2-3 generations.