Pre-kid, my wife and I used to do it a decent amount because Toronto is awesome. It is a 6 hour drive, so basically driving to Buffalo, crossing the border and then another 90 minutes. I've never had an issue with it outside of the usual traffic at the border. It is enjoyable and worth it to make the journey to one of the best cities in North America.