Here are some options

1. Tune a smaller network, then apply the hyperparameters to the larger one and 'hope for the best'.
2. As others have said, train less, for example, 10 epochs rather than 100. I typically find this produces the wrong results though (the best performer is often poor early on)
3. For low dim (2d) perform a very coarse grid search (space samples an order of magnitude apart, maybe two), then use just the best model. This is often the best method as you don't want to overtune the hyperparameters.
4. For high dim, just use random search, then marginalize over all but one parameter using the mean of the best 5-runs. This works really well.
5. If the goal is often to compare two methods rather than to maximize the score, you can use other people's hyperparameters.
6. Baysian optimization is usually not worth the time. In small dims do grid search, in large do random search.
7. If you have the resources then train your models in parallel. This is a really easy way to make use of multiple GPUs if you have them.
8. In some cases you can perform early stopping for models which are clearly not working. I try not to do this though.
9. When I do HPS I'm doing it on another dataset than my main one. This helps make things quicker. I'm doing RL though, so it's a bit different I guess.