Thanks a lot for the detailed reply! I will try these suggestions.

Thanks for the reply. But do these three quantization techniques work on the GPU without TensorRT? The supported backends led me to believe they are not: https://pytorch.org/docs/stable/quantization.html#backend-hardware-support

Thanks for the reply!

I personally find TensorRT hard to debug and I prefer to use it only in production when I'm absolutely sure that the model produces the desired results.

Thanks for the nice episode. Don't see a point to criticize the talk they had, but instead enjoy all the interesting parts (such a the books, e.g.).