Submitted by Singularian2501 t3_z1b2rp in MachineLearning
Paper: https://arxiv.org/abs/2211.10438
Github: https://github.com/mit-han-lab/smoothquant
Abstract:
Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, for LLMs beyond 100 billion parameters, existing methods cannot maintain accuracy or do not run efficiently on hardware. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs that can be implemented efficiently. We observe that systematic outliers appear at fixed activation channels. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the GEMMs in LLMs, including OPT-175B, BLOOM-176B and GLM-130B. SmoothQuant has better hardware efficiency than existing techniques using mixed-precision activation quantization or weight-only quantization. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. Thanks to the hardware-friendly design, we integrate SmoothQuant into FasterTransformer, a state-of-the-art LLM serving framework, and achieve faster inference speed with half the number of GPUs compared to FP16. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code will be released at: https://github.com/mit-han-lab/smoothquant in ~2 weeks.
​
singularperturbation t1_ixat83k wrote
This is highly relevant for my work, I'm very excited about this!
(Ah n/m I assumed the submitter was one of the authors.)
I saw that you've uploaded activation scales (equation 4) for a number of models, but if calculating this for a new model, how do you use a calibration dataset when doing that? Do you take the maximum across all of the calibration values, or calculate the maximum for each value individually and then average? I see that> Code will be released at: https://github.com/mit-han-lab/smoothquant in ~2 weeks.
so I guess I may just need to be patient until this is released lol.