noun · AI technique
Quantization
also called model compression
Rounding the model's numbers to be shorter. The model gets smaller and quicker, at the cost of a little precision.
Reducing the numeric precision of a model's weights to shrink its size and speed up inference, with some loss of accuracy.
The weights, and sometimes other values, are stored with fewer bits, for example 8 or 4 instead of 16. That shrinks memory use and can speed up inference on suitable hardware, while rounding error slightly changes the model's outputs.
Real-life example
You want to run a model on your own laptop, but it does not fit in memory. You download a quantized version of the same model and it fits and runs fine. On a few hard questions its answers are a little worse than the full version, which is the price of the smaller numbers.
Where the analogy breaks
-
It is not free. Rounding causes small quality losses that grow as you use fewer bits, so test the version you plan to run.
-
It is not the same as distillation. Quantization shrinks the same model's numbers, while distillation trains a different, smaller model.
-
Speed gains depend on hardware. Smaller numbers save memory for sure, but the speed-up varies with the chip and the software.