noun · AI process
Distillation
also called knowledge distillation
A small AI learns from a big one. The big model answers lots of questions, and the small one is trained to answer the same way.
Training a smaller model to reproduce the behavior of a larger one, to get similar results at lower cost and latency.
A large teacher model produces outputs, often with its probabilities or reasoning, for many inputs. A smaller student model is trained to imitate them, so it keeps much of the teacher's behavior on the target task at a fraction of the size and cost.
Real-life example
Your app sends thousands of requests a day to a large model, and the bill and the wait are both high. You save its answers for a task, then train a small model to match them. The small one handles that task almost as well, far faster. It is much worse at everything else.
Where the analogy breaks
-
The student does not match the teacher everywhere. It is good on the task it was taught, and usually weaker outside it.
-
It needs many teacher outputs. Collecting enough good examples takes time and compute before you save any.
-
Check the terms of the model you learn from. Some providers restrict using their outputs to train other models.