Model Distillation

Training a smaller model to mimic a larger model so it runs faster and uses less compute.

Also known as: Knowledge Distillation, KD

In plain English

Model distillation is like making a pocket-sized cheat-sheet from a big textbook: it keeps the most useful parts but fits on your phone. A large, slow AI teaches a smaller one how to answer similarly so the small model can run quickly on cheaper hardware. There may be a little loss of detail compared with the full model.

In practice

When buying or deploying AI, distillation lets you achieve near the large model's accuracy with a smaller model that reduces cloud costs, latency and energy use. It makes on-device or high‑throughput deployment practical but requires extra engineering for the distillation pipeline and may introduce a small performance trade‑off.

Under the hood

Knowledge distillation trains a compact 'student' model to match a larger 'teacher' model's outputs (typically logits or softened probabilities) using a distillation loss often combined with the supervised loss. Common techniques include temperature scaling of logits, matching intermediate representations, and using augmented or synthetic data; the goal is fewer parameters and cheaper inference with acceptable accuracy loss.

Example

“Can we distil the current foundation model into a smaller student for edge deployment to cut latency and hosting costs?”

Related terms

Fine-tuning · Small Language Model · Large Language Model

•