# How do I distil a large model into a smaller one?

> How knowledge distillation works: use a large teacher model to train a small student for your task, then test it and keep the large model as a fallback.

Source: https://greenai.guide/how-do-i/distil-a-large-model-into-a-small-one/  
Updated: 2026-11-19

## In short

Check first whether a distilled version of your model already exists. If not, use the large "teacher" model to label or generate examples for your task, train a small "student" model on them, and test the student on your own real cases. DistilBERT, an early example, was 40% smaller and 60% faster than BERT while keeping 97% of its language understanding.

## Steps

1. Search for an existing distilled or smaller version of the model for your task.
2. Collect a set of real inputs from your task.
3. Use the large model to produce labels or outputs for them, and check a sample by hand.
4. Train a small student model on that data.
5. Test the student against your quality bar and route its uncertain cases to the large model.

**Distillation moves what a large model knows about your task into a small model that is cheap to run.** You pay
once for training the student, then save on every request.

## Why does it matter?

A large general model deployed for one narrow job, such as classifying tickets or extracting fields, pays for
knowledge the job never uses. Luccioni, Jernite and Strubell found task-specific systems orders of magnitude cheaper
to run than multi-purpose generative models for the same tasks. The Green Software Foundation lists distillation
among its patterns for right-sized, energy-efficient AI models.

## How does it work?

The **teacher** is the large model; the **student** is a small one. In the classic method, the student learns to
match the teacher's output probabilities, not just its final answers, which carries more information per example.
Sanh and colleagues' DistilBERT used this to produce a model 40% smaller and 60% faster than BERT that kept 97% of
its language-understanding ability.

For generative models, a common practical route is simpler: have the teacher produce answers for your real inputs,
check a sample, and fine-tune a small model on them.

## Common mistakes

- **Distilling when a small model already exists.** Search first: see
  [how to find a task-specific model](/how-do-i/find-a-task-specific-model/).
- **Trusting teacher outputs blindly.** Errors in the teacher become errors in the student.
- **No fallback.** Keep the large model for cases the student is unsure about, using
  [model routing](/how-do-i/set-up-model-routing/).
- **Ignoring the licence.** Some model terms restrict using outputs to train other models.

Read more in [efficient training and inference](/practice/efficient-training-and-inference/).

## Sources

- [DistilBERT, a distilled version of BERT](https://arxiv.org/abs/1910.01108), Sanh et al. (arXiv 1910.01108) (2019)
- [Green Software Patterns: right-sized, energy-efficient AI models](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/development/right-sized-energy-efficient-ai-models.md), Green Software Foundation (2025)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://arxiv.org/abs/2311.16863), Luccioni, Jernite and Strubell, ACM FAccT (2024)
