greenai.guide

Home › How do I…? › How do I distil a large model into a smaller one?

How do I…?

How do I distil a large model into a smaller one?

Updated · 1 min read

In short

Check first whether a distilled version of your model already exists. If not, use the large "teacher" model to label or generate examples for your task, train a small "student" model on them, and test the student on your own real cases. DistilBERT, an early example, was 40% smaller and 60% faster than BERT while keeping 97% of its language understanding.

Steps

  1. Search for an existing distilled or smaller version of the model for your task.
  2. Collect a set of real inputs from your task.
  3. Use the large model to produce labels or outputs for them, and check a sample by hand.
  4. Train a small student model on that data.
  5. Test the student against your quality bar and route its uncertain cases to the large model.

Distillation moves what a large model knows about your task into a small model that is cheap to run. You pay once for training the student, then save on every request.

Why does it matter?

A large general model deployed for one narrow job, such as classifying tickets or extracting fields, pays for knowledge the job never uses. Luccioni, Jernite and Strubell found task-specific systems orders of magnitude cheaper to run than multi-purpose generative models for the same tasks. The Green Software Foundation lists distillation among its patterns for right-sized, energy-efficient AI models.

How does it work?

The teacher is the large model; the student is a small one. In the classic method, the student learns to match the teacher's output probabilities, not just its final answers, which carries more information per example. Sanh and colleagues' DistilBERT used this to produce a model 40% smaller and 60% faster than BERT that kept 97% of its language-understanding ability.

For generative models, a common practical route is simpler: have the teacher produce answers for your real inputs, check a sample, and fine-tune a small model on them.

Common mistakes

  • Distilling when a small model already exists. Search first: see how to find a task-specific model.
  • Trusting teacher outputs blindly. Errors in the teacher become errors in the student.
  • No fallback. Keep the large model for cases the student is unsure about, using model routing.
  • Ignoring the licence. Some model terms restrict using outputs to train other models.

Read more in efficient training and inference.

Sources

  1. DistilBERT, a distilled version of BERT, Sanh et al. (arXiv 1910.01108) (2019)
  2. Green Software Patterns: right-sized, energy-efficient AI models, Green Software Foundation (2025)
  3. Power Hungry Processing: Watts Driving the Cost of AI Deployment?, Luccioni, Jernite and Strubell, ACM FAccT (2024)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy