In short
Pair your large model with a small draft model from the same family: the draft proposes several tokens and the large model checks them in one pass, keeping only those it agrees with. The output is the same as the large model alone, and the original study reported 2 to 3 times faster inference. Check your serving stack supports it, then measure energy per request, not just speed.
Steps
- Check whether your inference library or serving engine supports speculative or assisted decoding.
- Choose a small draft model that shares the large model's tokeniser.
- Turn it on and confirm outputs match on your own workload.
- Measure energy per request and latency at your real batch sizes.
- Keep it only if energy per request falls, not just latency.
Let a small model do the guessing and the large model do the checking. Large models generate one token at a time, keeping big hardware busy for every word; speculative decoding lets them confirm several tokens at once.
How does it work?
A small draft model quickly proposes the next few tokens. The large target model then checks all of them in a single forward pass and accepts those it would have produced itself. Where it disagrees, it substitutes its own token and drafting resumes. Leviathan, Kalman and Matias showed a 2 to 3 times speed-up on the T5-XXL model with identical outputs. Hugging Face's documentation notes there is no degradation in accuracy, because the verification pass ensures the same outputs are generated.
Variants include prompt lookup decoding, which drafts from text already in the prompt, and suits tasks such as editing or summarising where the answer repeats the input.
Does faster mean greener?
Not automatically. You now run two models, and the large one does extra checking work. Whether energy per request falls depends on how often drafts are accepted, your batch size and your hardware. GreenCode researchers found on llama.cpp that energy and speed can tell different stories: one change saved 4.7% GPU energy with only a 3% speed gain. Measure energy directly.
Common mistakes
- A draft model too large, which slows everything down.
- Mismatched tokenisers between draft and target.
- Testing on a different workload from production; acceptance rates vary by task.
For other serving techniques, see how to cut the energy of serving a model and efficient training and inference.
Sources
- Fast inference from transformers via speculative decoding, Leviathan, Kalman and Matias, ICML 2023 (2022)
- Optimizing inference, Hugging Face Transformers documentation (2026)
- GreenCode Constitution: agent principles for optimizing code and LLM inference energy use, Kapenekakis, Kim, Ionescu, Thomsen and Albano, GreenAI workshop, ECML PKDD 2026 (2026)
Related
How do I cut the energy of serving a model in production?
How to serve a self-hosted model with less energy per request: efficient runtimes, batching, half precision, compiled caches and tested hardware.
How do I batch AI requests to save energy?
How batching raises accelerator utilisation and cuts energy per answer: continuous batching for live traffic and queued batches for offline work.
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.
The maths that makes AI more efficient
Pruning, sparsity, sketching, random matrix theory and physics-informed models: how mathematics cuts the computation, memory and energy AI needs.