# How do I speed up a large model with speculative decoding?

> How speculative decoding lets a small draft model propose tokens that a large model checks, giving faster answers with identical outputs.

Source: https://greenai.guide/how-do-i/use-speculative-decoding/  
Updated: 2026-11-19

## In short

Pair your large model with a small draft model from the same family: the draft proposes several tokens and the large model checks them in one pass, keeping only those it agrees with. The output is the same as the large model alone, and the original study reported 2 to 3 times faster inference. Check your serving stack supports it, then measure energy per request, not just speed.

## Steps

1. Check whether your inference library or serving engine supports speculative or assisted decoding.
2. Choose a small draft model that shares the large model's tokeniser.
3. Turn it on and confirm outputs match on your own workload.
4. Measure energy per request and latency at your real batch sizes.
5. Keep it only if energy per request falls, not just latency.

**Let a small model do the guessing and the large model do the checking.** Large models generate one token at a
time, keeping big hardware busy for every word; speculative decoding lets them confirm several tokens at once.

## How does it work?

A small **draft model** quickly proposes the next few tokens. The large **target model** then checks all of them in
a single forward pass and accepts those it would have produced itself. Where it disagrees, it substitutes its own
token and drafting resumes. Leviathan, Kalman and Matias showed a 2 to 3 times speed-up on the T5-XXL model with
identical outputs. Hugging Face's documentation notes there is no degradation in accuracy, because the verification
pass ensures the same outputs are generated.

Variants include prompt lookup decoding, which drafts from text already in the prompt, and suits tasks such as
editing or summarising where the answer repeats the input.

## Does faster mean greener?

**Not automatically.** You now run two models, and the large one does extra checking work. Whether energy per
request falls depends on how often drafts are accepted, your batch size and your hardware. GreenCode researchers
found on llama.cpp that energy and speed can tell different stories: one change saved 4.7% GPU energy with only a 3%
speed gain. Measure energy directly.

## Common mistakes

- **A draft model too large**, which slows everything down.
- **Mismatched tokenisers** between draft and target.
- **Testing on a different workload** from production; acceptance rates vary by task.

For other serving techniques, see [how to cut the energy of serving a model](/how-do-i/cut-the-energy-of-serving-a-model/)
and [efficient training and inference](/practice/efficient-training-and-inference/).

## Sources

- [Fast inference from transformers via speculative decoding](https://arxiv.org/abs/2211.17192), Leviathan, Kalman and Matias, ICML 2023 (2022)
- [Optimizing inference](https://huggingface.co/docs/transformers/main/en/llm_optims), Hugging Face Transformers documentation (2026)
- [GreenCode Constitution: agent principles for optimizing code and LLM inference energy use](https://greencode.ai/publications/greencode-constitution-agent-principles-for-optimizing-code-and-llm-inference-en/), Kapenekakis, Kim, Ionescu, Thomsen and Albano, GreenAI workshop, ECML PKDD 2026 (2026)
