In short
Serve the smallest model that meets your bar, on a runtime built for inference, with batching turned on, weights in half precision or quantised, and the library's optimised attention and cache paths enabled. Then benchmark two or three runtimes and hardware options at your real traffic, and pick on energy per request as well as latency.
Steps
- Confirm the model is the smallest one that meets your quality bar.
- Serve it with an inference engine that supports continuous batching.
- Load weights in half precision or a tested quantised format.
- Turn on efficient attention and a static key-value cache where your library supports them.
- Benchmark two or three runtimes and GPU types at your real batch sizes.
- Scale down or to zero when traffic is low.
Most of a model's lifetime energy is spent serving it, so serving settings matter. Defaults are rarely the most efficient option, and the optimised paths usually have to be switched on.
What makes the biggest difference?
- Batching. An accelerator serving one request at a time is underused but still drawing power. Kwon and colleagues reported that vLLM, with paged key-value cache management, gave 2 to 4 times the throughput of earlier systems at the same latency. See how to batch AI requests.
- Precision. Half precision or a tested quantised model needs less memory and often fewer GPUs.
- Optimised code paths. Hugging Face documents efficient attention implementations and a static key-value
cache combined with
torch.compile, which it says can give up to a 4x speed-up. - Runtime and hardware chosen on evidence. The Green Software Foundation recommends selecting efficient frameworks and inference runtimes; benchmark them rather than using whatever trained the model.
Common mistakes
- Treating speed as energy. GreenCode researchers working on llama.cpp cut GPU energy by 4.7% while speed rose only 3%: the two tell different stories. Measure both.
- Benchmarking at batch size one when production runs at much higher concurrency, or the reverse.
- Leaving replicas running overnight for traffic that isn't there. See how to stop GPUs sitting idle.
- Not warming up compiled models before measuring, which distorts the numbers.
For the full toolkit, read efficient training and inference.
Sources
- Optimizing inference, Hugging Face Transformers documentation (2026)
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. (arXiv 2309.06180) (2023)
- GreenCode Constitution: agent principles for optimizing code and LLM inference energy use, Kapenekakis, Kim, Ionescu, Thomsen and Albano, GreenAI workshop, ECML PKDD 2026 (2026)
- Green Software Patterns: select efficient ML frameworks and inference runtimes, Green Software Foundation (2025)
Related
How do I batch AI requests to save energy?
How batching raises accelerator utilisation and cuts energy per answer: continuous batching for live traffic and queued batches for offline work.
How do I quantise a model?
How to quantise an AI model to 8 or 4 bits: which tools to use, how to test quality, and when quantisation saves energy and when it doesn't.
How do I speed up a large model with speculative decoding?
How speculative decoding lets a small draft model propose tokens that a large model checks, giving faster answers with identical outputs.
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.