# How do I cut the energy of serving a model in production?

> How to serve a self-hosted model with less energy per request: efficient runtimes, batching, half precision, compiled caches and tested hardware.

Source: https://greenai.guide/how-do-i/cut-the-energy-of-serving-a-model/  
Updated: 2026-11-19

## In short

Serve the smallest model that meets your bar, on a runtime built for inference, with batching turned on, weights in half precision or quantised, and the library's optimised attention and cache paths enabled. Then benchmark two or three runtimes and hardware options at your real traffic, and pick on energy per request as well as latency.

## Steps

1. Confirm the model is the smallest one that meets your quality bar.
2. Serve it with an inference engine that supports continuous batching.
3. Load weights in half precision or a tested quantised format.
4. Turn on efficient attention and a static key-value cache where your library supports them.
5. Benchmark two or three runtimes and GPU types at your real batch sizes.
6. Scale down or to zero when traffic is low.

**Most of a model's lifetime energy is spent serving it, so serving settings matter.** Defaults are rarely the most
efficient option, and the optimised paths usually have to be switched on.

## What makes the biggest difference?

1. **Batching.** An accelerator serving one request at a time is underused but still drawing power. Kwon and
   colleagues reported that vLLM, with paged key-value cache management, gave 2 to 4 times the throughput of
   earlier systems at the same latency. See [how to batch AI requests](/how-do-i/batch-ai-requests/).
2. **Precision.** Half precision or a tested [quantised](/how-do-i/quantise-a-model/) model needs less memory and
   often fewer GPUs.
3. **Optimised code paths.** Hugging Face documents efficient attention implementations and a static key-value
   cache combined with `torch.compile`, which it says can give up to a 4x speed-up.
4. **Runtime and hardware chosen on evidence.** The Green Software Foundation recommends selecting efficient
   frameworks and inference runtimes; benchmark them rather than using whatever trained the model.

## Common mistakes

- **Treating speed as energy.** GreenCode researchers working on llama.cpp cut GPU energy by 4.7% while speed rose
  only 3%: the two tell different stories. Measure both.
- **Benchmarking at batch size one** when production runs at much higher concurrency, or the reverse.
- **Leaving replicas running overnight** for traffic that isn't there. See
  [how to stop GPUs sitting idle](/how-do-i/stop-gpus-sitting-idle/).
- **Not warming up compiled models** before measuring, which distorts the numbers.

For the full toolkit, read [efficient training and inference](/practice/efficient-training-and-inference/).

## Sources

- [Optimizing inference](https://huggingface.co/docs/transformers/main/en/llm_optims), Hugging Face Transformers documentation (2026)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180), Kwon et al. (arXiv 2309.06180) (2023)
- [GreenCode Constitution: agent principles for optimizing code and LLM inference energy use](https://greencode.ai/publications/greencode-constitution-agent-principles-for-optimizing-code-and-llm-inference-en/), Kapenekakis, Kim, Ionescu, Thomsen and Albano, GreenAI workshop, ECML PKDD 2026 (2026)
- [Green Software Patterns: select efficient ML frameworks and inference runtimes](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/development/select-efficient-ml-frameworks-inference-runtimes.md), Green Software Foundation (2025)
