In short
Serve live traffic with an inference engine that uses continuous batching, so concurrent requests share the accelerator. Queue work that doesn't need an instant answer, such as overnight document processing, and run it in large batches, ideally when the grid is cleaner. An accelerator serving one request at a time is underused but still draws substantial power.
Steps
- Separate interactive requests from work that can wait minutes or hours.
- Serve interactive traffic with an engine that supports continuous batching.
- Queue non-urgent work and process it in batches.
- Use your provider's batch interface for hosted models where one exists.
- Measure energy per request at real batch sizes, alongside latency.
Batching lets one pass of the hardware serve many requests. Energy per answer depends heavily on how busy the accelerator is.
Why does it matter?
GPUs draw substantial power even when lightly loaded, so serving requests one at a time spreads that power over very few answers. Kwon and colleagues reported that vLLM, with continuous batching and paged key-value cache management, improved throughput 2 to 4 times at the same latency compared with earlier systems. Hugging Face's inference documentation also covers continuous batching.
Two kinds of batching
- Continuous batching for live traffic. The serving engine adds new requests to a running batch as others finish, so users don't wait for a batch to fill. Use an engine that supports it rather than a simple one-request-per-call loop.
- Offline batches for work that can wait. Summarising yesterday's documents, tagging an archive or scoring a dataset doesn't need instant answers. Queue it and process in bulk. The Green Software Foundation recommends running batch AI workloads in low-carbon regions and times where latency is less critical; see how to schedule work when the grid is greener.
Common mistakes
- Batches so large they hurt latency for interactive users. Tune on real traffic.
- Padding waste. Grouping very short and very long inputs together wastes computation in some engines.
- Leaving batch servers running between jobs; see how to stop GPUs sitting idle.
Read more in efficient training and inference.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. (arXiv 2309.06180) (2023)
- Optimizing inference, Hugging Face Transformers documentation (2026)
- Use carbon-aware scheduling and region selection for AI workloads, Green Software Foundation, Green Software Patterns (2025)
Related
How do I cut the energy of serving a model in production?
How to serve a self-hosted model with less energy per request: efficient runtimes, batching, half precision, compiled caches and tested hardware.
How do I schedule training when the grid is greener?
Run training and batch AI jobs when grid carbon intensity is low: use forecasts, make jobs pausable with checkpoints, and keep deadlines as hard limits.
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.
How do I stop GPUs and notebooks sitting idle?
Stop idle AI hardware wasting energy: start resources on demand, auto-stop idle notebooks, scale to zero out of hours and report idle GPU-hours.