# How do I batch AI requests to save energy?

> How batching raises accelerator utilisation and cuts energy per answer: continuous batching for live traffic and queued batches for offline work.

Source: https://greenai.guide/how-do-i/batch-ai-requests/  
Updated: 2026-11-19

## In short

Serve live traffic with an inference engine that uses continuous batching, so concurrent requests share the accelerator. Queue work that doesn't need an instant answer, such as overnight document processing, and run it in large batches, ideally when the grid is cleaner. An accelerator serving one request at a time is underused but still draws substantial power.

## Steps

1. Separate interactive requests from work that can wait minutes or hours.
2. Serve interactive traffic with an engine that supports continuous batching.
3. Queue non-urgent work and process it in batches.
4. Use your provider's batch interface for hosted models where one exists.
5. Measure energy per request at real batch sizes, alongside latency.

**Batching lets one pass of the hardware serve many requests.** Energy per answer depends heavily on how busy the
accelerator is.

## Why does it matter?

GPUs draw substantial power even when lightly loaded, so serving requests one at a time spreads that power over very
few answers. Kwon and colleagues reported that vLLM, with continuous batching and paged key-value cache management,
improved throughput 2 to 4 times at the same latency compared with earlier systems. Hugging Face's inference
documentation also covers continuous batching.

## Two kinds of batching

- **Continuous batching for live traffic.** The serving engine adds new requests to a running batch as others
  finish, so users don't wait for a batch to fill. Use an engine that supports it rather than a simple
  one-request-per-call loop.
- **Offline batches for work that can wait.** Summarising yesterday's documents, tagging an archive or scoring a
  dataset doesn't need instant answers. Queue it and process in bulk. The Green Software Foundation recommends
  running batch AI workloads in low-carbon regions and times where latency is less critical; see
  [how to schedule work when the grid is greener](/how-do-i/schedule-training-when-the-grid-is-greener/).

## Common mistakes

- **Batches so large they hurt latency** for interactive users. Tune on real traffic.
- **Padding waste.** Grouping very short and very long inputs together wastes computation in some engines.
- **Leaving batch servers running** between jobs; see [how to stop GPUs sitting idle](/how-do-i/stop-gpus-sitting-idle/).

Read more in [efficient training and inference](/practice/efficient-training-and-inference/).

## Sources

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180), Kwon et al. (arXiv 2309.06180) (2023)
- [Optimizing inference](https://huggingface.co/docs/transformers/main/en/llm_optims), Hugging Face Transformers documentation (2026)
- [Use carbon-aware scheduling and region selection for AI workloads](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/operations/carbon-aware-ai-scheduling.md), Green Software Foundation, Green Software Patterns (2025)
