In short
Benchmark your actual model on two or three hardware and runtime options and compare energy per request or per training run at realistic batch sizes, alongside latency and cost. Patterson and colleagues found that the choice of model, data centre and processor together can cut the carbon of training by up to around 100 to 1,000 times. Keep good hardware busy and in service, because making new kit has its own footprint.
Steps
- Benchmark your own model on two or three accelerator and instance options.
- Measure energy per request or per run at your real batch sizes, alongside latency.
- Try more than one inference runtime on each option.
- Choose the option with the lowest energy that meets your latency and quality needs.
- Weigh the embodied carbon of new hardware against the expected saving.
- Re-test when the model or traffic changes.
Choose hardware on measured energy for your workload, not on peak specifications. The same model can use very different energy on different accelerators, instance types and runtimes, and the best choice depends on your batch sizes and traffic.
How do I compare options fairly?
The Green Software Foundation's patterns recommend selecting efficient accelerators and instance types, and efficient frameworks and inference runtimes, on evidence. In practice:
- run the same model, inputs and settings on each candidate;
- measure energy per request (or per run) at the batch sizes you will really use, not just at the best case;
- record latency and quality alongside energy;
- include the runtime: a different serving engine on the same chip can change the result.
Patterson and colleagues found that the choice of model architecture, data centre and processor can reduce the carbon footprint of training by up to around 100 to 1,000 times, so hardware choice belongs with those other decisions.
Should I buy new hardware?
Not automatically. Wu and colleagues at Meta reported that embodied carbon, from making hardware, was roughly 30% of the footprint of Meta's large-scale machine learning. A new accelerator must save enough energy over its life to pay back its manufacture. See accounting for embodied carbon.
Common mistakes
- Underusing big accelerators. A large GPU at low utilisation can use more energy per answer than a smaller one kept busy; see batching AI requests.
- Benchmarking once. Re-test when the model, runtime or traffic changes.
See efficient training and inference.
Sources
- Carbon emissions and large neural network training, Patterson et al. (2021)
- Select efficient accelerators and instance types for AI workloads, Green Software Foundation, Green Software Patterns (2025)
- Select efficient ML frameworks and inference runtimes, Green Software Foundation, Green Software Patterns (2025)
- Sustainable AI: environmental implications, challenges and opportunities, Wu et al., MLSys (2022)
Related
How do I cut the energy of serving a model in production?
How to serve a self-hosted model with less energy per request: efficient runtimes, batching, half precision, compiled caches and tested hardware.
How do I account for the embodied carbon of AI hardware?
Include the emissions from making AI chips and servers: allocate a share by use time, ask suppliers for data, and keep hardware in service longer.
How do I compare the energy use of different AI models?
How to compare AI models on energy fairly: use benchmarks like the AI Energy Score, or measure them yourself on the same task, hardware and settings.
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.