# Efficient AI training and inference

> Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.

Source: https://greenai.guide/practice/efficient-training-and-inference/  
Updated: 2026-11-19

## In short

Most AI energy can be cut without losing quality by reusing models instead of training from scratch, shrinking them with pruning, quantisation and distillation, serving them with batching and caching, and running flexible jobs when and where electricity is cleaner. Measure energy before and after every change, because faster is not always greener.

## Key points

- Reuse before you train: fine-tuning a small adapter with LoRA cut trainable parameters 10,000-fold and GPU memory threefold against full fine-tuning of GPT-3 175B.
- Large language models can be pruned to at least 50% sparsity in one shot without retraining, at minimal loss of accuracy (SparseGPT).
- 8-bit inference can halve memory needs without loss of quality (LLM.int8()); test 4-bit too.
- Serving matters: batching and paged KV-cache management gave 2–4 times the throughput in vLLM at the same latency.
- Carbon-aware scheduling cuts emissions, not energy; do it on top of efficiency work, never instead of it.

## Where should I start?

**Start by measuring, then reuse, shrink, serve well and schedule, in that order.** The early steps usually save the
most and carry the least risk.

Patterson and colleagues at Google estimated in 2021 that the choice of model, data centre and processor can
change the carbon footprint of training by up to around 100 to 1,000 times. Most of that range is in your hands.
Before you change anything, set up [energy measurement](/practice/measure-ai-energy/) on the same hardware and load,
so you can keep only the changes that save energy without hurting quality. Speed is not a reliable stand-in: a
change can be faster and hungrier, or slower and leaner.

## Can I avoid training from scratch?

**Usually, yes: start from a pre-trained model and adapt only what you need.**

- **Parameter-efficient fine-tuning.** LoRA trains small low-rank adapter matrices instead of the full model.
  Against full fine-tuning of GPT-3 175B, its authors reported 10,000 times fewer trainable parameters and three
  times less GPU memory. QLoRA combines this with 4-bit weights and fine-tuned a 65-billion-parameter model on a
  single 48 GB GPU.
- **Retrieval instead of retraining.** Put facts that change in a searchable index and retrieve them at query time.
  Fine-tune for skills and style, not for facts.
- **Model merging ("souping").** Averaging the weights of models fine-tuned from the same base can beat the best
  single model with no extra inference cost, according to Wortsman and colleagues. Merging specialist fine-tunes is
  a way of reusing work already paid for.
- **Continual learning.** Update a model incrementally rather than retraining from scratch when data changes.

## How do pruning and sparsity help?

**Most weights in a trained network can be removed or left inactive with little loss, which means less computation
and memory per answer.**

- **Unstructured pruning** removes individual weights. SparseGPT pruned GPT-family models to at least 50% sparsity
  in one shot without retraining, at minimal loss of accuracy, and reached 60% on the largest open models of the
  time with negligible increase in perplexity.
- **Structured pruning** removes whole neurons, heads or layers, which ordinary hardware can exploit more easily.
- **Activation sparsity** leaves most neurons inactive for a given input.
- **Mixture-of-experts** models send each token to a few specialist sub-networks. Patterson and colleagues found
  large, sparsely activated models can use less than a tenth of the energy of large dense ones. All experts still
  have to sit in memory.

A caveat from a 2026 UK green AI sandpit of mathematicians, computer scientists and industry: sparsity is
under-used partly because hardware and libraries are built for dense maths. Check that your runtime actually turns
sparsity into saved energy. The sandpit also discussed using random matrix theory to choose sparsity levels before
training, and sparse approximations of attention that can be dropped into pre-trained models. See
[maths for efficient AI](/articles/maths-for-efficient-ai/).

## What about quantisation, distillation and mixed precision?

**Use fewer bits and smaller models wherever the quality bar allows.**

| Technique | What it does | Published result |
|---|---|---|
| [Quantisation](/glossary/#quantisation) | Stores and computes weights in 8 or 4 bits instead of 16 or 32 | LLM.int8() halved inference memory with no loss of quality up to 175B parameters; UNESCO and UCL report model compression such as quantisation saving up to 44% of energy |
| [Distillation](/glossary/#distillation) | Trains a small "student" to imitate a large "teacher" | DistilBERT was 40% smaller and 60% faster than BERT while keeping 97% of its language understanding |
| Mixed precision | Trains in half precision with a full-precision master copy | Nearly halved memory use in training (Micikevicius et al.) |

Test each against your own evaluation set. Hugging Face notes that quantisation can add latency when a workload is
not limited by memory, so measure energy and latency per request before and after.

## How do I serve models efficiently?

**Keep the accelerator busy with useful work and avoid recomputing anything.** An underused GPU still draws
substantial power.

1. **Batch requests.** Serving engines with continuous batching group concurrent requests. vLLM's PagedAttention,
   which manages the [KV cache](/glossary/#kv-cache) like virtual memory, gave 2 to 4 times the throughput of
   earlier systems at the same latency.
2. **Use the KV cache well.** Cache keys and values for earlier tokens; use a static cache with a compiled forward
   pass where your library supports it.
3. **Turn on the optimised paths.** Half-precision weights, efficient attention kernels (such as SDPA or
   FlashAttention) and compilation are often available but off by default.
4. **Speculative decoding.** A small draft model proposes tokens that the large model checks. Leviathan and
   colleagues reported 2 to 3 times faster generation on T5-XXL with identical outputs.
5. **Cap output length** to what the task needs, and cache answers to identical requests.
6. **Compare runtimes and hardware.** Benchmark two or three serving stacks at your real batch sizes.

## When and where should jobs run?

**Run flexible work when and where the electricity is cleaner, but treat this as a bonus on top of efficiency, not a
substitute for it.**

- **Time-shift.** Training runs, evaluations and batch inference can often wait hours. Schedule them against a
  carbon-intensity forecast; tools such as CATS wrap cluster jobs to do this. In Great Britain, the Carbon
  Intensity API publishes regional forecasts.
- **Make long jobs pausable.** Checkpoint regularly, including optimiser state, so a job can pause when the grid is
  dirty and resume later without lost work.
- **Choose region and hardware.** Where latency doesn't matter, prefer a low-carbon region and efficient
  accelerators, allowing for the cost of moving data. See [hosting and data centres](/practice/hosting-and-data-centres/).
- **Watch absolute energy.** Carbon-aware scheduling moves when work runs, not how much it uses. Track energy per
  unit of work alongside carbon.

## Checklist

- [ ] Energy measured before and after each change, same set-up, several runs
- [ ] Pre-trained model and adapters considered before training from scratch
- [ ] Retrieval used for changing facts instead of retraining
- [ ] Pruning, quantisation or distillation tested against the quality bar
- [ ] Mixed precision used in training
- [ ] Batching, KV-cache optimisation and efficient attention on in serving
- [ ] Output length capped; identical requests cached
- [ ] Flexible jobs checkpointed and scheduled for cleaner hours
- [ ] Training energy and hardware recorded in the model card

## What to do next

1. Add an emissions tracker such as CodeCarbon to your training and evaluation scripts, and put energy next to
   accuracy in your results table.
2. Try LoRA or QLoRA on your next fine-tune and compare with full fine-tuning.
3. Test an 8-bit and a 4-bit version of your deployed model against your evaluation set.
4. Benchmark your serving stack with batching and the optimised attention path switched on.
5. Wrap one recurring cluster job in a carbon-aware scheduler.
6. Look for training in the [courses directory](/courses/) and related work in [research](/research/) and
   [PhDs](/phds/).

## Frequently asked questions

### Does quantisation reduce accuracy?

Sometimes, slightly; often not noticeably for 8-bit. Always test the quantised model on your own evaluation set, and note that quantisation can add latency when a workload is not limited by memory.

### What is a KV cache?

When a transformer generates text, it stores the key and value matrices for earlier tokens so they are not recomputed at every step. Managing that cache well is one of the biggest serving efficiencies.

### Is training or inference the bigger energy cost?

It depends on how widely a model is used. Training is a large one-off cost; inference is paid on every request, so for popular models it usually dominates over the model's life.

### What is model merging or souping?

Averaging the weights of several models fine-tuned from the same starting point. The result runs as one model, with no extra inference cost, and can outperform the best individual model.

## Sources

- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685), Hu et al. (arXiv 2106.09685) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314), Dettmers et al. (arXiv 2305.14314) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774), Frantar and Alistarh (arXiv 2301.00774) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339), Dettmers et al. (arXiv 2208.07339) (2022)
- [DistilBERT, a distilled version of BERT](https://arxiv.org/abs/1910.01108), Sanh et al. (arXiv 1910.01108) (2019)
- [Mixed Precision Training](https://arxiv.org/abs/1710.03740), Micikevicius et al. (arXiv 1710.03740) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180), Kwon et al. (arXiv 2309.06180) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192), Leviathan et al. (arXiv 2211.17192) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://arxiv.org/abs/2203.05482), Wortsman et al. (arXiv 2203.05482) (2022)
- [Carbon Emissions and Large Neural Network Training](https://arxiv.org/abs/2104.10350), Patterson et al. (arXiv 2104.10350) (2021)
- [AI large language models: new report shows small changes can reduce energy use by 90%](https://www.unesco.org/en/articles/ai-large-language-models-new-report-shows-small-changes-can-reduce-energy-use-90), UNESCO (2025)
- [Optimizing inference](https://huggingface.co/docs/transformers/main/en/llm_optims), Hugging Face Transformers documentation (2025)
- [Green Software Patterns: carbon-aware scheduling and region selection for AI workloads](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/operations/carbon-aware-ai-scheduling.md), Green Software Foundation (2025)
- [CATS: Climate-Aware Task Scheduler](https://github.com/GreenScheduler/cats), GreenScheduler (2025)
