greenai.guide

Home › Good practice › Efficient AI training and inference

Good practice

Efficient AI training and inference

Updated · 5 min read

In short

Most AI energy can be cut without losing quality by reusing models instead of training from scratch, shrinking them with pruning, quantisation and distillation, serving them with batching and caching, and running flexible jobs when and where electricity is cleaner. Measure energy before and after every change, because faster is not always greener.

Key points

  • Reuse before you train: fine-tuning a small adapter with LoRA cut trainable parameters 10,000-fold and GPU memory threefold against full fine-tuning of GPT-3 175B.
  • Large language models can be pruned to at least 50% sparsity in one shot without retraining, at minimal loss of accuracy (SparseGPT).
  • 8-bit inference can halve memory needs without loss of quality (LLM.int8()); test 4-bit too.
  • Serving matters: batching and paged KV-cache management gave 2–4 times the throughput in vLLM at the same latency.
  • Carbon-aware scheduling cuts emissions, not energy; do it on top of efficiency work, never instead of it.

Where should I start?

Start by measuring, then reuse, shrink, serve well and schedule, in that order. The early steps usually save the most and carry the least risk.

Patterson and colleagues at Google estimated in 2021 that the choice of model, data centre and processor can change the carbon footprint of training by up to around 100 to 1,000 times. Most of that range is in your hands. Before you change anything, set up energy measurement on the same hardware and load, so you can keep only the changes that save energy without hurting quality. Speed is not a reliable stand-in: a change can be faster and hungrier, or slower and leaner.

Can I avoid training from scratch?

Usually, yes: start from a pre-trained model and adapt only what you need.

  • Parameter-efficient fine-tuning. LoRA trains small low-rank adapter matrices instead of the full model. Against full fine-tuning of GPT-3 175B, its authors reported 10,000 times fewer trainable parameters and three times less GPU memory. QLoRA combines this with 4-bit weights and fine-tuned a 65-billion-parameter model on a single 48 GB GPU.
  • Retrieval instead of retraining. Put facts that change in a searchable index and retrieve them at query time. Fine-tune for skills and style, not for facts.
  • Model merging ("souping"). Averaging the weights of models fine-tuned from the same base can beat the best single model with no extra inference cost, according to Wortsman and colleagues. Merging specialist fine-tunes is a way of reusing work already paid for.
  • Continual learning. Update a model incrementally rather than retraining from scratch when data changes.

How do pruning and sparsity help?

Most weights in a trained network can be removed or left inactive with little loss, which means less computation and memory per answer.

  • Unstructured pruning removes individual weights. SparseGPT pruned GPT-family models to at least 50% sparsity in one shot without retraining, at minimal loss of accuracy, and reached 60% on the largest open models of the time with negligible increase in perplexity.
  • Structured pruning removes whole neurons, heads or layers, which ordinary hardware can exploit more easily.
  • Activation sparsity leaves most neurons inactive for a given input.
  • Mixture-of-experts models send each token to a few specialist sub-networks. Patterson and colleagues found large, sparsely activated models can use less than a tenth of the energy of large dense ones. All experts still have to sit in memory.

A caveat from a 2026 UK green AI sandpit of mathematicians, computer scientists and industry: sparsity is under-used partly because hardware and libraries are built for dense maths. Check that your runtime actually turns sparsity into saved energy. The sandpit also discussed using random matrix theory to choose sparsity levels before training, and sparse approximations of attention that can be dropped into pre-trained models. See maths for efficient AI.

What about quantisation, distillation and mixed precision?

Use fewer bits and smaller models wherever the quality bar allows.

Technique What it does Published result
Quantisation Stores and computes weights in 8 or 4 bits instead of 16 or 32 LLM.int8() halved inference memory with no loss of quality up to 175B parameters; UNESCO and UCL report model compression such as quantisation saving up to 44% of energy
Distillation Trains a small "student" to imitate a large "teacher" DistilBERT was 40% smaller and 60% faster than BERT while keeping 97% of its language understanding
Mixed precision Trains in half precision with a full-precision master copy Nearly halved memory use in training (Micikevicius et al.)

Test each against your own evaluation set. Hugging Face notes that quantisation can add latency when a workload is not limited by memory, so measure energy and latency per request before and after.

How do I serve models efficiently?

Keep the accelerator busy with useful work and avoid recomputing anything. An underused GPU still draws substantial power.

  1. Batch requests. Serving engines with continuous batching group concurrent requests. vLLM's PagedAttention, which manages the KV cache like virtual memory, gave 2 to 4 times the throughput of earlier systems at the same latency.
  2. Use the KV cache well. Cache keys and values for earlier tokens; use a static cache with a compiled forward pass where your library supports it.
  3. Turn on the optimised paths. Half-precision weights, efficient attention kernels (such as SDPA or FlashAttention) and compilation are often available but off by default.
  4. Speculative decoding. A small draft model proposes tokens that the large model checks. Leviathan and colleagues reported 2 to 3 times faster generation on T5-XXL with identical outputs.
  5. Cap output length to what the task needs, and cache answers to identical requests.
  6. Compare runtimes and hardware. Benchmark two or three serving stacks at your real batch sizes.

When and where should jobs run?

Run flexible work when and where the electricity is cleaner, but treat this as a bonus on top of efficiency, not a substitute for it.

  • Time-shift. Training runs, evaluations and batch inference can often wait hours. Schedule them against a carbon-intensity forecast; tools such as CATS wrap cluster jobs to do this. In Great Britain, the Carbon Intensity API publishes regional forecasts.
  • Make long jobs pausable. Checkpoint regularly, including optimiser state, so a job can pause when the grid is dirty and resume later without lost work.
  • Choose region and hardware. Where latency doesn't matter, prefer a low-carbon region and efficient accelerators, allowing for the cost of moving data. See hosting and data centres.
  • Watch absolute energy. Carbon-aware scheduling moves when work runs, not how much it uses. Track energy per unit of work alongside carbon.

Checklist

  • [ ] Energy measured before and after each change, same set-up, several runs
  • [ ] Pre-trained model and adapters considered before training from scratch
  • [ ] Retrieval used for changing facts instead of retraining
  • [ ] Pruning, quantisation or distillation tested against the quality bar
  • [ ] Mixed precision used in training
  • [ ] Batching, KV-cache optimisation and efficient attention on in serving
  • [ ] Output length capped; identical requests cached
  • [ ] Flexible jobs checkpointed and scheduled for cleaner hours
  • [ ] Training energy and hardware recorded in the model card

What to do next

  1. Add an emissions tracker such as CodeCarbon to your training and evaluation scripts, and put energy next to accuracy in your results table.
  2. Try LoRA or QLoRA on your next fine-tune and compare with full fine-tuning.
  3. Test an 8-bit and a 4-bit version of your deployed model against your evaluation set.
  4. Benchmark your serving stack with batching and the optimised attention path switched on.
  5. Wrap one recurring cluster job in a carbon-aware scheduler.
  6. Look for training in the courses directory and related work in research and PhDs.

Frequently asked questions

Does quantisation reduce accuracy?

Sometimes, slightly; often not noticeably for 8-bit. Always test the quantised model on your own evaluation set, and note that quantisation can add latency when a workload is not limited by memory.

What is a KV cache?

When a transformer generates text, it stores the key and value matrices for earlier tokens so they are not recomputed at every step. Managing that cache well is one of the biggest serving efficiencies.

Is training or inference the bigger energy cost?

It depends on how widely a model is used. Training is a large one-off cost; inference is paid on every request, so for popular models it usually dominates over the model's life.

What is model merging or souping?

Averaging the weights of several models fine-tuned from the same starting point. The result runs as one model, with no extra inference cost, and can outperform the best individual model.

Sources

  1. LoRA: Low-Rank Adaptation of Large Language Models, Hu et al. (arXiv 2106.09685) (2021)
  2. QLoRA: Efficient Finetuning of Quantized LLMs, Dettmers et al. (arXiv 2305.14314) (2023)
  3. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, Frantar and Alistarh (arXiv 2301.00774) (2023)
  4. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, Dettmers et al. (arXiv 2208.07339) (2022)
  5. DistilBERT, a distilled version of BERT, Sanh et al. (arXiv 1910.01108) (2019)
  6. Mixed Precision Training, Micikevicius et al. (arXiv 1710.03740) (2017)
  7. Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. (arXiv 2309.06180) (2023)
  8. Fast Inference from Transformers via Speculative Decoding, Leviathan et al. (arXiv 2211.17192) (2022)
  9. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, Wortsman et al. (arXiv 2203.05482) (2022)
  10. Carbon Emissions and Large Neural Network Training, Patterson et al. (arXiv 2104.10350) (2021)
  11. AI large language models: new report shows small changes can reduce energy use by 90%, UNESCO (2025)
  12. Optimizing inference, Hugging Face Transformers documentation (2025)
  13. Green Software Patterns: carbon-aware scheduling and region selection for AI workloads, Green Software Foundation (2025)
  14. CATS: Climate-Aware Task Scheduler, GreenScheduler (2025)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy