greenai.guide

Home › How do I…? › How do I cut the energy of hyperparameter searches?

How do I…?

How do I cut the energy of hyperparameter searches?

Updated · 1 min read

In short

Replace large grid searches with random or adaptive search, stop poor trials early, run pilots on small data or small models first, and start from published defaults for your architecture. Track energy for every trial, not just the winner, and set a compute budget before you start. Tuning runs often cost more than the final training run, and they rarely get reported.

Steps

  1. Set a compute budget for the search before you begin.
  2. Start from published default hyperparameters for your model family.
  3. Use random or adaptive search instead of a full grid.
  4. Stop trials early when they are clearly worse than the best so far.
  5. Pilot on a subset of data or a smaller model before scaling up.
  6. Record energy and emissions for every trial alongside accuracy.

Tune smarter, not wider. A full grid over many hyperparameters multiplies the cost of training by the number of combinations, and most of those runs are clearly worse early on.

Why does it matter?

Strubell, Ganesh and McCallum's 2019 paper quantified the cost of developing language models, including the experimentation around them, not just a single final training run. Schwartz and colleagues' "Green AI" argued that researchers should report the computational cost of their results, including the experiments it took to reach them, so that efficiency is valued alongside accuracy.

What makes the biggest difference?

  • Early stopping of poor trials. Successive-halving style methods give every trial a small budget and continue only the promising ones. Most tuning libraries support this.
  • Random or adaptive search. Instead of running every combination, sample settings at random or let an adaptive method use earlier results to choose the next trial.
  • Small pilots. Rule out ideas on a subset of data or a smaller model.
  • Defaults first. Well-tested defaults for common architectures are often close to optimal.
  • Track every trial. CodeCarbon's documentation shows how to track each configuration and compare accuracy against emissions.

Common mistakes

For more, read the guide for researchers.

Sources

  1. Energy and policy considerations for deep learning in NLP, Strubell, Ganesh and McCallum, ACL (2019)
  2. Green AI, Schwartz, Dodge, Smith and Etzioni, Communications of the ACM (2020)
  3. Comparing model efficiency with carbon tracking, CodeCarbon documentation (2025)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy