greenai.guide

Home › How do I…? › How do I choose a smaller AI model?

How do I…?

How do I choose a smaller AI model?

Updated · 1 min read

In short

Define what "good enough" means on real examples, then test models from smallest to largest and stop at the first that passes. Small and task-specific models can use a fraction of the energy of large general ones: UNESCO and University College London report savings of up to 90% on suitable tasks.

Steps

  1. Write down the task and a pass mark, such as the share of answers that must be correct.
  2. Collect 20 to 50 real examples to test against.
  3. Try the smallest or "mini" model first, then move up only if it fails the pass mark.
  4. Check public energy ratings such as the AI Energy Score where they exist.
  5. Keep a larger model for the cases the small one can't handle.
  6. Re-test when models or your task change.

Use the smallest model that meets your quality bar, not the most capable one available. A large general model pays, on every request, for abilities your task may never use.

Why does model size matter so much?

It is usually the biggest single lever on energy. Luccioni, Jernite and Strubell found multi-purpose generative models to be orders of magnitude more expensive to run than task-specific systems for the same tasks. UNESCO and University College London found that small models tailored to specific tasks could cut energy use by up to 90%.

How do I test candidates?

  1. Define the bar on your own work, for example "extracts the invoice total correctly 98% of the time".
  2. Test small first. Many providers offer "mini", "lite" or "flash" versions; open-weight families come in several sizes.
  3. Stop at the first pass. Bigger is not automatically more accurate for narrow tasks.
  4. Compare energy, not just price. Hugging Face's AI Energy Score rates open models from one to five stars on energy per query for common tasks; see read an AI Energy Score rating.

For everyday users, the same idea applies: pick the fast or lighter option in your chatbot for simple questions.

Common mistakes

  • Choosing on reputation rather than testing on your own examples.
  • Measuring cost per call, not per success. A model that is too small and fails, forcing a rerun on a big one, can cost more overall.
  • Never revisiting. Small models improve quickly; re-test every few months.

To send each request to the right size automatically, see route requests to smaller models. The full method is in choose the right model.

Sources

  1. Smarter, smaller, stronger: resource-efficient generative AI and the future of digital transformation, UNESCO and University College London (2025)
  2. Power Hungry Processing: Watts Driving the Cost of AI Deployment?, Luccioni, Jernite and Strubell, ACM FAccT (2024)
  3. AI Energy Score, Hugging Face (2025)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy