In short
Use the smallest model, or the simplest non-AI method, that meets your quality bar, and keep large general models for the cases that need them. Task-specific models can use a small fraction of the energy of general-purpose generative ones, and routing lets you get most of the benefit of both.
Key points
- Start by asking whether the task needs AI at all; rules, search or a classical algorithm often do the job.
- Task-specific models were found to be orders of magnitude cheaper to run than multi-purpose generative models for the same tasks.
- UNESCO and UCL report that small task-specific models can cut energy use by up to 90%.
- Routing sends easy requests to small models and hard ones to large models; set thresholds carefully, because a failed small attempt plus a retry can cost more.
- Hybrid pipelines use classical or physics-based methods for most steps and a small network only where it adds value.
Why does model choice matter so much?
Model choice is usually the biggest single lever on the energy of an AI feature. The same task can be done by models whose running costs differ by orders of magnitude.
In a systematic comparison published at ACM FAccT 2024, Luccioni, Jernite and Strubell measured the energy of running 1,000 inferences across many tasks and models. They found multi-purpose generative models to be orders of magnitude more expensive than task-specific systems for the same tasks, even when model size was controlled. A 2025 UNESCO and UCL report found that small models tailored to specific tasks can cut energy use by up to 90%.
The pattern is simple: a general model pays, on every request, for knowledge and abilities your task never uses.
Does this task need AI at all?
Check first whether a non-AI method already does the job. The cheapest inference is the one you never run.
A 2026 UK green AI sandpit of mathematicians, computer scientists and industry returned to this point repeatedly: people often reach for AI when a well-understood algorithm already exists, and the greenest computation is often the one you avoid. The W3C Web Sustainability Guidelines likewise ask teams to evaluate the need for AI before implementing it.
Ask:
- Could a rule, lookup table or regular expression do it?
- Could search over your own content answer the question?
- Is there an established numerical or statistical method (an optimiser, a regression, a solver)?
- Does the answer need to be exactly reproducible or auditable? Deterministic methods are easier to check.
- How often will this run? A feature used millions of times deserves more design effort than a one-off.
AI is not free even when it saves time. A 2025 study comparing manual and AI-assisted coding on 12 programming tasks found the AI-assisted approach averaged about 33 times the carbon footprint of the manual approach, with the gap widening as tasks became harder. That does not mean never use assistants; it means use them where they clearly pay back.
Small, specialised or frontier?
Match the model to the task, not to the most capable thing available.
| Task type | Usually enough | When to go bigger |
|---|---|---|
| Classifying, tagging, routing, spam or sentiment | A small task-specific classifier | Rarely |
| Extracting fields from documents | A small or fine-tuned model, or rules | Messy, varied layouts |
| Search and question answering over your own content | Retrieval plus a small or mid-sized model | Complex multi-document reasoning |
| Translation, summarisation | A mid-sized or specialised model | High-stakes or nuanced text |
| Open-ended reasoning, planning, hard code | A large general model | This is where it earns its cost |
Specialised small models can come from several routes, covered in efficient training and inference:
- Distillation: a large "teacher" trains a small "student". DistilBERT was 40% smaller than BERT and 60% faster while keeping 97% of its language-understanding ability.
- Fine-tuning a small model on your own task data.
- Quantised versions of an existing model, which need less memory and energy.
Where public benchmarks exist, compare candidates on energy as well as quality. Hugging Face's AI Energy Score rates models on GPU energy per 1,000 queries for ten common tasks, with one to five stars relative to other models on the same task.
How does model routing work?
A router sends each request to the cheapest model likely to answer it well, and escalates only when needed.
Common designs:
- Classifier first: a small model or rule set predicts how hard a request is and picks a model.
- Cascade: try the small model; if its confidence is low or a check fails, pass to a larger one.
- Learned router: trained on examples of which model did well on which request. The RouteLLM study reported cost reductions of more than half in some cases without lowering response quality.
Set the threshold carefully
The sandpit raised an important caveat: in agentic systems, a model that is too small can fail, forcing a rerun on the big model. A failed cheap attempt plus an expensive retry can cost more than going straight to the big model. So:
- measure the end-to-end cost per successful task, not per call;
- tune the escalation threshold on your own data;
- log how often requests escalate, and revisit when models change;
- put the routing decision in the system, not on the end user.
What are hybrid pipelines?
Hybrid pipelines use classical, physics-based or rule-based methods for most steps and a small model only where it adds value. They are often cheaper, more accurate and easier to trust than an end-to-end neural approach.
Examples:
- A physical or numerical model does the simulation; a small learned component replaces only its most expensive part (a "surrogate").
- Deterministic code handles parsing, validation and formatting; a model handles only the free-text judgement.
- A generative model plans or chooses, but the output is assembled from a library of pre-checked components rather than generated afresh each time. See orchestration over generation.
The sandpit also discussed building mathematical or physical structure (symmetries, conservation laws, convexity) into models so that they need fewer parameters and less training. See maths for efficient AI.
Checklist
- [ ] Simplest non-AI method written down and tested first
- [ ] Quality bar defined on your own real cases
- [ ] At least one small or task-specific model tested against it
- [ ] Energy per request compared for each candidate, measured the same way
- [ ] Large model used only where the small one fails the bar
- [ ] Routing cost measured per successful task, including retries
- [ ] Choice recorded, with a date to review it
What to do next
- List your AI features and mark which could use a smaller model, a classical method or no AI.
- Build a small evaluation set of real cases with a pass mark.
- Test two or three model sizes against it and measure the energy of each.
- Pilot routing on one high-volume feature and track cost per successful task.
- Write "use the smallest model that meets the bar" into your green AI policy.
- Ask suppliers which model sizes they offer and their energy per request; see sustainable AI procurement.
Frequently asked questions
Are small language models good enough?
For many well-defined tasks, such as classification, extraction and routing, yes. Test them on your own cases against a clear quality bar before deciding.
What is model routing?
A router looks at each request and sends it to the cheapest model likely to handle it well, escalating to a larger model only when needed. Published routers have cut costs by more than half in some tests without lowering quality.
When should I not use AI?
When a rule, a lookup, a search or an established numerical method already does the task reliably. Those are usually cheaper, faster, more predictable and easier to check.
Is a bigger model always more accurate?
No. For narrow tasks a fine-tuned or distilled small model can match or beat a general model, and its errors are often easier to understand.
Sources
- Power Hungry Processing: Watts Driving the Cost of AI Deployment?, Luccioni, Jernite and Strubell, ACM FAccT 2024 (arXiv 2311.16863) (2024)
- AI large language models: new report shows small changes can reduce energy use by 90%, UNESCO (2025)
- RouteLLM: Learning to Route LLMs with Preference Data, Ong et al. (arXiv 2406.18665) (2024)
- DistilBERT, a distilled version of BERT, Sanh et al. (arXiv 1910.01108) (2019)
- Comparative Analysis of Carbon Footprint in Manual vs. LLM-Assisted Code Development, arXiv 2505.04521 (2025)
- Green Software Patterns: right-sized, energy-efficient AI models, Green Software Foundation (2025)
- AI Energy Score, Hugging Face (2025)
- Web Sustainability Guidelines, W3C (2025)
Related
How to measure the energy and carbon of AI
A practical guide to metering or estimating the energy and carbon of AI work: tools, system boundaries, per-task metrics and honest uncertainty.
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.
Orchestration over generation
Why reusing vetted components and routing requests to the smallest capable model can cut AI's energy use, and what the evidence says so far.
Efficiency is not sufficiency: the rebound effect in AI
Why making AI more efficient does not guarantee it uses less energy overall, what the rebound or Jevons effect is, and what helps keep savings real.