In short
Put a router in front of your models: a rule, a small classifier or a cascade that tries a small model first and escalates when confidence is low or a check fails. Published routers have cut costs by more than half in some tests without lowering quality. Tune the threshold on your own data and measure cost per successful task, not per call.
Steps
- Build an evaluation set of real requests with a pass mark for each.
- Measure how often a small model passes on its own.
- Choose a router design, such as rules, a difficulty classifier or a cascade with a check.
- Tune the escalation threshold on your evaluation set.
- Track cost and energy per successful task, including escalations and retries.
- Re-tune when either model changes.
Routing gives you most of the quality of a large model at much of the cost of a small one. The idea is simple: send each request to the cheapest model likely to handle it well.
Which design should I use?
- Rules. Route by request type, length or channel. Easy to understand, and often enough.
- Difficulty classifier. A small model predicts how hard a request is. Ding and colleagues' Hybrid LLM work routes queries between a small and a large model this way, balancing cost and quality.
- Cascade. Try the small model; if a check fails or confidence is low, pass to a larger one. FrugalGPT, by Chen, Zaharia and Zou, studied cascades like this for cutting cost.
- Learned router. Trained on which model did well on which request. The RouteLLM study reported cost reductions of more than half in some cases without lowering response quality.
Common mistakes
- Measuring per call. A failed small attempt plus a retry on the big model can cost more than going straight to the big one. A 2026 UK green AI sandpit of mathematicians, computer scientists and industry flagged this as a real risk in agentic systems. Measure per successful task.
- Pushing the choice onto users. Put routing in the system, not in a drop-down most people ignore.
- Setting and forgetting. Log escalation rates and revisit when models change.
What makes the biggest difference?
A good evaluation set. Without real examples and a pass mark, you cannot tune the threshold or prove the saving. Read choose the right model for how to build one.
Sources
- RouteLLM: learning to route LLMs with preference data, Ong et al., ICLR 2025 (2024)
- FrugalGPT: how to use large language models while reducing cost and improving performance, Chen, Zaharia and Zou, Transactions on Machine Learning Research (2023)
- Hybrid LLM: cost-efficient and quality-aware query routing, Ding et al., ICLR (2024)
Related
Choose the right model for the job
How to right-size AI: when to use small or specialised models, when a frontier model earns its cost, how routing works, and when not to use AI.
Orchestration over generation
Why reusing vetted components and routing requests to the smallest capable model can cut AI's energy use, and what the evidence says so far.
How do I choose a smaller AI model?
How to pick the smallest AI model that does your task well: set a quality bar, test small to large, check energy ratings and keep a big model as fallback.
How do I stop AI agents wasting tokens?
Practical guardrails for AI agents: stop conditions, step and token limits, loop detection, context trimming and small models for simple steps.