greenai.guide

Home › How do I…? › How do I route requests to smaller models?

How do I…?

How do I route requests to smaller models?

Updated · 1 min read

In short

Put a router in front of your models: a rule, a small classifier or a cascade that tries a small model first and escalates when confidence is low or a check fails. Published routers have cut costs by more than half in some tests without lowering quality. Tune the threshold on your own data and measure cost per successful task, not per call.

Steps

  1. Build an evaluation set of real requests with a pass mark for each.
  2. Measure how often a small model passes on its own.
  3. Choose a router design, such as rules, a difficulty classifier or a cascade with a check.
  4. Tune the escalation threshold on your evaluation set.
  5. Track cost and energy per successful task, including escalations and retries.
  6. Re-tune when either model changes.

Routing gives you most of the quality of a large model at much of the cost of a small one. The idea is simple: send each request to the cheapest model likely to handle it well.

Which design should I use?

  • Rules. Route by request type, length or channel. Easy to understand, and often enough.
  • Difficulty classifier. A small model predicts how hard a request is. Ding and colleagues' Hybrid LLM work routes queries between a small and a large model this way, balancing cost and quality.
  • Cascade. Try the small model; if a check fails or confidence is low, pass to a larger one. FrugalGPT, by Chen, Zaharia and Zou, studied cascades like this for cutting cost.
  • Learned router. Trained on which model did well on which request. The RouteLLM study reported cost reductions of more than half in some cases without lowering response quality.

Common mistakes

  • Measuring per call. A failed small attempt plus a retry on the big model can cost more than going straight to the big one. A 2026 UK green AI sandpit of mathematicians, computer scientists and industry flagged this as a real risk in agentic systems. Measure per successful task.
  • Pushing the choice onto users. Put routing in the system, not in a drop-down most people ignore.
  • Setting and forgetting. Log escalation rates and revisit when models change.

What makes the biggest difference?

A good evaluation set. Without real examples and a pass mark, you cannot tune the threshold or prove the saving. Read choose the right model for how to build one.

Sources

  1. RouteLLM: learning to route LLMs with preference data, Ong et al., ICLR 2025 (2024)
  2. FrugalGPT: how to use large language models while reducing cost and improving performance, Chen, Zaharia and Zou, Transactions on Machine Learning Research (2023)
  3. Hybrid LLM: cost-efficient and quality-aware query routing, Ding et al., ICLR (2024)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy