# Orchestration over generation

> Why reusing vetted components and routing requests to the smallest capable model can cut AI's energy use, and what the evidence says so far.

Source: https://greenai.guide/articles/orchestration-over-generation/  
Updated: 2026-11-19

## In short

Generating every answer, line of code or image from scratch with a large model is the most expensive way to use AI. Orchestration uses AI to plan and select, then assembles the result from tested components and routes each step to the cheapest model that can do it. The cheapest token is the one you never generate.

## Key points

- Generation is the costly step in generative AI; selecting and reusing existing, tested components is far cheaper.
- Routing sends each request to the smallest model that can handle it, and published routers cut cost substantially with little loss of quality.
- Retrieval supplies only the relevant fragment of a document or codebase, so the model reads fewer tokens.
- Reusing vetted components also helps security and maintenance: one fix reaches every place the component is used.
- Routing has a catch: a model that is too small can fail and force a costly retry on a big one.

Most discussion of AI's footprint asks how to make generation cheaper: smaller models, faster chips, cleaner power.
A different question is how much generation is needed at all. Orchestration answers it by treating generation as the
last resort rather than the default.

## What does "orchestration over generation" mean?

**It means using AI to plan a task and pick the parts, then building the result mostly from things that already
exist and have been checked.** A large model is asked to generate something new only when nothing suitable exists.

A great deal of what people ask generative AI for is not new. Software is full of the same patterns: log-in screens,
file uploads, date handling, data validation. Contracts reuse standard clauses. Reports reuse standard sections.
Generating each of these afresh, every time, repeats the energy cost and produces slightly different versions that
each need checking.

An orchestrated system works more like a well-run engineering team:

1. A planner works out what the task needs.
2. A cheap classifier or search finds an existing, tested component that fits.
3. Only the genuinely new parts are generated, by the smallest model that can do them.
4. Newly generated parts are tested and added to the library, so the next request can reuse them.

The library behaves like a cache. A 2026 UK green AI sandpit of mathematicians, computer scientists and industry
discussed this pattern under the heading "the cheapest token is the one you never generate".

## Why would reuse save energy?

**Because retrieving and adapting a component costs a tiny fraction of generating it, and tested components need
less rework.** Generation is where the tokens, and so most of the inference energy, are spent.

The energy cost of AI-written output is not trivial. A King's College London study (Cheung and colleagues, 2025), a partner
paper in the GreenCode project, compared solving programming tasks by hand with solving them with an AI assistant. Averaged over
the tasks, which were programming-contest problems, AI-assisted code generation had 32.72 times the carbon footprint of the manual approach, and the gap
widened as tasks became more complex. Reuse attacks exactly that cost.

There are benefits beyond energy:

- **Quality.** A vetted component has been tested once and can be trusted many times.
- **Security.** If a component turns out to be flawed, you know every place it is used, and one fix reaches all of
  them.
- **Maintenance.** A codebase built from shared parts is smaller and easier to understand than one full of
  near-duplicates.

## What is model routing?

**Routing sends each request to the model best suited to it, usually the smallest that is likely to answer well.**
Simple requests go to small, cheap models; hard ones are escalated.

Published research shows the potential:

- **FrugalGPT** (Chen, Zaharia and Zou, 2023) chained models from cheap to expensive, stopping when an answer was
  judged good enough, and reported matching GPT-4's performance with up to 98% lower cost.
- **RouteLLM** (Ong and colleagues, 2024) trained routers on human preference data to decide when a request needs a
  strong model, and reported cutting costs by more than half in some cases without losing quality.
- **Hybrid LLM** (Ding and colleagues, 2024) routed queries between a small and a large model according to predicted
  difficulty, making up to 40% fewer calls to the large model with no drop in response quality.

These studies measure cost rather than energy directly, but for the same provider and hardware, fewer tokens on
large models generally means less energy. See [choosing the right model](/practice/choose-the-right-model/).

## What is retrieval, and why does it help?

**Retrieval looks up the relevant information and gives the model only that, instead of making it read or recall
everything.** Retrieval-augmented generation (RAG), introduced by Lewis and colleagues in 2020, combines a search
step with a generator.

For energy, the point is token count. Pushing whole documents or codebases through a model's context window costs
energy on every request. GreenCode, a European (ITEA) research project on energy-efficient software, uses retrieval from an index
of the codebase so that "the token count reflects the task rather than the repository", alongside small fine-tuned
models and routing to the smallest capable component.

## What does the GreenCode work show?

**That the cost of using AI has to be counted, and that how an agent works matters as much as which model it uses.**
Two partner papers illustrate this:

- **EnergyBench** (Ionescu, Jensen and Thomsen, 2025) scores AI-generated code for both correctness and the energy
  it uses when run. It found that prompt wording could improve the energy efficiency of the generated code by up to
  91.9% in some cases, but often at the cost of accuracy. Prompting for green code is not a reliable fix on its own.
- **The GreenCode Constitution** (Kapenekakis and colleagues, 2026) gave a coding agent written principles,
  correctness first, then 37 energy anti-patterns, and had it measure energy before and after each change. On
  llama.cpp, an already highly tuned inference engine, it cut GPU energy by 4.7%. The energy saving exceeded the
  speed gain, a reminder that faster is not always greener, so energy has to be measured directly.

GreenCode's own pipeline applies the orchestration principle: it prefers small task-specific models, retrieval, and
routing, and runs a pre-assessment so that a job whose inference cost would not be repaid is not run.

## What can go wrong?

**Routing and reuse save energy only when they pick well.** The main risks:

- **Under-sizing.** A model that is too small may fail, and the retry on a larger model costs more than going there
  first. In agentic systems, where one failure can derail many later steps, this matters more. Setting the threshold
  is a real design problem.
- **Stale libraries.** Components need owners, tests and updates, or reuse spreads old mistakes.
- **Hidden costs.** The classifier, the index and the retrieval step all use energy. They are usually far cheaper
  than generation, but should be measured too.
- **Rebound.** Cheaper AI invites more AI. See [efficiency is not sufficiency](/articles/efficiency-is-not-sufficiency/).

## What to do next

- Before generating, search: does a tested component, template or earlier answer already exist?
- Route requests by difficulty; use a small model by default and escalate. See [use AI efficiently](/practice/use-ai-efficiently/).
- Use retrieval to keep prompts short, rather than pasting whole documents.
- Keep a library of vetted outputs, with tests and an owner, and add to it as you go.
- Measure energy or tokens per completed task, including retries, not per request.
- Read [agentic AI and token waste](/articles/agentic-ai-and-token-waste/) if you run multi-step agents.

## Frequently asked questions

### What is orchestration in AI?

Coordinating several models, tools and stored components to complete a task, rather than asking one large model to generate the whole answer. A planner decides what is needed; cheaper parts do most of the work.

### What is model routing?

Sending each request to the model best suited to it, usually the smallest and cheapest one likely to answer well, and escalating to a larger model only when needed.

### Does routing reduce quality?

Published routing systems report large cost savings while keeping most of the quality of the strongest model, but the result depends on the task and on how well the router judges difficulty.

## Sources

- [FrugalGPT: how to use large language models while reducing cost and improving performance](https://arxiv.org/abs/2305.05176), Chen, Zaharia and Zou, Transactions on Machine Learning Research (2023)
- [RouteLLM: learning to route LLMs with preference data](https://arxiv.org/abs/2406.18665), Ong et al., ICLR 2025 (2024)
- [Hybrid LLM: cost-efficient and quality-aware query routing](https://arxiv.org/abs/2404.14618), Ding et al., ICLR (2024)
- [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://arxiv.org/abs/2005.11401), Lewis et al., NeurIPS (2020)
- [Green optimised generative AI](https://greencode.ai/feature/green-optimised-generative-ai/), GreenCode (2026)
- [Green AI and energy-aware computing](https://greencode.ai/benefit/green-ai-and-energy-aware-computing/), GreenCode (2026)
- [Comparative analysis of carbon footprint in manual vs. LLM-assisted code development](https://doi.org/10.1145/3711919.3728678), Cheung, Kaul, Jahangirova, Mousavi and Zie, ResponsibleSE 2025 (ACM) (2025)
- [EnergyBench: holistic benchmark for correct and energy-efficient LLM-generated code](https://doi.org/10.1109/flta67013.2025.11336593), Ionescu, Jensen and Thomsen, FLTA 2025 (2025)
- [GreenCode Constitution: agent principles for optimizing code and LLM inference energy use](https://greencode.ai/publications/greencode-constitution-agent-principles-for-optimizing-code-and-llm-inference-en/), Kapenekakis, Kim, Ionescu, Thomsen and Albano, GreenAI workshop, ECML PKDD 2026 (2026)
