In short
Use an approximate nearest-neighbour index instead of comparing every query with every stored vector, retrieve a few short, relevant passages rather than many long ones, and re-embed only content that has changed. Then use the smallest generation model that answers well from good context. Most of the waste in retrieval systems comes from brute-force search and oversized prompts.
Steps
- Replace brute-force vector comparison with an approximate nearest-neighbour index.
- Split documents into short chunks and retrieve only the top few.
- Re-embed only new or changed content rather than rebuilding the whole index.
- Log input tokens per answer and trim context that doesn't improve quality.
- Test a smaller generation model once retrieval quality is good.
Retrieval should make the model read less, not more. A well-built retrieval-augmented generation (RAG) system finds a few relevant passages quickly and hands only those to the model.
Where does RAG waste energy?
- Brute-force search. Comparing a query with every stored embedding, one by one, grows with the size of the collection. The Green Software Foundation recommends an approximate nearest-neighbour index instead.
- Oversized context. Retrieving twenty long passages "to be safe" multiplies input tokens. Epoch AI estimates a typical query of about 0.3 Wh rises to about 2.5 Wh with a 10,000-token input.
- Rebuilding everything. Re-embedding a whole document store each night when only a few documents changed.
What makes the biggest difference?
Better retrieval, so the model needs less context. Good chunking, a reranking step and metadata filters (date, department, product) mean three short passages can do the work of fifteen long ones. Once the context is good, a smaller generation model often answers just as well; see how to choose a smaller AI model.
Common mistakes
- Using a large embedding model where a small one retrieves just as well on your data. Test both.
- No evaluation set. Measure retrieval quality (did the right passage come back?) separately from answer quality.
- Caching nothing. Popular questions can reuse answers; see how to use caching.
For the bigger idea of reusing rather than regenerating, read orchestration over generation.
Sources
- Green Software Patterns: optimise data storage formats for AI training and inference, Green Software Foundation (2025)
- Retrieval-augmented generation for knowledge-intensive NLP tasks, Lewis et al., NeurIPS (2020)
- How much energy does ChatGPT use?, Epoch AI (2025)
Related
How do I choose between retrieval and fine-tuning to save energy?
When to use retrieval (RAG) and when to fine-tune: retrieval for facts that change, fine-tuning for skills and style, and why mixing them up wastes energy.
How do I stop my chatbot resending the whole conversation?
How to keep chatbot context small: cap past turns, summarise older messages, retrieve what's relevant and cache fixed prompts, so input stops growing.
Orchestration over generation
Why reusing vetted components and routing requests to the smallest capable model can cut AI's energy use, and what the evidence says so far.
How do I store AI training data so pipelines read less?
How columnar and binary formats, partitioning and one-off preprocessing cut the energy AI pipelines spend reading and parsing training data.