greenai.guide

Home › How do I…? › How do I make vector search and RAG more efficient?

How do I…?

How do I make vector search and RAG more efficient?

Updated · 1 min read

In short

Use an approximate nearest-neighbour index instead of comparing every query with every stored vector, retrieve a few short, relevant passages rather than many long ones, and re-embed only content that has changed. Then use the smallest generation model that answers well from good context. Most of the waste in retrieval systems comes from brute-force search and oversized prompts.

Steps

  1. Replace brute-force vector comparison with an approximate nearest-neighbour index.
  2. Split documents into short chunks and retrieve only the top few.
  3. Re-embed only new or changed content rather than rebuilding the whole index.
  4. Log input tokens per answer and trim context that doesn't improve quality.
  5. Test a smaller generation model once retrieval quality is good.

Retrieval should make the model read less, not more. A well-built retrieval-augmented generation (RAG) system finds a few relevant passages quickly and hands only those to the model.

Where does RAG waste energy?

  • Brute-force search. Comparing a query with every stored embedding, one by one, grows with the size of the collection. The Green Software Foundation recommends an approximate nearest-neighbour index instead.
  • Oversized context. Retrieving twenty long passages "to be safe" multiplies input tokens. Epoch AI estimates a typical query of about 0.3 Wh rises to about 2.5 Wh with a 10,000-token input.
  • Rebuilding everything. Re-embedding a whole document store each night when only a few documents changed.

What makes the biggest difference?

Better retrieval, so the model needs less context. Good chunking, a reranking step and metadata filters (date, department, product) mean three short passages can do the work of fifteen long ones. Once the context is good, a smaller generation model often answers just as well; see how to choose a smaller AI model.

Common mistakes

  • Using a large embedding model where a small one retrieves just as well on your data. Test both.
  • No evaluation set. Measure retrieval quality (did the right passage come back?) separately from answer quality.
  • Caching nothing. Popular questions can reuse answers; see how to use caching.

For the bigger idea of reusing rather than regenerating, read orchestration over generation.

Sources

  1. Green Software Patterns: optimise data storage formats for AI training and inference, Green Software Foundation (2025)
  2. Retrieval-augmented generation for knowledge-intensive NLP tasks, Lewis et al., NeurIPS (2020)
  3. How much energy does ChatGPT use?, Epoch AI (2025)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy