# How do I make vector search and RAG more efficient?

> Ways to cut the compute of retrieval-augmented generation: approximate indexes, fewer and shorter passages, incremental indexing and smaller models.

Source: https://greenai.guide/how-do-i/make-vector-search-and-rag-more-efficient/  
Updated: 2026-11-19

## In short

Use an approximate nearest-neighbour index instead of comparing every query with every stored vector, retrieve a few short, relevant passages rather than many long ones, and re-embed only content that has changed. Then use the smallest generation model that answers well from good context. Most of the waste in retrieval systems comes from brute-force search and oversized prompts.

## Steps

1. Replace brute-force vector comparison with an approximate nearest-neighbour index.
2. Split documents into short chunks and retrieve only the top few.
3. Re-embed only new or changed content rather than rebuilding the whole index.
4. Log input tokens per answer and trim context that doesn't improve quality.
5. Test a smaller generation model once retrieval quality is good.

**Retrieval should make the model read less, not more.** A well-built retrieval-augmented generation (RAG) system
finds a few relevant passages quickly and hands only those to the model.

## Where does RAG waste energy?

- **Brute-force search.** Comparing a query with every stored embedding, one by one, grows with the size of the
  collection. The Green Software Foundation recommends an approximate nearest-neighbour index instead.
- **Oversized context.** Retrieving twenty long passages "to be safe" multiplies input tokens. Epoch AI estimates a
  typical query of about 0.3 Wh rises to about 2.5 Wh with a 10,000-token input.
- **Rebuilding everything.** Re-embedding a whole document store each night when only a few documents changed.

## What makes the biggest difference?

**Better retrieval, so the model needs less context.** Good chunking, a reranking step and metadata filters (date,
department, product) mean three short passages can do the work of fifteen long ones. Once the context is good, a
smaller generation model often answers just as well; see [how to choose a smaller AI model](/how-do-i/choose-a-smaller-ai-model/).

## Common mistakes

- **Using a large embedding model** where a small one retrieves just as well on your data. Test both.
- **No evaluation set.** Measure retrieval quality (did the right passage come back?) separately from answer quality.
- **Caching nothing.** Popular questions can reuse answers; see [how to use caching](/how-do-i/cache-llm-calls/).

For the bigger idea of reusing rather than regenerating, read
[orchestration over generation](/articles/orchestration-over-generation/).

## Sources

- [Green Software Patterns: optimise data storage formats for AI training and inference](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/development/data-handling/optimize-data-storage-ai-training.md), Green Software Foundation (2025)
- [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://arxiv.org/abs/2005.11401), Lewis et al., NeurIPS (2020)
- [How much energy does ChatGPT use?](https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use), Epoch AI (2025)
