greenai.guide

Home › How do I…? › How do I choose between retrieval and fine-tuning to save energy?

How do I…?

How do I choose between retrieval and fine-tuning to save energy?

Updated · 1 min read

In short

Use retrieval when the model needs facts that change or are specific to you: put them in a searchable index and fetch only the relevant passages. Use fine-tuning when the model needs a skill, format or style it lacks. Retraining to keep facts up to date repeats training costs, while pasting whole documents into every prompt wastes inference, so each approach has its place.

Steps

  1. List what the model is missing: facts, a skill, a format or a tone.
  2. Put changing or private facts in a searchable index and retrieve only relevant passages.
  3. Fine-tune a small adapter only for skills, formats or styles that prompting can't achieve.
  4. Update the index when content changes, processing only what changed.
  5. Measure tokens per request and quality for each option before deciding.

Retrieve facts; fine-tune skills. Choosing the wrong one means paying again and again, either for retraining or for oversized prompts.

What is the difference?

  • Retrieval-augmented generation (RAG) looks up relevant passages from your own content and gives them to the model with the question. Lewis and colleagues introduced the approach in 2020 for knowledge-intensive tasks. The model's weights do not change, so updating knowledge means updating an index.
  • Fine-tuning changes the model's weights, or a small adapter such as LoRA, so it learns a behaviour: a format, a classification scheme, a house style.

Which uses less energy?

It depends on how often things change and how much text each request carries.

Situation Usually better
Facts change weekly or monthly Retrieval
Private documents, policies, product data Retrieval
A fixed output format or labelling scheme Fine-tuning a small model
Very high volume, narrow task Fine-tuning, so prompts can be short

GreenCode, an ITEA research project, describes using retrieval so its own AI reads only the data it needs, and processing only what changed between runs.

Common mistakes

  • Retraining to add facts. Every update repeats the training cost.
  • "Stuff everything in the prompt" RAG. Retrieving twenty long passages when two would do multiplies input tokens. See how to make vector search and RAG more efficient.
  • Fine-tuning a big model when a small fine-tuned one would do.

For the wider principle, read orchestration over generation.

Sources

  1. Retrieval-augmented generation for knowledge-intensive NLP tasks, Lewis et al., NeurIPS (2020)
  2. Green AI and energy-aware computing, GreenCode (2026)
  3. LoRA: low-rank adaptation of large language models, Hu et al. (2021)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy