greenai.guide

Home › How do I…? › How do I fine-tune a model without huge compute?

How do I…?

How do I fine-tune a model without huge compute?

Updated · 1 min read

In short

Start from a pre-trained model and train a small adapter instead of all its weights, using LoRA, or QLoRA with 4-bit weights. LoRA cut trainable parameters by 10,000 times and GPU memory by three times compared with full fine-tuning of GPT-3, and QLoRA fine-tuned a 65-billion-parameter model on a single 48 GB GPU. Compare the result with your quality bar before considering anything bigger.

Steps

  1. Check whether prompting or retrieval already meets your need before fine-tuning at all.
  2. Pick the smallest pre-trained model that comes close on your task.
  3. Fine-tune a LoRA adapter, or QLoRA if memory is tight.
  4. Use a small, clean, task-specific dataset and stop early when validation stops improving.
  5. Track energy with an emissions tracker and record it with the results.

Adapt a small part of a pre-trained model rather than retraining all of it. Parameter-efficient fine-tuning gets most of the benefit of full fine-tuning for a small fraction of the compute.

Why does it matter?

Full fine-tuning updates every weight, which needs large GPU clusters and long runs. Hu and colleagues showed that LoRA (low-rank adaptation), which trains small extra matrices alongside frozen weights, could reduce trainable parameters by 10,000 times and GPU memory by three times compared with fine-tuning GPT-3 175B. Dettmers and colleagues' QLoRA combines this with 4-bit weights and fine-tuned a 65-billion-parameter model on a single 48 GB GPU.

How do I do it?

  1. Question the need. Fine-tuning teaches skills and style; it is a poor way to add facts that change. See retrieval or fine-tuning.
  2. Start small. A small model fine-tuned on your task can match a much larger general one on that task.
  3. Use an adapter library. Popular open-source libraries support LoRA and QLoRA with a few lines of configuration.
  4. Keep data tight. Fewer, cleaner examples train faster. Remove duplicates.
  5. Measure. Wrap the run in a tracker such as CodeCarbon and record energy next to accuracy.

Common mistakes

  • Fine-tuning the biggest model available before testing a smaller one.
  • Long runs with no early stopping, which keep spending after quality has levelled off.
  • Retraining for every data update instead of updating a retrieval index.

For more training techniques, read efficient training and inference.

Sources

  1. LoRA: low-rank adaptation of large language models, Hu et al. (2021)
  2. QLoRA: efficient finetuning of quantized LLMs, Dettmers, Pagnoni, Holtzman and Zettlemoyer (2023)
  3. Green Software Patterns: leverage pre-trained models and transfer learning, Green Software Foundation (2025)
  4. CodeCarbon, CodeCarbon (2026)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy