greenai.guide

Home › How do I…? › How do I avoid training a model from scratch?

How do I…?

How do I avoid training a model from scratch?

Updated · 1 min read

In short

Search for a pre-trained model close to your task, then adapt it: fine-tune it, train a small adapter such as LoRA, or add knowledge through retrieval instead of retraining. Where you already have several models fine-tuned from the same base, averaging their weights can improve accuracy at no extra inference cost. Training from scratch is rarely necessary outside foundation-model research.

Steps

  1. Search public model hubs for pre-trained models in your domain and language.
  2. Test the closest candidates on your task before deciding anything is missing.
  3. Fine-tune or train a LoRA adapter on your own data.
  4. Use retrieval for knowledge that changes instead of retraining.
  5. Reuse and merge models fine-tuned from the same base where you have them.
  6. Publish your fine-tuned model and its training emissions so others can reuse it.

Reuse first. Somebody has almost always trained a model close to what you need, and adapting it costs a small fraction of training your own.

Why does it matter?

Pre-training is the most energy-intensive stage of a model's life. The Green Software Foundation's patterns recommend leveraging pre-trained models and transfer learning rather than training from scratch. A 2026 UK green AI sandpit of mathematicians, computer scientists and industry also noted that a lot of useful work sits in little-known repositories, so a careful search is the first step of any project.

What are the options?

  • Fine-tuning a pre-trained model on your data. With LoRA, Hu and colleagues reduced trainable parameters by 10,000 times compared with fully fine-tuning GPT-3 175B. See how to fine-tune without huge compute.
  • Retrieval for knowledge that changes: see retrieval or fine-tuning.
  • Model merging. Wortsman and colleagues showed that averaging the weights of several models fine-tuned from the same base, a "model soup", can improve accuracy without increasing inference time.
  • Distillation into a small specialist: see how to distil a model.

Common mistakes

  • Assuming your domain is too specialist. Test general and domain models first; they often get close.
  • Ignoring licences, which can rule out some reuse.
  • Not sharing back. Publishing your fine-tuned model, with training emissions in the model card metadata as Hugging Face supports, saves others repeating your work.

For the research angle, read maths for efficient AI.

Sources

  1. Green Software Patterns: leverage pre-trained models and transfer learning, Green Software Foundation (2025)
  2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, Wortsman et al., ICML (2022)
  3. LoRA: low-rank adaptation of large language models, Hu et al. (2021)
  4. Displaying carbon emissions for your model, Hugging Face Hub documentation (2026)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy