In short
Search for a pre-trained model close to your task, then adapt it: fine-tune it, train a small adapter such as LoRA, or add knowledge through retrieval instead of retraining. Where you already have several models fine-tuned from the same base, averaging their weights can improve accuracy at no extra inference cost. Training from scratch is rarely necessary outside foundation-model research.
Steps
- Search public model hubs for pre-trained models in your domain and language.
- Test the closest candidates on your task before deciding anything is missing.
- Fine-tune or train a LoRA adapter on your own data.
- Use retrieval for knowledge that changes instead of retraining.
- Reuse and merge models fine-tuned from the same base where you have them.
- Publish your fine-tuned model and its training emissions so others can reuse it.
Reuse first. Somebody has almost always trained a model close to what you need, and adapting it costs a small fraction of training your own.
Why does it matter?
Pre-training is the most energy-intensive stage of a model's life. The Green Software Foundation's patterns recommend leveraging pre-trained models and transfer learning rather than training from scratch. A 2026 UK green AI sandpit of mathematicians, computer scientists and industry also noted that a lot of useful work sits in little-known repositories, so a careful search is the first step of any project.
What are the options?
- Fine-tuning a pre-trained model on your data. With LoRA, Hu and colleagues reduced trainable parameters by 10,000 times compared with fully fine-tuning GPT-3 175B. See how to fine-tune without huge compute.
- Retrieval for knowledge that changes: see retrieval or fine-tuning.
- Model merging. Wortsman and colleagues showed that averaging the weights of several models fine-tuned from the same base, a "model soup", can improve accuracy without increasing inference time.
- Distillation into a small specialist: see how to distil a model.
Common mistakes
- Assuming your domain is too specialist. Test general and domain models first; they often get close.
- Ignoring licences, which can rule out some reuse.
- Not sharing back. Publishing your fine-tuned model, with training emissions in the model card metadata as Hugging Face supports, saves others repeating your work.
For the research angle, read maths for efficient AI.
Sources
- Green Software Patterns: leverage pre-trained models and transfer learning, Green Software Foundation (2025)
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, Wortsman et al., ICML (2022)
- LoRA: low-rank adaptation of large language models, Hu et al. (2021)
- Displaying carbon emissions for your model, Hugging Face Hub documentation (2026)
Related
How do I fine-tune a model without huge compute?
How to adapt a pre-trained model with parameter-efficient fine-tuning such as LoRA and QLoRA, so it fits on one GPU and uses far less energy.
How do I choose between retrieval and fine-tuning to save energy?
When to use retrieval (RAG) and when to fine-tune: retrieval for facts that change, fine-tuning for skills and style, and why mixing them up wastes energy.
How do I reduce the energy of training a model?
The main levers for cutting training energy: start pre-trained, right-size, use mixed precision, stop early, pick clean places and times, measure.
The maths that makes AI more efficient
Pruning, sparsity, sketching, random matrix theory and physics-informed models: how mathematics cuts the computation, memory and energy AI needs.