In short
Install a local runner such as Ollama or llama.cpp, download a small open-weight model in a quantised format such as GGUF, and run it on your own computer. Pick the smallest model that does your task: quantised versions need far less memory, so an 8-billion-parameter model that needs about 16 GB in 16-bit form fits in about 4–5 GB at 4 bits.
Steps
- Check your computer's memory and whether it has a usable GPU or Apple silicon.
- Install a local runner such as Ollama or llama.cpp.
- Download a small open-weight model in a quantised format that fits your memory.
- Test it on your real tasks and move up a size only if it falls short.
- Measure its energy with a plug meter or a tool such as CodeCarbon.
- Close the model when you're not using it so it doesn't hold memory or power.
Running a model locally means choosing a runner, a model that fits, and then measuring. It keeps data on your machine and lets you see exactly what the AI costs in energy, which a hosted service never shows you.
What do I need?
- A runner. Ollama is a popular way to use open models on your own computer. llama.cpp is a lower-level C/C++ engine aimed at running models with minimal set-up on a wide range of hardware.
- A model file. GGUF is a binary format optimised for quick loading and efficient inference, with quantised versions from under 2 bits up to 8 bits per weight.
- Enough memory. Hugging Face's documentation gives a rule of thumb: an 8-billion-parameter model needs about 16 GB for its weights in 16-bit form, about 8 GB at 8 bits and about 4–5 GB at 4 bits. Fewer bits cost some accuracy.
Is it greener than the cloud?
Not automatically. A laptop running one request at a time can be less efficient per answer than a busy, well-batched data-centre server. Local wins when the model is small, the task is frequent, or it saves sending large data over the network. See device or cloud.
Common mistakes
- Downloading the biggest model your machine can hold. Start small; see choose a smaller AI model.
- Leaving the model loaded all day on a desktop with a power-hungry GPU.
- Buying a new GPU before trying a quantised model; see how to quantise a model.
To check the energy, use a plug-in meter or CodeCarbon, as described in measure the energy AI uses on your own computer. For the trade-offs, read edge or cloud.
Sources
- Ollama documentation, Ollama (2026)
- llama.cpp: LLM inference in C/C++, ggml-org (GitHub) (2026)
- GGUF, Hugging Face Hub documentation (2026)
- Quantization overview, Hugging Face Transformers documentation (2026)
- CodeCarbon, CodeCarbon (2026)
Related
How do I quantise a model?
How to quantise an AI model to 8 or 4 bits: which tools to use, how to test quality, and when quantisation saves energy and when it doesn't.
How do I decide between running AI on my device or in the cloud?
How to weigh on-device (edge) AI against cloud AI for energy, privacy and quality, and which kinds of task suit each, with honest caveats on the evidence.
How do I measure the energy AI uses on my own computer or GPU?
Measure local AI energy with hardware counters, GPU telemetry or a plug-in meter, sample over the whole job, and subtract the idle baseline.
Edge or cloud: where should AI run?
How to weigh on-device and edge AI against central data centres: energy, network, hardware, privacy and resilience, with a step-by-step way to decide.