# How do I run an AI model locally?

> How to run an open AI model on your own laptop or server with tools like Ollama or llama.cpp, choose a quantised model that fits, and measure its energy.

Source: https://greenai.guide/how-do-i/run-an-ai-model-locally/  
Updated: 2026-11-19

## In short

Install a local runner such as Ollama or llama.cpp, download a small open-weight model in a quantised format such as GGUF, and run it on your own computer. Pick the smallest model that does your task: quantised versions need far less memory, so an 8-billion-parameter model that needs about 16 GB in 16-bit form fits in about 4–5 GB at 4 bits.

## Steps

1. Check your computer's memory and whether it has a usable GPU or Apple silicon.
2. Install a local runner such as Ollama or llama.cpp.
3. Download a small open-weight model in a quantised format that fits your memory.
4. Test it on your real tasks and move up a size only if it falls short.
5. Measure its energy with a plug meter or a tool such as CodeCarbon.
6. Close the model when you're not using it so it doesn't hold memory or power.

**Running a model locally means choosing a runner, a model that fits, and then measuring.** It keeps data on your
machine and lets you see exactly what the AI costs in energy, which a hosted service never shows you.

## What do I need?

- **A runner.** Ollama is a popular way to use open models on your own computer. llama.cpp is a lower-level
  C/C++ engine aimed at running models with minimal set-up on a wide range of hardware.
- **A model file.** GGUF is a binary format optimised for quick loading and efficient inference, with quantised
  versions from under 2 bits up to 8 bits per weight.
- **Enough memory.** Hugging Face's documentation gives a rule of thumb: an 8-billion-parameter model needs about
  16 GB for its weights in 16-bit form, about 8 GB at 8 bits and about 4–5 GB at 4 bits. Fewer bits cost some
  accuracy.

## Is it greener than the cloud?

**Not automatically.** A laptop running one request at a time can be less efficient per answer than a busy,
well-batched data-centre server. Local wins when the model is small, the task is frequent, or it saves sending
large data over the network. See [device or cloud](/how-do-i/choose-between-device-and-cloud-ai/).

## Common mistakes

- **Downloading the biggest model your machine can hold.** Start small; see
  [choose a smaller AI model](/how-do-i/choose-a-smaller-ai-model/).
- **Leaving the model loaded all day** on a desktop with a power-hungry GPU.
- **Buying a new GPU before trying a quantised model**; see [how to quantise a model](/how-do-i/quantise-a-model/).

To check the energy, use a plug-in meter or CodeCarbon, as described in
[measure the energy AI uses on your own computer](/how-do-i/measure-energy-of-ai-on-my-own-computer/). For the
trade-offs, read [edge or cloud](/practice/edge-or-cloud/).

## Sources

- [Ollama documentation](https://docs.ollama.com), Ollama (2026)
- [llama.cpp: LLM inference in C/C++](https://github.com/ggml-org/llama.cpp), ggml-org (GitHub) (2026)
- [GGUF](https://huggingface.co/docs/hub/gguf), Hugging Face Hub documentation (2026)
- [Quantization overview](https://huggingface.co/docs/transformers/main/en/quantization/overview), Hugging Face Transformers documentation (2026)
- [CodeCarbon](https://codecarbon.io/), CodeCarbon (2026)
