greenai.guide

Home › How do I…? › How do I run an AI model locally?

How do I…?

How do I run an AI model locally?

Updated · 1 min read

In short

Install a local runner such as Ollama or llama.cpp, download a small open-weight model in a quantised format such as GGUF, and run it on your own computer. Pick the smallest model that does your task: quantised versions need far less memory, so an 8-billion-parameter model that needs about 16 GB in 16-bit form fits in about 4–5 GB at 4 bits.

Steps

  1. Check your computer's memory and whether it has a usable GPU or Apple silicon.
  2. Install a local runner such as Ollama or llama.cpp.
  3. Download a small open-weight model in a quantised format that fits your memory.
  4. Test it on your real tasks and move up a size only if it falls short.
  5. Measure its energy with a plug meter or a tool such as CodeCarbon.
  6. Close the model when you're not using it so it doesn't hold memory or power.

Running a model locally means choosing a runner, a model that fits, and then measuring. It keeps data on your machine and lets you see exactly what the AI costs in energy, which a hosted service never shows you.

What do I need?

  • A runner. Ollama is a popular way to use open models on your own computer. llama.cpp is a lower-level C/C++ engine aimed at running models with minimal set-up on a wide range of hardware.
  • A model file. GGUF is a binary format optimised for quick loading and efficient inference, with quantised versions from under 2 bits up to 8 bits per weight.
  • Enough memory. Hugging Face's documentation gives a rule of thumb: an 8-billion-parameter model needs about 16 GB for its weights in 16-bit form, about 8 GB at 8 bits and about 4–5 GB at 4 bits. Fewer bits cost some accuracy.

Is it greener than the cloud?

Not automatically. A laptop running one request at a time can be less efficient per answer than a busy, well-batched data-centre server. Local wins when the model is small, the task is frequent, or it saves sending large data over the network. See device or cloud.

Common mistakes

  • Downloading the biggest model your machine can hold. Start small; see choose a smaller AI model.
  • Leaving the model loaded all day on a desktop with a power-hungry GPU.
  • Buying a new GPU before trying a quantised model; see how to quantise a model.

To check the energy, use a plug-in meter or CodeCarbon, as described in measure the energy AI uses on your own computer. For the trade-offs, read edge or cloud.

Sources

  1. Ollama documentation, Ollama (2026)
  2. llama.cpp: LLM inference in C/C++, ggml-org (GitHub) (2026)
  3. GGUF, Hugging Face Hub documentation (2026)
  4. Quantization overview, Hugging Face Transformers documentation (2026)
  5. CodeCarbon, CodeCarbon (2026)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy