# How do I quantise a model?

> How to quantise an AI model to 8 or 4 bits: which tools to use, how to test quality, and when quantisation saves energy and when it doesn't.

Source: https://greenai.guide/how-do-i/quantise-a-model/  
Updated: 2026-11-19

## In short

Quantising stores a model's weights with fewer bits, typically 8 or 4 instead of 16 or 32, so it needs less memory and often less energy per answer. Load a ready-quantised version or quantise with your library's built-in methods, then test it on your own examples against the full-precision model. Keep it if quality holds and energy per request falls.

## Steps

1. Check whether a ready-quantised version of your model already exists.
2. Otherwise quantise with a supported method in your library, starting at 8 bits.
3. Run the quantised and original models on the same evaluation set.
4. Try 4 bits if 8-bit quality holds and you need to save more memory.
5. Measure energy and latency per request at your real batch size before switching.

**Quantisation is often the quickest efficiency win for a model you run yourself.** It shrinks memory, so the model
fits on fewer or smaller GPUs, and it can cut energy per answer.

## Why does it matter?

Hugging Face's documentation gives a sense of scale: an 8-billion-parameter model needs about 16 GB for its weights
at 16-bit precision, about 8 GB at 8 bits and about 4 to 5 GB at 4 bits. Dettmers and colleagues showed with
LLM.int8() that 8-bit inference can cut the memory needed by half while retaining full-precision performance, for
models up to 175 billion parameters. UNESCO and University College London report that compression such as
quantisation can save up to 44% of energy.

## How do I do it?

- **Download a quantised version.** Many open models are published in quantised formats. GGUF, used by llama.cpp,
  offers quantisation types from about 1.5 to 8 bits per weight.
- **Quantise on load.** Libraries such as Hugging Face Transformers support more than 20 quantisation methods,
  several of which quantise weights as the model loads.
- **Calibrate where needed.** Some methods use a small sample of your data to choose how to round, which tends to
  preserve quality better at 4 bits.

## Common mistakes

- **Skipping the quality test.** Fewer bits cost some accuracy, and each method trades memory, speed and accuracy
  differently. Test on your own cases.
- **Assuming it is always faster.** Hugging Face notes that if you aren't limited by GPU memory, quantisation can
  increase latency slightly because of the extra quantise and dequantise steps. Measure energy, not just speed.
- **Stopping at quantisation.** Combine it with a smaller model or [distillation](/how-do-i/distil-a-large-model-into-a-small-one/).

Read more in [efficient training and inference](/practice/efficient-training-and-inference/).

## Sources

- [Quantization overview](https://huggingface.co/docs/transformers/main/en/quantization/overview), Hugging Face Transformers documentation (2026)
- [Optimizing inference](https://huggingface.co/docs/transformers/main/en/llm_optims), Hugging Face Transformers documentation (2026)
- [LLM.int8(): 8-bit matrix multiplication for transformers at scale](https://arxiv.org/abs/2208.07339), Dettmers, Lewis, Belkada and Zettlemoyer, NeurIPS (2022)
- [Smarter, smaller, stronger: resource-efficient generative AI and the future of digital transformation](https://www.unesco.org/en/articles/ai-large-language-models-new-report-shows-small-changes-can-reduce-energy-use-90), UNESCO and University College London (2025)
- [GGUF](https://huggingface.co/docs/hub/gguf), Hugging Face Hub documentation (2026)
