In short
Quantising stores a model's weights with fewer bits, typically 8 or 4 instead of 16 or 32, so it needs less memory and often less energy per answer. Load a ready-quantised version or quantise with your library's built-in methods, then test it on your own examples against the full-precision model. Keep it if quality holds and energy per request falls.
Steps
- Check whether a ready-quantised version of your model already exists.
- Otherwise quantise with a supported method in your library, starting at 8 bits.
- Run the quantised and original models on the same evaluation set.
- Try 4 bits if 8-bit quality holds and you need to save more memory.
- Measure energy and latency per request at your real batch size before switching.
Quantisation is often the quickest efficiency win for a model you run yourself. It shrinks memory, so the model fits on fewer or smaller GPUs, and it can cut energy per answer.
Why does it matter?
Hugging Face's documentation gives a sense of scale: an 8-billion-parameter model needs about 16 GB for its weights at 16-bit precision, about 8 GB at 8 bits and about 4 to 5 GB at 4 bits. Dettmers and colleagues showed with LLM.int8() that 8-bit inference can cut the memory needed by half while retaining full-precision performance, for models up to 175 billion parameters. UNESCO and University College London report that compression such as quantisation can save up to 44% of energy.
How do I do it?
- Download a quantised version. Many open models are published in quantised formats. GGUF, used by llama.cpp, offers quantisation types from about 1.5 to 8 bits per weight.
- Quantise on load. Libraries such as Hugging Face Transformers support more than 20 quantisation methods, several of which quantise weights as the model loads.
- Calibrate where needed. Some methods use a small sample of your data to choose how to round, which tends to preserve quality better at 4 bits.
Common mistakes
- Skipping the quality test. Fewer bits cost some accuracy, and each method trades memory, speed and accuracy differently. Test on your own cases.
- Assuming it is always faster. Hugging Face notes that if you aren't limited by GPU memory, quantisation can increase latency slightly because of the extra quantise and dequantise steps. Measure energy, not just speed.
- Stopping at quantisation. Combine it with a smaller model or distillation.
Read more in efficient training and inference.
Sources
- Quantization overview, Hugging Face Transformers documentation (2026)
- Optimizing inference, Hugging Face Transformers documentation (2026)
- LLM.int8(): 8-bit matrix multiplication for transformers at scale, Dettmers, Lewis, Belkada and Zettlemoyer, NeurIPS (2022)
- Smarter, smaller, stronger: resource-efficient generative AI and the future of digital transformation, UNESCO and University College London (2025)
- GGUF, Hugging Face Hub documentation (2026)
Related
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.
How do I run an AI model locally?
How to run an open AI model on your own laptop or server with tools like Ollama or llama.cpp, choose a quantised model that fits, and measure its energy.
How do I cut the energy of serving a model in production?
How to serve a self-hosted model with less energy per request: efficient runtimes, batching, half precision, compiled caches and tested hardware.
How do I prune a neural network?
How to prune a trained neural network to remove weights that contribute little, test what accuracy survives, and check the hardware really saves.