# How do I make a model small enough for an edge device?

> Shrink a model for phones, sensors and embedded boards: start small, distil, quantise and prune, then test accuracy and energy on the real device.

Source: https://greenai.guide/how-do-i/make-a-model-small-enough-for-an-edge-device/  
Updated: 2026-11-19

## In short

Start from the smallest model that could meet your quality bar, then shrink it further with distillation, quantisation and pruning, testing each step on the target device. Measure energy per inference, latency and accuracy together on the real hardware, and let the model step down when battery is low or the device is hot.

## Steps

1. Set a quality bar and a memory, latency and energy budget for the target device.
2. Start from a small or distilled model rather than shrinking a large general one.
3. Quantise the weights to 8 or 4 bits and check accuracy on your own test cases.
4. Prune weights that contribute little, then re-test on the device.
5. Measure energy per inference, latency and accuracy together on the real hardware.
6. Switch to a smaller model or a lower rate when the battery is low or the device is hot.

**Combine a small starting model with distillation, quantisation and pruning, and judge every step on the device
itself.** Edge devices have little memory and often run on batteries, so a model that is too big either won't fit
or drains the device and throttles. Getting the size right also means data need not be sent to the cloud.

## Which techniques shrink a model most?

- **Distillation** trains a small "student" to copy a large "teacher". DistilBERT was 40% smaller than BERT and 60%
  faster while keeping 97% of its language-understanding ability.
- **Quantisation** stores weights in fewer bits. Hugging Face's documentation gives the example of an 8-billion
  parameter model needing about 16 GB for its weights at 16-bit precision, about 8 GB at 8 bits and about 4–5 GB at
  4 bits, at some cost in accuracy. See [how to quantise a model](/how-do-i/quantise-a-model/).
- **Pruning** removes weights that barely affect the output. See [how to prune a neural network](/how-do-i/prune-a-neural-network/).

## How should the model behave on the device?

The Green Software Foundation's edge AI pattern recommends tracking energy per inference, latency and accuracy
together, and adapting to conditions: a smaller model or lower inference frequency when the battery is low or the
device is hot.

## Common mistakes

- **Testing only on a laptop.** Accelerators, memory and thermal limits differ; measure on the target hardware.
- **Shrinking past the quality bar.** A model that fails and has to call the cloud as a fallback can cost more than
  it saves.
- **Running inference constantly.** Trigger it on events where you can, instead of every frame or second.

Whether edge or cloud is greener depends on the case: see [edge or cloud](/practice/edge-or-cloud/) and
[AI at the edge](/articles/ai-at-the-edge/).

## Sources

- [Optimize power consumption for edge AI inference](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/operations/optimize-edge-ai-power-performance.md), Green Software Foundation, Green Software Patterns (2025)
- [Run AI models at the edge](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/architecture/system-topology/run-ai-models-edge.md), Green Software Foundation, Green Software Patterns (2025)
- [Quantization overview](https://huggingface.co/docs/transformers/main/en/quantization/overview), Hugging Face Transformers documentation (2026)
- [DistilBERT, a distilled version of BERT](https://arxiv.org/abs/1910.01108), Sanh et al. (arXiv 1910.01108) (2019)
