greenai.guide

Home › How do I…? › How do I make a model small enough for an edge device?

How do I…?

How do I make a model small enough for an edge device?

Updated · 1 min read

In short

Start from the smallest model that could meet your quality bar, then shrink it further with distillation, quantisation and pruning, testing each step on the target device. Measure energy per inference, latency and accuracy together on the real hardware, and let the model step down when battery is low or the device is hot.

Steps

  1. Set a quality bar and a memory, latency and energy budget for the target device.
  2. Start from a small or distilled model rather than shrinking a large general one.
  3. Quantise the weights to 8 or 4 bits and check accuracy on your own test cases.
  4. Prune weights that contribute little, then re-test on the device.
  5. Measure energy per inference, latency and accuracy together on the real hardware.
  6. Switch to a smaller model or a lower rate when the battery is low or the device is hot.

Combine a small starting model with distillation, quantisation and pruning, and judge every step on the device itself. Edge devices have little memory and often run on batteries, so a model that is too big either won't fit or drains the device and throttles. Getting the size right also means data need not be sent to the cloud.

Which techniques shrink a model most?

  • Distillation trains a small "student" to copy a large "teacher". DistilBERT was 40% smaller than BERT and 60% faster while keeping 97% of its language-understanding ability.
  • Quantisation stores weights in fewer bits. Hugging Face's documentation gives the example of an 8-billion parameter model needing about 16 GB for its weights at 16-bit precision, about 8 GB at 8 bits and about 4–5 GB at 4 bits, at some cost in accuracy. See how to quantise a model.
  • Pruning removes weights that barely affect the output. See how to prune a neural network.

How should the model behave on the device?

The Green Software Foundation's edge AI pattern recommends tracking energy per inference, latency and accuracy together, and adapting to conditions: a smaller model or lower inference frequency when the battery is low or the device is hot.

Common mistakes

  • Testing only on a laptop. Accelerators, memory and thermal limits differ; measure on the target hardware.
  • Shrinking past the quality bar. A model that fails and has to call the cloud as a fallback can cost more than it saves.
  • Running inference constantly. Trigger it on events where you can, instead of every frame or second.

Whether edge or cloud is greener depends on the case: see edge or cloud and AI at the edge.

Sources

  1. Optimize power consumption for edge AI inference, Green Software Foundation, Green Software Patterns (2025)
  2. Run AI models at the edge, Green Software Foundation, Green Software Patterns (2025)
  3. Quantization overview, Hugging Face Transformers documentation (2026)
  4. DistilBERT, a distilled version of BERT, Sanh et al. (arXiv 1910.01108) (2019)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy