In short
Start from the smallest model that could meet your quality bar, then shrink it further with distillation, quantisation and pruning, testing each step on the target device. Measure energy per inference, latency and accuracy together on the real hardware, and let the model step down when battery is low or the device is hot.
Steps
- Set a quality bar and a memory, latency and energy budget for the target device.
- Start from a small or distilled model rather than shrinking a large general one.
- Quantise the weights to 8 or 4 bits and check accuracy on your own test cases.
- Prune weights that contribute little, then re-test on the device.
- Measure energy per inference, latency and accuracy together on the real hardware.
- Switch to a smaller model or a lower rate when the battery is low or the device is hot.
Combine a small starting model with distillation, quantisation and pruning, and judge every step on the device itself. Edge devices have little memory and often run on batteries, so a model that is too big either won't fit or drains the device and throttles. Getting the size right also means data need not be sent to the cloud.
Which techniques shrink a model most?
- Distillation trains a small "student" to copy a large "teacher". DistilBERT was 40% smaller than BERT and 60% faster while keeping 97% of its language-understanding ability.
- Quantisation stores weights in fewer bits. Hugging Face's documentation gives the example of an 8-billion parameter model needing about 16 GB for its weights at 16-bit precision, about 8 GB at 8 bits and about 4–5 GB at 4 bits, at some cost in accuracy. See how to quantise a model.
- Pruning removes weights that barely affect the output. See how to prune a neural network.
How should the model behave on the device?
The Green Software Foundation's edge AI pattern recommends tracking energy per inference, latency and accuracy together, and adapting to conditions: a smaller model or lower inference frequency when the battery is low or the device is hot.
Common mistakes
- Testing only on a laptop. Accelerators, memory and thermal limits differ; measure on the target hardware.
- Shrinking past the quality bar. A model that fails and has to call the cloud as a fallback can cost more than it saves.
- Running inference constantly. Trigger it on events where you can, instead of every frame or second.
Whether edge or cloud is greener depends on the case: see edge or cloud and AI at the edge.
Sources
- Optimize power consumption for edge AI inference, Green Software Foundation, Green Software Patterns (2025)
- Run AI models at the edge, Green Software Foundation, Green Software Patterns (2025)
- Quantization overview, Hugging Face Transformers documentation (2026)
- DistilBERT, a distilled version of BERT, Sanh et al. (arXiv 1910.01108) (2019)
Related
How do I quantise a model?
How to quantise an AI model to 8 or 4 bits: which tools to use, how to test quality, and when quantisation saves energy and when it doesn't.
How do I distil a large model into a smaller one?
How knowledge distillation works: use a large teacher model to train a small student for your task, then test it and keep the large model as a fallback.
How do I decide between running AI on my device or in the cloud?
How to weigh on-device (edge) AI against cloud AI for energy, privacy and quality, and which kinds of task suit each, with honest caveats on the evidence.
Edge or cloud: where should AI run?
How to weigh on-device and edge AI against central data centres: energy, network, hardware, privacy and resilience, with a step-by-step way to decide.