In short
Prune a trained network by removing the weights, neurons or layers that contribute least, then fine-tune briefly so it recovers, and test it against your quality bar. Pruning cut AlexNet's parameters ninefold with no loss of accuracy, and very large language models have been pruned to at least 50% sparsity in one shot. Prefer structured pruning if you want ordinary hardware to run faster.
Steps
- Measure the accuracy, latency and energy of the unpruned model first.
- Choose structured pruning (whole neurons, heads or layers) unless your hardware exploits sparse weights.
- Remove a modest share of weights by importance, such as smallest magnitude.
- Fine-tune briefly to let the network recover.
- Repeat in steps, testing each time, and stop when quality drops below your bar.
Large networks usually carry far more weights than their task needs, so many can be removed after training. Pruning makes the model smaller and, on the right hardware, cheaper to run.
What does the evidence show?
Han and colleagues showed in 2015 that pruning reduced the parameters of the AlexNet image model by a factor of nine and of VGG-16 by a factor of thirteen, with no loss of accuracy. Frankle and Carbin's lottery ticket hypothesis found that large networks contain much smaller sub-networks that can match the full network's accuracy. For language models, Frantar and Alistarh's SparseGPT pruned very large models to at least 50% sparsity in one shot, without retraining, with little increase in perplexity.
Structured or unstructured?
- Unstructured pruning removes individual weights. It can remove the most, but leaves scattered zeros that ordinary GPUs and CPUs often can't skip.
- Structured pruning removes whole neurons, attention heads, channels or layers. It removes less, but the smaller model runs faster on any hardware.
Common mistakes
- Counting zeros instead of measuring energy. Theoretical savings often exceed measured ones. A 2026 UK green AI sandpit of mathematicians, computer scientists and industry noted that sparsity remains under-used partly because mainstream chips struggle to exploit it.
- Pruning too much at once. Go in steps and fine-tune between them.
- Testing on benchmarks only. Check your own cases, including rare ones, which pruning can hurt first.
Hoefler and colleagues' 2021 survey of more than 300 papers is the standard guide. For the theory, read maths for efficient AI.
Sources
- Learning both weights and connections for efficient neural networks, Han, Pool, Tran and Dally, NeurIPS (2015)
- SparseGPT: massive language models can be accurately pruned in one-shot, Frantar and Alistarh, ICML (2023)
- Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks, Hoefler, Alistarh, Ben-Nun, Dryden and Peste, Journal of Machine Learning Research (2021)
- The lottery ticket hypothesis: finding sparse, trainable neural networks, Frankle and Carbin, ICLR (2019)
Related
The maths that makes AI more efficient
Pruning, sparsity, sketching, random matrix theory and physics-informed models: how mathematics cuts the computation, memory and energy AI needs.
How do I quantise a model?
How to quantise an AI model to 8 or 4 bits: which tools to use, how to test quality, and when quantisation saves energy and when it doesn't.
How do I distil a large model into a smaller one?
How knowledge distillation works: use a large teacher model to train a small student for your task, then test it and keep the large model as a fallback.
Efficient AI training and inference
Techniques for developers and researchers to cut AI energy: pruning, sparsity, quantisation, distillation, KV cache, batching, mixed precision and scheduling.