greenai.guide

Home › Articles › The maths that makes AI more efficient

Article

The maths that makes AI more efficient

Updated · 6 min read

In short

Much of the energy AI uses goes on computation and data movement that the task does not need. Mathematical methods such as pruning, sparsity, randomised sketching, random matrix theory and models with built-in structure remove that redundancy, often with little or no loss of accuracy. They save energy only when the savings are measured and not simply spent on bigger models.

Key points

  • Large neural networks are over-parameterised; pruning can remove most of their weights after training with little loss of accuracy.
  • Sparsity saves energy only when hardware and software can skip the zeros; otherwise it is a saving on paper.
  • Moving data often costs more energy than arithmetic, so methods that reduce memory traffic, such as sketching and IO-aware attention, matter.
  • Building known physics or symmetry into a model can cut the parameters and data it needs.
  • Classical numerical methods are sometimes the greener answer, and a hybrid pipeline can use a small network only where it helps.

Most of the energy a neural network uses goes on two things: arithmetic, and moving numbers between memory and the processor. Mathematics can reduce both. This article explains the main methods in plain terms, with the published papers behind them, and where each one stops paying off.

A 2026 UK green AI sandpit of mathematicians, computer scientists and industry put this question at its centre: how can the mathematical sciences reduce the energy and resources AI needs as demand grows? The methods below are the ones that came up most often, set against the published evidence.

Why is there redundancy to remove?

Large models are bigger than the task strictly needs, because size makes them easier to train. Training works better in a high-dimensional space, but once training is done, many of the weights add little.

That leaves room for three kinds of saving: fewer weights (pruning), fewer active values at any moment (sparsity), and less data movement (sketching and memory-aware algorithms). A fourth kind avoids the redundancy in the first place, by building known structure into the model.

What is pruning, and how much can it remove?

Pruning deletes the weights that contribute least, then lets the network recover. Han and colleagues showed in 2015 that pruning reduced the parameters of the AlexNet image model by a factor of nine and of VGG-16 by a factor of thirteen, with no loss of accuracy.

Later work made the idea sharper:

  • The lottery ticket hypothesis (Frankle and Carbin, 2019) found that large networks contain much smaller sub-networks that, trained from the right starting point, match the full network's accuracy.
  • SparseGPT (Frantar and Alistarh, 2023) pruned very large language models to at least 50% sparsity in one shot, without retraining, with little increase in perplexity.
  • Hoefler and colleagues' 2021 survey of more than 300 papers is the standard guide to when pruning works and when it does not.

Does sparsity actually save energy?

Only if the zeros are skipped. A matrix that is 80% zeros still costs the full amount if the hardware multiplies every entry. It helps to distinguish unstructured sparsity (zeros scattered anywhere) and structured sparsity (whole blocks, channels or patterns of zeros), which standard chips can exploit far more easily.

There are two places to find sparsity:

  • In the weights, through pruning.
  • In the activations, the values that flow through the network for a given input. Kurtz and colleagues (2020) showed that raising the threshold of the ReLU activation function makes many more activations exactly zero, which can then be skipped at inference. Mirzadeh and colleagues (2024) argued that large language models using ReLU activations can exploit the same effect to cut computation.

Activation sparsity is dynamic: which values are zero changes with every input. That makes it harder to exploit in hardware, and it is one reason the sandpit treated hardware–software co-design as part of the mathematics, not an afterthought.

Why does data movement matter so much?

Fetching a number from memory can cost more energy than doing arithmetic with it. Algorithms that move less data often save more than algorithms that do fewer sums.

Two families of methods attack this:

  • Randomised sketching. A sketch is a much smaller random projection of a large matrix that keeps the properties you need. Halko, Martinsson and Tropp's 2011 paper and Martinsson and Tropp's 2020 survey set out the theory of randomised numerical linear algebra, which gives provable error bounds for these approximations. Applied to neural networks, sketches can compress large intermediate results before they are moved off-chip.
  • IO-aware algorithms. FlashAttention (Dao and colleagues, 2022) computes exactly the same attention result as the standard method but reorganises the work to reduce reads and writes between a GPU's large, slow memory and its small, fast memory. The saving comes from scheduling, not approximation.

What does random matrix theory add?

Random matrix theory describes how large random matrices behave, which is what a neural network's weights are at the start of training. That lets mathematicians predict properties of a network before training it.

Pennington and Worah (2017) used it to analyse how signals pass through wide networks. Martin and Mahoney (2021) studied the spectra of trained weight matrices and found patterns that indicate how well a layer has learned. Results of this kind help set sparsity levels and diagnose layers that are larger than they need to be, without the trial and error of training many variants.

Can structure replace data and parameters?

Yes, where the structure is known. If a problem obeys a physical law or a symmetry, a model that has that knowledge built in does not have to learn it from data.

  • Physics-informed neural networks (Raissi, Perdikaris and Karniadakis, 2019) add the governing equations to the training objective, so the network is penalised for physically impossible answers.
  • Geometric deep learning (Bronstein and colleagues, 2021) designs networks that respect symmetries such as rotation or permutation, so they need fewer parameters and examples.
  • Algorithm unrolling (Monga, Li and Eldar, 2021) turns the steps of a classical iterative algorithm into the layers of a network, which gives small, interpretable models.

A related idea came up repeatedly at the sandpit: hybrid pipelines. Keep the classical or physics-based method for the steps it does well and use a small network only for the part that suits learning, such as a surrogate for one expensive component of a simulation. Sometimes the greenest answer is an existing numerical method and no neural network at all.

Where do automatic differentiation and quantisation fit?

Automatic differentiation (AD) computes exact derivatives of a program, and it is the engine of training. Backpropagation is reverse-mode AD, as Baydin and colleagues' 2018 survey explains. Efficient AD matters for green AI in its own right, and it is what makes techniques such as pruning-aware training and differentiable energy penalties possible.

Quantisation stores numbers with fewer bits, for example 8-bit integers instead of 32-bit floating point. Jacob and colleagues (2018) showed how to train networks for integer-only inference, which cuts memory and energy per operation. Fewer bits also means less data to move.

Can we reuse models instead of training new ones?

Often. Wortsman and colleagues (2022) showed that averaging the weights of several fine-tuned versions of the same model, a "model soup", can improve accuracy without any extra cost at inference. Merging and adapting existing models avoids repeating the cost of training from scratch. Orchestration over generation takes the same reuse idea further.

What are the limits?

Efficiency gains are real, but they are not automatically savings. Three caveats apply to every method above:

  • A saving on paper (fewer parameters, fewer operations) must be confirmed by measuring energy on real hardware. See measuring AI energy.
  • Accuracy must be checked on the actual task, not just a benchmark average.
  • Cheaper computation tends to be used for more computation. Efficiency is not sufficiency explains this rebound effect.

What to do next

  • If you train or fine-tune models, try post-training pruning and quantisation first: they are the cheapest wins. See efficient training and inference.
  • Check whether your hardware and libraries actually exploit sparsity before counting on it.
  • Before reaching for a neural network, ask whether a classical method, or a hybrid, already solves the problem.
  • Measure energy before and after any change, on the hardware you will use.
  • Researchers: look for funded projects and PhDs in efficient AI on the research and PhDs listings, and see the guide for researchers.

Frequently asked questions

What is pruning in AI?

Pruning removes weights or whole neurons from a trained neural network that contribute little to its output. The smaller network needs less memory and, on suitable hardware, less computation and energy.

Does a sparse model always use less energy?

No. A model full of zeros saves energy only if the hardware and software skip them. Unstructured sparsity on a standard GPU can run no faster than the dense model, which is why structured sparsity and sparse kernels matter.

What is a physics-informed neural network?

A neural network trained to respect known physical laws, usually by adding the governing equations to its loss function. Building that knowledge in means the network needs less data to learn a useful solution.

Can mathematics really make a difference next to better chips?

Yes, and the two combine. Better algorithms reduce the work to be done; better hardware does that work more cheaply. Several of the largest savings in AI, such as quantisation and efficient attention, came from mathematics and algorithms rather than new chips.

Sources

  1. Learning both weights and connections for efficient neural networks, Han, Pool, Tran and Dally, NeurIPS (2015)
  2. The lottery ticket hypothesis: finding sparse, trainable neural networks, Frankle and Carbin, ICLR (2019)
  3. Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks, Hoefler, Alistarh, Ben-Nun, Dryden and Peste, Journal of Machine Learning Research (2021)
  4. SparseGPT: massive language models can be accurately pruned in one-shot, Frantar and Alistarh, ICML (2023)
  5. Inducing and exploiting activation sparsity for fast inference on deep neural networks, Kurtz et al., ICML (2020)
  6. ReLU strikes back: exploiting activation sparsity in large language models, Mirzadeh et al., ICLR (2024)
  7. Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions, Halko, Martinsson and Tropp, SIAM Review (2011)
  8. Randomized numerical linear algebra: foundations and algorithms, Martinsson and Tropp, Acta Numerica (2020)
  9. FlashAttention: fast and memory-efficient exact attention with IO-awareness, Dao, Fu, Ermon, Rudra and Ré, NeurIPS (2022)
  10. Implicit self-regularization in deep neural networks: evidence from random matrix theory and implications for learning, Martin and Mahoney, Journal of Machine Learning Research (2021)
  11. Nonlinear random matrix theory for deep learning, Pennington and Worah, NeurIPS (2017)
  12. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Raissi, Perdikaris and Karniadakis, Journal of Computational Physics (2019)
  13. Geometric deep learning: grids, groups, graphs, geodesics, and gauges, Bronstein, Bruna, Cohen and Veličković (2021)
  14. Algorithm unrolling: interpretable, efficient deep learning for signal and image processing, Monga, Li and Eldar, IEEE Signal Processing Magazine (2021)
  15. Automatic differentiation in machine learning: a survey, Baydin, Pearlmutter, Radul and Siskind, Journal of Machine Learning Research (2018)
  16. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, Wortsman et al., ICML (2022)
  17. Quantization and training of neural networks for efficient integer-arithmetic-only inference, Jacob et al., CVPR (2018)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy