greenai.guide

Home › How do I…? › How do I decide whether a mixture-of-experts model will save energy?

How do I…?

How do I decide whether a mixture-of-experts model will save energy?

Updated · 1 min read

In short

A mixture-of-experts model routes each token to a few specialist sub-networks, so only part of the model does work per token. Mixtral, for example, has 47 billion parameters but uses 13 billion per token. That can cut computation per answer compared with a dense model of similar quality, but every expert must still sit in memory, so compare quality, latency and energy per request on your own hardware.

Steps

  1. Shortlist a mixture-of-experts model and a dense model of similar quality for your task.
  2. Check both fit your available GPU memory, counting all experts.
  3. Run both on your evaluation set at your real batch size.
  4. Compare quality, latency and energy per request side by side.
  5. Choose on all three, and consider a smaller dense model if it passes.

Mixture-of-experts (MoE) models can deliver large-model quality with less computation per token, but not less memory. Whether they save energy for you depends on your hardware and traffic.

How does a mixture of experts work?

A dense model uses every parameter for every token. An MoE model has many "expert" sub-networks and a small router that picks a few experts per token. Jiang and colleagues' Mixtral of Experts gives each token access to 47 billion parameters but uses only 13 billion active parameters during inference, and the authors report it outperforms or matches Llama 2 70B and GPT-3.5 across the benchmarks they evaluated. See also the glossary entry.

When does it save energy?

  • Compared with a dense model of similar quality, fewer active parameters usually means less computation per token.
  • When you have the memory. All experts still have to be loaded, so you may need as many GPUs as for the dense model. Idle memory still draws power.
  • At good utilisation. With batching, different tokens use different experts, which keeps hardware busier.

Common mistakes

  • Comparing with the wrong baseline. If a small dense model meets your quality bar, it will usually beat both. See how to choose a smaller AI model.
  • Counting active parameters as the whole cost. Memory, communication between GPUs and routing overhead count too.
  • Not measuring. Where public ratings such as the AI Energy Score cover your candidates, use them; otherwise benchmark both on the same hardware.

For the wider method, read choose the right model.

Sources

  1. Mixtral of experts, Jiang et al., Mistral AI (2024)
  2. AI Energy Score, Hugging Face (2025)

Join us

Help cut AI's impact on the planet

Get new guides, funding calls and directory listings by email, about once a month. Or add your product, course, PhD or research to the directories.

No tracking. Unsubscribe in one click. Other ways to join · Privacy