In short
A mixture-of-experts model routes each token to a few specialist sub-networks, so only part of the model does work per token. Mixtral, for example, has 47 billion parameters but uses 13 billion per token. That can cut computation per answer compared with a dense model of similar quality, but every expert must still sit in memory, so compare quality, latency and energy per request on your own hardware.
Steps
- Shortlist a mixture-of-experts model and a dense model of similar quality for your task.
- Check both fit your available GPU memory, counting all experts.
- Run both on your evaluation set at your real batch size.
- Compare quality, latency and energy per request side by side.
- Choose on all three, and consider a smaller dense model if it passes.
Mixture-of-experts (MoE) models can deliver large-model quality with less computation per token, but not less memory. Whether they save energy for you depends on your hardware and traffic.
How does a mixture of experts work?
A dense model uses every parameter for every token. An MoE model has many "expert" sub-networks and a small router that picks a few experts per token. Jiang and colleagues' Mixtral of Experts gives each token access to 47 billion parameters but uses only 13 billion active parameters during inference, and the authors report it outperforms or matches Llama 2 70B and GPT-3.5 across the benchmarks they evaluated. See also the glossary entry.
When does it save energy?
- Compared with a dense model of similar quality, fewer active parameters usually means less computation per token.
- When you have the memory. All experts still have to be loaded, so you may need as many GPUs as for the dense model. Idle memory still draws power.
- At good utilisation. With batching, different tokens use different experts, which keeps hardware busier.
Common mistakes
- Comparing with the wrong baseline. If a small dense model meets your quality bar, it will usually beat both. See how to choose a smaller AI model.
- Counting active parameters as the whole cost. Memory, communication between GPUs and routing overhead count too.
- Not measuring. Where public ratings such as the AI Energy Score cover your candidates, use them; otherwise benchmark both on the same hardware.
For the wider method, read choose the right model.
Sources
- Mixtral of experts, Jiang et al., Mistral AI (2024)
- AI Energy Score, Hugging Face (2025)
Related
How do I compare the energy use of different AI models?
How to compare AI models on energy fairly: use benchmarks like the AI Energy Score, or measure them yourself on the same task, hardware and settings.
How do I choose a smaller AI model?
How to pick the smallest AI model that does your task well: set a quality bar, test small to large, check energy ratings and keep a big model as fallback.
How do I cut the energy of serving a model in production?
How to serve a self-hosted model with less energy per request: efficient runtimes, batching, half precision, compiled caches and tested hardware.
Choose the right model for the job
How to right-size AI: when to use small or specialised models, when a frontier model earns its cost, how routing works, and when not to use AI.