In short
Make on-demand the default: pipelines and training jobs should start resources when triggered and release them when done. Shut down idle notebooks and development GPUs automatically after a set time, scale non-production environments to zero out of hours, and report idle GPU-hours per team next to cost so the waste is visible.
Steps
- Make pipelines and training jobs start resources when triggered and release them when they finish.
- Set notebooks and development GPUs to shut down automatically after a period of inactivity.
- Scale non-production environments to zero outside working hours.
- Give temporary environments an automatic destroy date.
- Report idle GPU-hours per team alongside cost.
An idle GPU still draws power and still ties up hardware that took energy to make. Notebooks, clusters and test environments set up for one experiment are easily left running between experiments, or forgotten.
What should be on demand?
The Green Software Foundation's patterns for AI and agent workloads recommend running infrastructure only while it works: serverless or on-demand execution for model development, pipelines and agents, so resources start when triggered and stop when finished.
What about development and test?
Microsoft's Azure Well-Architected guidance says to "shut down non-production environments when not in use". On Kubernetes, KEDA's cron scaler can scale workloads up during a time window and to zero outside it. Exclude only the shared services that genuinely must stay up, and write down why.
What makes the biggest difference
- Auto-shutdown on notebooks, which are the most commonly forgotten GPU users.
- Visibility. Put idle GPU-hours on the same report as spend; teams act on what they can see.
- Sharing. A shared pool or queue usually keeps GPUs busier than one machine per person.
Common mistakes
- Reserving big GPUs "just in case" for work that runs once a week.
- Keeping models loaded in memory permanently for an internal tool used a few times a day.
See hosting and data centres and efficient training and inference.
Sources
- Serverless architecture for AI/ML workload processes, Green Software Foundation, Green Software Patterns (2025)
- Use on-demand execution for AI and agent workloads, Green Software Foundation, Green Software Patterns (2025)
- Reliability recommendations for sustainable workloads on Azure, Microsoft Azure Well-Architected Framework (2026)
- Cron scaler, KEDA (CNCF) (2025)
Related
How do I run AI jobs on spare cloud capacity?
Run interruptible AI work on spot or preemptible capacity: make jobs checkpoint and retry cleanly, and keep work that can't be interrupted elsewhere.
How do I batch AI requests to save energy?
How batching raises accelerator utilisation and cuts energy per answer: continuous batching for live traffic and queued batches for offline work.
How do I reduce the footprint of AI experiments on a research cluster?
Cut the energy of AI experiments on HPC: read Slurm energy accounting, request only what you use, pilot small, checkpoint and schedule flexible jobs.
Green AI hosting and data centres
How to choose where AI runs: grid carbon intensity, PUE and WUE, renewable energy claims, water use and siting, with questions to ask hosts.