In short
Cloud providers sell spare capacity cheaply on condition they can take it back at short notice, as spot or preemptible instances. Training, evaluation, batch inference and data processing can use it if they save checkpoints and retry cleanly. Microsoft's guidance is to use it only for interrupt-tolerant work, because the interruptions can add instability.
Steps
- List AI workloads that can be stopped and restarted, such as training, evaluation and batch inference.
- Add regular checkpoints and clean retry logic to those jobs.
- Run them on spot or preemptible capacity across several instance types.
- Keep interactive or deadline-critical work on standard capacity.
- Measure how much work is lost to interruptions and adjust the checkpoint interval.
Spare capacity is hardware that is already built and powered; using it for flexible work makes better use of it. It is also usually much cheaper, which helps the case for doing it.
Which AI jobs suit it?
- training and fine-tuning runs that checkpoint regularly;
- hyperparameter searches, where each trial is independent;
- evaluation and benchmark runs;
- batch inference over a queue of documents or images;
- data preparation.
Microsoft's Azure Well-Architected sustainability guidance says to "use Spot VMs only for interrupt-tolerant workloads", noting that they can reduce infrastructure waste but can also introduce instability.
How do I make a job interruption-proof?
- Save checkpoints, including optimiser state, at a sensible interval.
- Make the job pick up from the last checkpoint automatically when restarted.
- Allow several instance types so the scheduler has more spare capacity to draw on.
- Write outputs incrementally, so finished work isn't lost.
Checkpointing also lets you pause jobs when the grid is dirty, as the Green Software Foundation's pattern for AI workloads suggests; see scheduling training when the grid is greener.
Common mistakes
- No checkpoints, so an interruption throws away hours of work and the energy is wasted.
- Checkpointing too often, so saving costs more than it protects.
- Putting user-facing services on spot capacity.
Sources
- Reliability recommendations for sustainable workloads on Azure, Microsoft Azure Well-Architected Framework (2026)
- Use carbon-aware scheduling and region selection for AI workloads, Green Software Foundation, Green Software Patterns (2025)
Related
How do I stop GPUs and notebooks sitting idle?
Stop idle AI hardware wasting energy: start resources on demand, auto-stop idle notebooks, scale to zero out of hours and report idle GPU-hours.
How do I schedule training when the grid is greener?
Run training and batch AI jobs when grid carbon intensity is low: use forecasts, make jobs pausable with checkpoints, and keep deadlines as hard limits.
How do I batch AI requests to save energy?
How batching raises accelerator utilisation and cuts energy per answer: continuous batching for live traffic and queued batches for offline work.
Green AI hosting and data centres
How to choose where AI runs: grid carbon intensity, PUE and WUE, renewable energy claims, water use and siting, with questions to ask hosts.