# How do I run AI jobs on spare cloud capacity?

> Run interruptible AI work on spot or preemptible capacity: make jobs checkpoint and retry cleanly, and keep work that can't be interrupted elsewhere.

Source: https://greenai.guide/how-do-i/run-ai-jobs-on-spare-cloud-capacity/  
Updated: 2026-11-19

## In short

Cloud providers sell spare capacity cheaply on condition they can take it back at short notice, as spot or preemptible instances. Training, evaluation, batch inference and data processing can use it if they save checkpoints and retry cleanly. Microsoft's guidance is to use it only for interrupt-tolerant work, because the interruptions can add instability.

## Steps

1. List AI workloads that can be stopped and restarted, such as training, evaluation and batch inference.
2. Add regular checkpoints and clean retry logic to those jobs.
3. Run them on spot or preemptible capacity across several instance types.
4. Keep interactive or deadline-critical work on standard capacity.
5. Measure how much work is lost to interruptions and adjust the checkpoint interval.

**Spare capacity is hardware that is already built and powered; using it for flexible work makes better use of
it.** It is also usually much cheaper, which helps the case for doing it.

## Which AI jobs suit it?

- training and fine-tuning runs that checkpoint regularly;
- hyperparameter searches, where each trial is independent;
- evaluation and benchmark runs;
- batch inference over a queue of documents or images;
- data preparation.

Microsoft's Azure Well-Architected sustainability guidance says to "use Spot VMs only for interrupt-tolerant
workloads", noting that they can reduce infrastructure waste but can also introduce instability.

## How do I make a job interruption-proof?

1. Save checkpoints, including optimiser state, at a sensible interval.
2. Make the job pick up from the last checkpoint automatically when restarted.
3. Allow several instance types so the scheduler has more spare capacity to draw on.
4. Write outputs incrementally, so finished work isn't lost.

Checkpointing also lets you pause jobs when the grid is dirty, as the Green Software Foundation's pattern for AI
workloads suggests; see [scheduling training when the grid is greener](/how-do-i/schedule-training-when-the-grid-is-greener/).

## Common mistakes

- **No checkpoints,** so an interruption throws away hours of work and the energy is wasted.
- **Checkpointing too often,** so saving costs more than it protects.
- **Putting user-facing services on spot capacity.**

See [hosting and data centres](/practice/hosting-and-data-centres/).

## Sources

- [Reliability recommendations for sustainable workloads on Azure](https://learn.microsoft.com/en-us/azure/well-architected/sustainability/sustainability-reliability-recommendations), Microsoft Azure Well-Architected Framework (2026)
- [Use carbon-aware scheduling and region selection for AI workloads](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/operations/carbon-aware-ai-scheduling.md), Green Software Foundation, Green Software Patterns (2025)
