In short
Use two layers. Provider prompt caching reuses the processing of a fixed prefix, such as a long system prompt or shared document, so put stable content first. Your own response cache stores answers to identical requests so the model is not called again. Both cut computation, cost and latency, as long as you set sensible expiry times.
Steps
- Put stable content, such as instructions and reference text, at the start of every prompt.
- Turn on or mark prompt caching as your provider's documentation describes.
- Cache full responses for identical inputs in your own application.
- Set expiry times that match how often the underlying answer changes.
- Log cache hit rates and input tokens so you can see the saving.
The cheapest model call is the one you don't make, and the next cheapest reuses work already done. Caching does both.
What is prompt caching?
The provider stores the processed form of the start of your prompt and reuses it on later calls. Anthropic's documentation says prompt caching "significantly reduces processing time and costs" for prompts with consistent elements, prices cache reads at a tenth of normal input tokens, and keeps the cache for five minutes by default. OpenAI enables prompt caching by default for supported models. Both work only on an identical prefix, so order matters: stable instructions and documents first, the changing question last.
What is response caching?
You store the model's answer and return it when the same request comes in again. It suits FAQs, classification of repeated items, and agents that re-ask identical questions. The Green Software Foundation's pattern on agent orchestration recommends cutting redundant model calls and repeated computation in exactly this way.
Some teams also use "semantic" caches that match similar rather than identical questions. These save more but can return a wrong answer for a subtly different question, so test them carefully.
Common mistakes
- Putting timestamps or user names at the top of the prompt, which breaks the prefix and the cache.
- Caching for ever. Set an expiry that fits how fast the answer goes stale.
- Not measuring. Without hit rates you won't know whether the cache helps.
For the wider idea of reusing rather than regenerating, read orchestration over generation.
Sources
- Prompt caching, Anthropic, Claude platform documentation (2026)
- Prompt caching, OpenAI API documentation (2026)
- Optimize agent orchestration to reduce unnecessary model calls, Green Software Foundation, Green Software Patterns (2025)
Related
How do I stop my chatbot resending the whole conversation?
How to keep chatbot context small: cap past turns, summarise older messages, retrieve what's relevant and cache fixed prompts, so input stops growing.
How do I stop AI agents wasting tokens?
Practical guardrails for AI agents: stop conditions, step and token limits, loop detection, context trimming and small models for simple steps.
Orchestration over generation
Why reusing vetted components and routing requests to the smallest capable model can cut AI's energy use, and what the evidence says so far.
Use AI efficiently: everyday habits
Everyday habits that cut the energy of using AI: clear prompts, fewer regeneration loops, care with images and video, caching and turning off defaults.