In short
Most chat features send the whole conversation back to the model on every turn, so input grows with every message. Cap how many past turns you send, summarise older ones into a short note, retrieve only the earlier messages or documents that matter, and cache fixed system prompts. Long inputs cost far more energy than short ones.
Steps
- Log input tokens per turn so growth is visible.
- Send only the last few turns verbatim.
- Replace older turns with a short running summary.
- Retrieve earlier messages or documents only when they are relevant to the new question.
- Use prompt caching for the fixed system prompt and shared context.
Send the model what it needs for this turn, not everything that has ever been said. Context that grows without limit is one of the most common hidden costs in chat features.
Why does it matter?
Energy rises with the amount of text a model reads. Delavande, Pierrard and Luccioni found energy grows linearly with prompt and generation length. Epoch AI estimates that a typical query of about 0.3 Wh rises to about 2.5 Wh with a 10,000-token input and almost 40 Wh with a 100,000-token input. A long support chat that resends every turn can drift towards the expensive end without anyone noticing.
How do I keep context small?
- A sliding window. Keep the last few turns word for word.
- A running summary. After each few turns, ask a small model (or your own code) to compress older history into a short note of decisions and facts.
- Retrieval. Store past messages and documents, and fetch only those relevant to the new question.
- Prompt caching. For fixed instructions, see how to use caching.
- Start fresh. Offer a "new topic" button that starts a clean context.
Common mistakes
- Re-attaching the same document every turn instead of retrieving the relevant passage.
- Summaries that lose key facts. Test that answers stay correct after summarisation.
- Not measuring. Without input-token logs, growth is invisible.
Users can do the same in consumer tools: see how to stop long AI chats wasting energy. For more habits, read use AI efficiently.
Sources
- How much energy does ChatGPT use?, Epoch AI (2025)
- Small talk, big impact: the energy cost of thanking AI, Delavande, Pierrard and Luccioni (arXiv 2601.22357) (2026)
- What are tokens and how to count them?, OpenAI Help Center (2025)
Related
How do I use caching to cut the energy of LLM calls?
Two kinds of caching that cut LLM work: provider prompt caching for repeated context, and your own cache so identical questions aren't answered twice.
How do I stop long AI chats wasting energy?
Why very long AI chat threads cost more with every message, and simple habits to keep them lean: new chats, short recaps and pasting only what matters.
How do I limit how much an LLM writes?
How developers can cap LLM output: set max tokens, ask for a format and length, use stop sequences and structured output, and handle cut-off answers.
Use AI efficiently: everyday habits
Everyday habits that cut the energy of using AI: clear prompts, fewer regeneration loops, care with images and video, caching and turning off defaults.