Context caching: how to cut LLM inference cost

Most LLM applications send the same text over and over. A long system prompt, a set of tool definitions, a large shared document, a few few-shot examples — all of it identical on every request, with only the user's last message changing. The model dutifully re-reads and re-processes that unchanging prefix every single time, and you pay for it every single time. Context caching is the fix, and on the right workload it cuts both latency and cost by a large margin.

What actually gets cached

When a transformer processes a prompt, it computes internal attention state — the key and value tensors, the "KV cache" — for every token. That computation is the expensive prefill work. The insight behind context caching is that if two requests share an identical prefix, the KV state for that prefix is also identical. So the provider computes it once, stores it, and on the next request with the same prefix it reuses the stored state instead of recomputing from token zero.

The critical constraint: caching works on an exact, contiguous prefix. Because each token's state depends on everything before it, the cache matches from the start of the prompt up to the first byte that differs. Put your stable content first and your variable content last, and the cache does its job. Flip that order and you get nothing.

Cache-hit flow — an exact prefix match reuses computed state; anything past the divergence point is recomputed.

The cost math

Cached input tokens are billed at a steep discount — commonly on the order of one-tenth the normal input price, though the exact ratio and any cache-write surcharge vary by provider, so treat these as illustrative. Suppose a request has a 4,000-token shared prefix (system prompt, tool specs, a reference doc) and 200 tokens of fresh user input.

Without caching: you pay full input price on all 4,200 tokens, every call.
With caching: you pay full price on the 200 new tokens and roughly a tenth on the 4,000 cached ones — the equivalent of about 600 billed input tokens instead of 4,200.

That is close to a 7× reduction on the input portion of the bill for a prefix-heavy workload, and prefill latency drops in step because the model skips the work entirely. The savings scale with how large your stable prefix is relative to the variable part — which is exactly the regime agents, RAG, and long-document chat live in.

Illustrative input cost per call — the same prefix-heavy request, before and after caching.

Where it pays off — and a real-world framing

Caching earns its keep whenever a large chunk of context repeats within the cache's lifetime. Multi-turn chat over a fixed document, tool-using agents that resend the same tool schema on every step, batch classification with a shared instruction block, and long system prompts are all natural fits.

It matters most where latency and per-call cost compound. In a real-time voice pipeline, for instance, every turn re-sends the system prompt and conversation history, and the per-minute economics are unforgiving. Combining aggressive context trimming— dropping history you no longer need — with caching of the stable prefix is a practical way to push voice-call inference toward a target like ~$0.06 a minute. I'd frame that as an engineering target you approach by removing waste, not a headline number that holds for every model or provider.

ordering for cache hits (sketch)

messages = [
  system_prompt,        # stable  ── cached
  tool_definitions,     # stable  ── cached
  shared_reference_doc, # stable  ── cached
  ...conversation_tail, # trimmed to recent turns
  user_message,         # variable ── never cached
]

Provider support and pitfalls

Major providers support prompt caching, but the ergonomics differ. Some cache automatically once a prefix crosses a minimum token length; others require you to mark cache breakpoints explicitly. Cache entries are also short-lived — often minutes of idle time — so caching helps sustained traffic far more than the occasional one-off request. Check your provider's minimum prefix length, time-to-live, and whether writing to the cache carries a surcharge.

Context caching is one of the highest-return, lowest-risk optimizations available for LLM systems: no model change, no quality trade-off, just a disciplined prompt layout that puts what repeats first. On prefix-heavy workloads it is often the difference between an application that is too expensive to run and one that isn't.