Cache LLM Responses Without Serving Stale Answers

You will build an application-level cache for LLM responses that survives contact with reality: a cache key that captures everything that makes two requests different, versioning that lets you invalidate by changing one string, per-tenant isolation, and clear rules for the two situations where caching is a liability instead of an optimisation, semantic matching and anything involving fresh data.

The naive version of this tutorial is ten lines long and wrong. Hash the prompt, store the response, serve it forever. It demos beautifully and then does one of three things in production: serves an answer generated from last month’s system prompt, leaks one customer’s answer to another, or confidently returns information the underlying data has moved on from. Caching is a correctness problem wearing a performance costume.

First, do not confuse the two caches

There are two different mechanisms both called “caching” in LLM systems, and conflating them causes real design errors.

This article is about the second kind. Provider prompt caching is complementary: even on a cache miss, a well-structured prompt with a stable prefix costs less to run. Design for both, but never let the existence of one convince you the other is handled.

Step 1: Build a key that means “same request”

A cache key must include every input that could change the output. Miss one and you serve the wrong answer with total confidence. The list is longer than most first implementations assume:

• The model identifier, including version. A silent model upgrade behind the same name is still a different answerer.
• The full message list: system prompt, history, and user input, in order.
• Generation parameters that affect output: temperature, tool definitions, response format, and anything else you set.
• A prompt version string you control, bumped whenever the prompt text or the code that builds it changes.
• The tenant or user scope, so answers can never cross accounts.
• A data version for retrieval-grounded features: the index build, the document set revision, or whatever token changes when the underlying facts change.

Canonicalise before you hash. Serialise to JSON with sorted keys and fixed separators, or two semantically identical requests with differently ordered fields will hash differently and your hit rate will quietly halve.

import hashlib
import json
 
 
def cache_key(*, model, messages, params, prompt_version,
              tenant_id, data_version) -> str:
    payload = {
        "model": model,
        "messages": messages,
        "params": params,
        "prompt_version": prompt_version,
        "tenant": tenant_id,
        "data": data_version,
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf8")).hexdigest()

Step 2: Store, expire, and stampede-proof

Wrap the model call so the cache is invisible to the rest of your code, store the response with the metadata you will want when debugging (“why did it say that?” should be answerable from the cache entry), and set a time-to-live based on how fast the underlying truth changes, not on how much you want hits.

import json
import time
 
 
def cached_complete(store, key, ttl_seconds, generate):
    entry = store.get(key)
    if entry is not None:
        record = json.loads(entry)
        if time.time() - record["stored_at"] < ttl_seconds:
            return record["response"]
    response = generate()
    store.set(key, json.dumps({
        "response": response,
        "stored_at": time.time(),
    }))
    return response

Two production refinements separate this sketch from a system you can trust. First, request coalescing: when a popular key expires, many concurrent requests miss at once and all call the model. A short-lived lock per key, so one request regenerates while others wait or serve the slightly stale entry, turns a thundering herd into one call. Second, negative results are results: if the model call fails or the model refuses, decide explicitly whether that outcome is cacheable. Caching a transient failure makes an outage sticky; failing to cache a permanent refusal makes you pay for the same “no” forever. Store them under different TTLs, on purpose.

Set TTLs from the data, not the request. A summary of a document should live exactly as long as the document is unchanged, which is why data_version belongs in the key: when the document changes, the version changes, and the old entry becomes unreachable garbage to be expired, not a wrong answer waiting to be served. TTL is your second line of defence for everything you forgot to version.

The semantic caching temptation

Semantic caching stores embeddings of requests and serves a stored response when a new request is similar enough. It promises hits where exact matching finds none, and for genuinely closed workloads, classifying tickets into fixed categories, answering from a fixed FAQ set, it can work. Everywhere else, it has a failure mode that should scare you: similar is not equivalent.

Questions that differ by one word are frequently opposite in meaning. Negation, dates, names, quantities, and “not” are all small edits with total semantic consequences, and embedding similarity is at its least reliable exactly on those edits. A threshold strict enough to be safe yields few extra hits; a threshold loose enough to pay yields confident wrong answers. There is no setting that gives you both, because the trade-off is the mechanism.

If you use semantic caching at all, fence it:

• Only for intents you have enumerated, never for open-ended generation.
• Only within a tenant, and never across permission boundaries: similarity does not know who was allowed to see the original answer.
• With a similarity threshold validated against a labelled set of question pairs that look alike and mean differently. If you have not built that set, you have not validated the cache.
• With logging that records both the served response and the cached request it matched, so a wrong answer can be traced to its match.

Streaming, tools, and other cache poisoners

Three features complicate the picture. Streaming responses should be cached only once complete: store the assembled final text, never a partial stream, and replay it as a single response or a re-chunked stream on a hit. Tool calls make a response an action, not an answer; never serve a cached tool call, because the world it acted on has changed since it was computed. Cache the final user-facing message of a completed exchange if you must, and even then only for read-only flows. Personalised outputs, anything containing a name, a balance, a user’s own data, must be keyed by tenant at minimum, and the honest default is not to cache them at all.

Measure the cache like a system, not a hope

A cache you cannot observe will betray you quietly, so log every decision: hit or miss, the key’s version components (model, prompt version, data version, never the raw user text), the age of the entry served, and the reason for any bypass. Three signals deserve dashboards. A hit rate that suddenly climbs after a prompt change usually means your prompt version did not change with it, and you are serving the old prompt’s answers at scale. Entries served close to their TTL boundary, repeatedly, for the same keys suggest the TTL is shorter than the natural repeat interval and you are paying full price for near-hits. And any hit served across a data version boundary is a bug by definition: it should be impossible by construction, so if it appears, your key is missing a component and the log is how you find out which one.

When not to cache

Keep an explicit do-not-cache list in code, next to the cache itself: requests whose answers depend on data fresher than your TTL, requests from flows with side effects, requests involving sensitive personal data you do not want replicated into another store, and any model call whose output a human is actively iterating on (your prompt-engineering loop will lie to you if yesterday’s prompt answers today’s tests). Every team that has been bitten by an LLM cache was bitten in one of these four places.

Verdict

Cache LLM responses at the level where requests genuinely repeat, key on everything that can change the answer (model, messages, parameters, prompt version, tenant, data version), expire by data freshness rather than hope, and coalesce concurrent misses. Treat semantic caching as a specialised instrument for closed, validated workloads, never a default. Do that and the cache is what it should be: a boring, auditable cost saver. Skip the key discipline and it becomes something else: a second, unmonitored model, trained on your history, answering your users with the past.