CSTALE: serve the slightly-old answer now, refresh it behind the scenes
A hard TTL turns a one-second-expired answer into a full model call. CSTALE serves the cached answer past its TTL with a stale flag, so you choose freshness versus latency per request.
Hard expiry is a blunt instrument: the moment an entry's TTL passes, the next request eats a full model call, even if the answer is one second stale and perfectly usable. CSTALE adds the stale-while-revalidate pattern that CDNs have used for years, it returns the cached answer even past its TTL, marked with a stale flag so the caller knows.
That flag is the control surface. For a dashboard number or a chat reply, a few-seconds-stale answer served instantly is the right call, and you ignore the flag. For a price or a balance, you treat the flag as a signal to kick off a refresh, serve the stale value for snappiness while the fresh one computes, or block for the fresh one when correctness outranks speed. The policy is yours, per request.
- 1incoming query
- 2intent classifier
- 3template match
- 4HNSW neighbours
- 5confidence gate
- 6trust + freshness
- 7answer · <1ms
- 8(nil) → your model
Reuse only when meaning, structure, confidence, and trust all agree.
The win is that expiry stops being a latency cliff. Instead of every TTL boundary becoming a synchronous model call for some unlucky user, the stale answer covers the gap while the cache refreshes, and the p99 that a hard TTL would spike stays flat.
The bottom line
Freshness and latency are a dial, not a switch. CSTALE hands you the dial, most answers can tolerate a moment of staleness in exchange for never paying the cold-miss tax, and the ones that can't are exactly the ones the flag lets you treat differently.