Prompt caching vs semantic caching: what the provider feature doesn't cover
OpenAI and Anthropic added prompt caching, and it's genuinely useful. But it only discounts the prefix you repeat verbatim. The moment the wording changes, you pay full price again.
Provider-side prompt caching is a real win and you should use it, repeating a long system prompt gets cheaper. But read the fine print: it discounts the exact prefix you send byte-for-byte. Change a word in the user's question and the discount evaporates, because the provider is matching tokens, not meaning.
Prompt caching can't see that these three are the same question. A semantic cache can.
The two stack, and they solve different halves of the bill. Prompt caching trims the cost of the context you resend. Semantic caching removes the call entirely when the question, however it's phrased, has already been answered. One discounts the input; the other deletes the request. For repetitive workloads, deleting the request is where the real money is.
The bottom line
Turn on your provider's prompt caching. Then put a semantic cache in front of it, so the rephrased questions never reach the provider at all. Belt and suspenders, and the suspenders save more.