The hidden cost of RAG nobody puts on the slide
Retrieval-augmented generation stuffs context into every prompt to make answers better. It also makes every repeated question dramatically more expensive, and repeated questions are most of them.
RAG earns its popularity, grounding answers in your own documents genuinely helps. But there's a line item nobody puts on the architecture slide: every RAG call is a big call. You're not sending a short question; you're sending the question plus pages of retrieved context, every time. And when users ask the same thing in different words, you resend all that context and pay for it again.
In RAG, each repeated question is an expensive call, the context rides along every time.
The savings are larger than for plain chat, precisely because each avoided call is bigger. You're not just skipping a model call; you're skipping the retrieval, the reranking, and the long prompt that came with it. For a knowledge assistant where the same questions dominate, caching the answers is often the single biggest lever on the bill.
The bottom line
RAG makes every answer richer and every repeat pricier. Cache the answers by meaning, and the expensive part only runs for questions you've genuinely never seen.