LRU throws away your most expensive knowledge
Least-recently-used eviction is semantically blind. It happily discards the rare, costly answer you'll pay dearly to regenerate, and keeps the cheap trivia everyone asks. There's a smarter way.
Every cache eventually fills up and has to throw something away. The default rule, evict whatever was used least recently, sounds reasonable until you notice what it optimizes for. It keeps the cheap, popular questions and discards the rare, expensive ones. But the expensive answer is exactly the one you don't want to pay the model to regenerate.
Recency is one input, not the whole rule, cost and uniqueness matter more.
The effect is a cache that gets smarter as it fills, not dumber. It protects the long-tail domain knowledge that's painful to recompute and lets the cheap, easily-regenerated trivia churn. You keep the answers that would hurt to lose.
The bottom line
Eviction is a value judgment, so make it about value. Keep the expensive, rare knowledge; let the cheap stuff go. Blind LRU has it exactly backwards.