Redis’s own documentation puts a number on semantic caching: teams running FAQ bots, helpdesks, and internal assistants can cut LLM token spend by 30 percent or more without a measurable drop in answer quality. That claim is worth testing before believing it, because the failure mode it hides is not a slow response. It is a wrong one, served instantly and with total confidence.
What is semantic caching, and does the cost claim hold up?
An ordinary cache matches keys byte for byte, which is nearly useless against natural language: “what’s your return policy” and “what is your return policy?” are different strings and the same question. Semantic caching embeds every incoming prompt, compares it against stored embeddings by vector distance, and returns the cached response above a similarity threshold instead of calling the model again. GPTCache, the most widely used open-source implementation, describes cutting LLM query latency by roughly 100x once the cache is warm, because a vector lookup beats a generation call by orders of magnitude. RedisVL’s SemanticCache ships with a default cosine distance_threshold of 0.1, tunable per deployment.
1from redisvl.extensions.cache.llm import SemanticCache 2 3cache = SemanticCache( 4 name="support_bot_cache", 5 redis_url="redis://localhost:6379", 6 distance_threshold=0.08, # stricter than the 0.1 default 7) 8 9hits = cache.check(prompt=user_query, filters={"tenant_id": tenant}) 10if not hits: 11 response = call_llm(user_query) 12 cache.store(user_query, response, filters={"tenant_id": tenant})
What does the 30 percent number leave out?
The counter-argument is not that the savings are fake; it is that the threshold doing the saving is also the threshold doing the damage. “What’s the return policy for electronics” and “what’s the return policy for groceries” sit close together in embedding space, close enough to clear a loosely tuned threshold, and one wrong number gets served with the same confidence as a correct one. Nothing in the response signals that it came from a cache built for a different question. A RAG pipeline’s retrieval stage has the same failure shape, and semantic caches inherit it because they are built on the same embedding-and-distance mechanics.
The second cost sits outside the model entirely. RedisVL’s own documentation demonstrates filtering cache lookups by a user_id field precisely because two different users can ask near-identical questions and must never receive each other’s cached, potentially account-specific, response. Skip that filter on a shared multi-tenant deployment and the cache becomes a data isolation problem, not a cost optimization.
- Semantic caching matches prompts by embedding distance rather than exact text, so paraphrased questions can hit the same cached answer.
- Redis reports token savings of 30 percent or more on repetitive workloads like FAQ bots and helpdesks, without measurable quality loss at a well-tuned threshold.
- GPTCache reports roughly a 100x latency reduction on cache hits, since a vector lookup replaces a full model call.
- A threshold set too loose returns confidently wrong answers instead of merely slow ones, and nothing in the response marks it as a near-miss.
- Multi-tenant deployments need cache lookups filtered by user or tenant ID, or the cache itself becomes a data leak between accounts.
Conclusion
The 30 percent figure is real, but it is a property of the threshold and the filters around it, not of semantic caching as an idea. Start conservative, watch the false-positive rate before trusting the savings number, and scope every lookup to the tenant that owns it. For where this cache typically sits in a larger system, see RAG architecture: a complete guide.
Frequently Asked Questions
What is semantic caching for LLM APIs?
Semantic caching stores past prompts and their model responses as embeddings, then checks new prompts against that store by vector similarity instead of exact text match. When a new prompt lands close enough to a cached one, the stored response is returned directly, skipping a fresh call to the model.
How much can semantic caching reduce LLM costs?
Redis reports that repetitive workloads such as FAQ bots, helpdesks, and internal assistants can see token spend drop by 30 percent or more without a measurable quality regression, at a properly tuned similarity threshold. GPTCache separately reports roughly a 100x latency reduction on cache hits versus a full model call.
What is the biggest risk with semantic caching?
A threshold set too loosely can match two prompts that are close in meaning but expect different correct answers, returning the wrong cached response with full confidence. This differs from a normal cache miss, which is merely slow; a bad semantic hit produces an answer that looks right and is not.
What similarity threshold should I start with?
RedisVL’s SemanticCache defaults to a cosine distance threshold of 0.1, where lower values are stricter. Starting conservative and loosening gradually while monitoring the false-positive rate on real traffic is safer than starting loose and tightening after wrong answers have already been served.
Is semantic caching safe for multi-tenant applications?
Only if cache lookups are filtered by tenant or user ID. Without that filter, two different users asking similar questions can retrieve each other’s cached, potentially account-specific responses. RedisVL supports this directly through filterable metadata fields attached to each cache entry.