Caching LLM responses correctly

Why bother caching LLM responses? (TL;DR)

To cache LLM responses correctly means storing the output of a Large Language Model for a given input query, then retrieving that stored response instead of hitting the API again when the same (or a very similar) query comes in. This isn't rocket science; it's a fundamental optimization strategy that directly translates into three core benefits: significantly lower API costs, reduced latency for your users, and increased reliability by sidestepping potential API rate limits or outages.

Think about it. Every time your application sends a prompt to an LLM provider like OpenAI, Anthropic, or Cohere, it costs money – typically per token. If 10,000 users ask 'What is the capital of France?' in a day, sending that same prompt 10,000 times is burning through your budget unnecessarily. Caching transforms those 10,000 calls into one actual API call and 9,999 lightning-fast, free lookups. At SISL, we've seen clients slash their monthly API bills by 30-70% with a well-implemented caching layer, especially for applications with high query volume on relatively static content.

Beyond the wallet, speed matters. An API call to an LLM, even with a low latency provider, still takes hundreds of milliseconds, often more. A cached response, pulled from a local Redis instance or an edge cache, can be served in single-digit milliseconds. This isn't just a marginal improvement; it's the difference between a sluggish user experience and one that feels instantaneous. For interactive applications, this distinction can be critical for user retention and satisfaction.

Not all responses are created equal: What to cache, what to skip?

Not every LLM interaction is a candidate for caching. Blindly caching everything is a recipe for stale data, security headaches, or simply wasted effort. Smart caching requires discernment.

The Good Candidates for Caching

• Static Informational Queries: Questions whose answers are unlikely to change over time. Examples: 'Explain the theory of relativity,' 'Summarize the plot of Hamlet,' 'List the benefits of regular exercise.'
• Commonly Asked Questions (FAQs): If your chatbot frequently answers 'How do I reset my password?' or 'What are your business hours?', these are prime caching targets.
• Summaries of Unchanging Documents: If you're using an LLM to summarize product manuals, legal documents, or archived news articles, and the source material doesn't change, cache the summary.
• Content Generation for Static Pages: Generating SEO meta descriptions for product pages where the product details rarely change? Cache them. Generating blog post outlines for evergreen topics? Cache those too.
• Aggregated Data Insights: If you're asking an LLM to extract general trends or insights from a stable dataset, and the insights themselves don't require real-time updates, cache the aggregated response.

The Risky Business: When to avoid caching

• Highly Personalized Content: Any response tailored specifically to an individual user's real-time context, preferences, or sensitive data should generally bypass the cache. Examples: 'Draft an email for John about his recent purchase of product X,' 'Suggest next steps based on my current portfolio.'
• Real-time Data Dependent Responses: If the LLM's response relies on rapidly changing data, caching it will lead to stale information. Examples: Current stock prices, live weather updates, real-time inventory levels.
• Sensitive User Inputs: Caching sensitive personal identifiable information (PII), financial data, or confidential business details creates a new security surface. Even if anonymized, the principle of least privilege often dictates against caching such data unless absolutely necessary and with robust security measures.
• Responses Requiring Absolute Freshness: Legal advice, medical diagnostics, or financial recommendations are domains where even a slightly outdated cached response could have severe consequences. Prioritize accuracy and freshness over caching in these critical areas.
• Unique or Infrequent Queries: If a prompt is truly one-off or rarely repeated, the overhead of caching (storage, invalidation logic) might outweigh the benefits.

The Nuts and Bolts: How to actually cache LLM responses

Once you've identified what to cache, the 'how' becomes critical. There are various strategies, each with its own trade-offs in complexity, cost, and effectiveness.

Simple Key-Value Caching: The Prompt Hash

The most straightforward approach is to use the prompt itself (or a hash of it) as the cache key. When an LLM request comes in:

1. Hash the incoming prompt (e.g., using MD5 or SHA256).
2. Check if this hash exists as a key in your cache.
3. If it exists, return the associated stored LLM response.
4. If not, send the prompt to the LLM, receive the response, store it in the cache with the hash as the key, and then return it to the user.

This method is excellent for exact prompt matches. Tools like Redis, Memcached, or even a simple database table (PostgreSQL, MySQL) can serve as your cache store. Redis is often preferred for its speed and in-memory nature, making it ideal for high-throughput caching. A basic setup might look like this:

import hashlib
import json
import redis

# Assuming `llm_api_call` is your function to hit the LLM API

def get_llm_response_with_cache(prompt: str, ttl: int = 3600):
    r = redis.Redis(host='localhost', port=6379, db=0)
    prompt_hash = hashlib.sha256(prompt.encode('utf-8')).hexdigest()
    
    cached_response = r.get(prompt_hash)
    if cached_response:
        print("Cache hit!")
        return json.loads(cached_response)
    
    print("Cache miss. Calling LLM...")
    llm_response = llm_api_call(prompt) # Replace with actual LLM call
    
    r.setex(prompt_hash, ttl, json.dumps(llm_response))
    return llm_response

Semantic Caching: Beyond Exact Matches

What if your users rephrase the same question? 'What is the capital of France?' and 'Which city is France's capital?' should ideally return the same cached response. This is where semantic caching shines. Instead of hashing the raw prompt, you convert the prompt into a vector embedding (a numerical representation of its meaning).

When a new prompt arrives:

1. Generate a vector embedding for the incoming prompt.
2. Perform a similarity search in your cache for existing prompt embeddings that are semantically close (e.g., within a certain cosine similarity threshold).
3. If a sufficiently similar embedding is found, return its associated LLM response.
4. If not, send the prompt to the LLM, generate an embedding for the new prompt, store both the new embedding and its response, then return the response.

This approach requires more sophisticated infrastructure, typically a vector database like Pinecone, Weaviate, Milvus, or even PostgreSQL with the `pgvector` extension. While more complex, semantic caching dramatically increases your cache hit rate for natural language applications, leading to even greater savings and speedups.

Invalidation Strategies: Keeping it Fresh (Enough)

A cache is only as good as its freshness. Stale data is often worse than no data. Effective invalidation strategies are crucial:

• Time-to-Live (TTL): The simplest and most common method. Each cached item is given an expiry time (e.g., 1 hour, 24 hours, 7 days). After this time, the item is automatically removed from the cache or marked as stale. This is perfect for data that doesn't need to be absolutely real-time.
• Content-based Invalidation: If your LLM responses are derived from specific source data (e.g., summarizing a document), you can invalidate the cache entry whenever that source document changes. This requires a more direct link between your source data management and your caching layer.
• Manual Invalidation: For critical updates or known errors, a manual trigger to clear specific cache entries or even the entire cache might be necessary. This is a blunt instrument but effective in emergencies.

Caching Layers: Where to put it?

Your caching layer can live in different places, depending on your application's architecture and scale:

• Application-level Cache: In-memory caches (like Python's functools.lru_cache or simple dictionaries) are fast but limited to a single application instance and disappear on restart. A local file-based cache offers persistence but is slower.
• Dedicated Caching Service: For distributed applications and serious scale, a dedicated, external caching service like Redis is the standard. It's fast, supports various data structures, and can be shared across multiple application instances.
• Edge Caching: Services like Vercel Edge Cache or Cloudflare Workers KV push your cache closer to your users, reducing latency even further by serving responses from geographically distributed data centers. This is particularly effective for global audiences. Integrating with these often requires minimal code changes if your application is already deployed on these platforms. For example, Vercel's stale-while-revalidate header can automatically handle re-fetching content in the background, ensuring users always get a response quickly.

Practical Considerations and Pitfalls

Implementing an LLM caching strategy isn't just about picking a tool; it requires thoughtful consideration of several factors.

Cost vs. Benefit

Caching isn't free. There's the cost of infrastructure (Redis server, vector database), development time for implementation, and ongoing maintenance. For a small application with minimal LLM usage, the ROI might not be there. However, for applications making hundreds or thousands of LLM calls daily, the savings can quickly dwarf the caching costs. For instance, reducing your OpenAI bill by a few hundred dollars monthly easily justifies a modest Redis instance that costs $10-50/month.

Cache Invalidation Hell

This is famously one of the two hardest problems in computer science (alongside naming things and off-by-one errors). Getting your invalidation strategy wrong means serving stale data or constantly re-fetching, negating the benefits. Start simple with TTLs and only add content-based or manual invalidation where absolutely necessary.

Data Security & Privacy

If you're caching LLM responses, you're storing data. This means you need to consider where that data lives, who has access to it, and how long it's retained. For sensitive information, caching might introduce compliance challenges (GDPR, CCPA) that outweigh the performance benefits. Ensure your caching solution aligns with your data governance policies.

Monitoring

You can't optimize what you don't measure. Monitor your cache hit rate – the percentage of requests served from the cache versus those that hit the LLM API. Track latency improvements for cached versus uncached requests. Tools like Sentry or PostHog can help monitor overall application performance, but custom metrics for your caching layer are essential to understand its effectiveness. A low hit rate might indicate an ineffective caching strategy (e.g., too short a TTL, or insufficient semantic matching).

Complexity

Semantic caching, while powerful, adds significant complexity due to vector embeddings, similarity search algorithms, and specialized databases. As a boutique studio, SISL often recommends starting with a robust key-value cache and scaling up only when semantic nuances genuinely justify the added complexity and cost. Don't over-engineer from day one.

API Provider Terms of Service

Always review the terms of service of your LLM provider. Some may have specific rules regarding caching their responses, especially concerning data retention and usage. Ensure your caching strategy is compliant.

The Bottom Line: Cache Smart, Not Hard

Caching LLM responses isn't a silver bullet, but it's an indispensable tool for anyone building production-ready applications that rely heavily on large language models. It's a pragmatic approach to managing costs, enhancing user experience, and building more resilient systems.

By thoughtfully deciding what to cache, implementing appropriate strategies (from simple hashes to advanced semantic matching), and maintaining a keen eye on invalidation and security, you can turn a potentially expensive and sluggish LLM integration into a lean, fast, and cost-effective component of your application. If your LLM bills are soaring or your responses are sluggish, it's time to consider a caching strategy. Not sure where to start? We've built these systems before, get in touch.