Day 30 of 100
The Cache That Made Things Worse
🔥 Problem Statement
A team adds caching to a read-heavy endpoint specifically to improve performance — the intent was straightforward: fewer database hits, faster responses. Instead, two problems show up almost immediately: users occasionally see data that's clearly out of date, and the cache hit rate is lower than anyone expected, meaning it isn't even delivering the performance win it was built for.
🏗️ Environment
- Recently added: in-memory cache layer in front of a read-heavy endpoint
- Cache: fixed TTL, no explicit invalidation on writes
- Multiple application instances, each with its own local cache
- Symptom: stale data reported by users, plus lower-than-expected cache hit rate
🤔 Your Challenge
What would you investigate to understand both the staleness problem and the disappointing hit rate?
- If there's no explicit invalidation on writes, how would a cache ever know data has changed underneath it?
- If each application instance has its own local cache, what does that mean for consistency and for hit rate?
- What's the difference between a cache being wrong and a cache just not being used enough to help?
- How would TTL length affect both staleness risk and hit rate, in opposite directions?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
Stale data and disappointing hit rate are actually two separate problems with two separate causes, even though they showed up in the same rollout. Staleness points at invalidation — or the lack of it. Low hit rate points at something about how requests are distributed relative to where cached data actually lives. Treating them as one problem risks fixing one and leaving the other unexplained.
Step 2 — Identify the likely bottleneck
Staleness: with no explicit invalidation on writes, the cache only ever becomes fresh again when its TTL expires — meaning any write that happens between cache population and TTL expiry is invisible to anyone reading from cache during that window. This isn't a bug in the traditional sense; it's the direct, designed consequence of TTL-only caching, and it's exactly why "add a cache" is a consistency decision, not just a performance one.
Hit rate: each application instance keeping its own local, in-memory cache means the same piece of data can be cached separately, redundantly, on every instance — and a request routed to an instance that hasn't cached this particular key yet gets a cache miss even if nine other instances already have it cached. With enough instances and even moderately distributed traffic, this fragmentation can keep the effective hit rate well below what a single shared cache would achieve.
Step 3 — Investigation
- Confirm the staleness window by tracing a specific write-then-read sequence — how long after a write does a read still return the old value? This should roughly match the configured TTL, confirming the mechanism.
- Check hit rate per instance versus in aggregate — if per-instance hit rates are reasonable but the effective experience is poor, fragmentation across instances is likely the cause.
- Check how traffic is distributed across instances for the specific keys being cached — if load balancing spreads requests for the same data across many instances, each with its own cold cache for that key, low hit rate is close to guaranteed.
- Check the TTL value itself against how frequently the underlying data actually changes — a TTL much longer than the data's real change frequency increases staleness risk; one much shorter than necessary hurts hit rate for no consistency benefit.
Step 4 — Recommended action
For staleness: add explicit invalidation on writes — when the underlying data changes, actively evict or update the corresponding cache entry rather than waiting for TTL to expire. This directly closes the staleness window instead of just shortening it.
For hit rate: move to a shared cache (e.g., Redis or a similar external cache) instead of per-instance local caches — every instance reads and writes the same cache, so a value cached by one request benefits every subsequent request regardless of which instance handles it. This also makes invalidation simpler, since there's one cache to update instead of potentially many independent ones.
Choose TTL based on how tolerable staleness actually is for this specific data, not an arbitrary default — some data can be minutes stale without consequence; other data can't be seconds stale without causing real problems, and the TTL should reflect that difference explicitly.
Step 5 — Engineering lesson
A cache doesn't just trade correctness for speed by default — it trades a specific, bounded amount of correctness (however stale the TTL allows) for speed, and that trade needs to be a deliberate decision, not an accident of adding a cache without thinking through invalidation. And a cache's benefit depends entirely on how it's shared across a system's actual topology — one cache per instance in a multi-instance architecture is a fundamentally different (and usually worse) design than one shared cache serving all of them.
🧠 Today’s Takeaway
Caching is a consistency decision as much as a performance decision.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

