Day 22 of 100
The Broken Health Check
🔥 Problem Statement
Instances keep getting pulled out of the load balancer's rotation. Every time it happens, someone checks manually — hits the application directly, confirms it responds correctly, confirms it's fast, confirms nothing is actually wrong. And yet, minutes later, the load balancer marks it unhealthy again and removes it, only for it to get added back once it "recovers," in a cycle that's happening several times an hour.
The application itself has never actually been broken during any of these episodes.
🏗️ Environment
- Platform: managed load balancer in front of multiple instances
- Health check: HTTP endpoint, custom timeout and interval
- Application: confirmed healthy via manual testing
- Symptom: instances repeatedly marked unhealthy and removed from rotation
🤔 Your Challenge
If the application is genuinely healthy every time it's checked manually, where does the actual problem live?
- What's the difference between the application being unhealthy and the health check itself being misconfigured?
- What would you check about the health-check endpoint's timeout versus the application's actual response time under load?
- Could the health-check endpoint be doing something different — and slower — than a normal request?
- What would repeatedly-failing-then-recovering instances look like if the check itself is marginal rather than the app?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
The application checking out fine on manual inspection, every single time, while the automated health check keeps failing it, points directly at the check itself rather than the application. If the application were genuinely intermittently unhealthy, manual checks would sometimes catch it in a bad state too — the fact that they never do, while the automated check fails repeatedly, is the strongest signal available that these are two different things being measured, and only one of them is actually broken.
Step 2 — Identify the likely bottleneck
A few common misconfigurations produce exactly this pattern: a health-check timeout set too aggressively short relative to how long the endpoint actually takes to respond under real (even normal) load — so it's marked unhealthy not because it's broken, but because it didn't answer fast enough by an arbitrary margin. Or the health-check endpoint itself does something heavier than a normal request — a deep dependency check, a database query, more work than the actual traffic the service handles — making it slower and less reliable than the application's real request path. Or the check interval and failure threshold are tuned so aggressively that a single slow response (not a real failure) is enough to pull an instance out.
Step 3 — Investigation
- Compare the health-check endpoint's actual response time against its configured timeout, under normal and slightly elevated load — if the margin is thin, occasional normal variance is enough to trip it.
- Check what the health-check endpoint actually does — is it a lightweight liveness check, or does it exercise dependencies (database, cache, downstream services) that add latency and failure modes the rest of the application doesn't have on a typical request?
- Check the failure threshold configuration — how many consecutive failed checks does it take to mark an instance unhealthy? A threshold of one is far more sensitive to normal jitter than a threshold of three.
- Correlate unhealthy markings against actual instance load — if instances get marked unhealthy specifically during moments of elevated (but still normal) traffic, that's a strong sign the check's timeout margin is simply too tight for real-world variance.
- Check whether the health-check path shares resources (thread pool, connection pool) with normal traffic — if it's competing for the same limited resources, a burst of real traffic could legitimately delay the health check without the application being unhealthy in any meaningful sense.
Step 4 — Recommended action
Widen the timeout to a realistic margin above actual observed response time, not an arbitrarily tight number. If the health-check endpoint does more work than it needs to, simplify it — a liveness check should generally answer "is this process able to respond," not "are all its downstream dependencies also healthy," which is a different, heavier question better suited to a separate readiness check with its own appropriate thresholds. Raise the failure threshold from a single failed check to a small consecutive count, so one slow response under normal variance doesn't trigger removal.
Step 5 — Engineering lesson
A health check is itself a piece of software with its own behavior, timing, and failure modes — it's easy to treat it as a neutral, ground-truth signal about the application, but a poorly tuned check can be wrong in a very convincing way: an instance repeatedly cycling in and out of rotation looks exactly like an unstable application, right up until someone checks manually and finds nothing wrong every single time.
🧠 Today’s Takeaway
A bad health check can make a healthy application look broken.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

