The Alert Storm

Step 1 — Understand the symptoms

Two hundred alerts across forty services in four minutes is a pattern, not a coincidence. Independent failures don't usually line up that tightly in time — services failing for forty unrelated reasons tend to trickle in over minutes to hours, each on its own schedule. A near-simultaneous burst across a wide swath of the architecture is the signature of something shared breaking underneath all of them.

Step 2 — Identify the likely bottleneck

The question worth asking isn't "which forty services are broken," it's "what do these forty services have in common." If they share a dependency — a database, a caching layer, an internal auth service, a shared network path — the far more likely story is: that shared dependency degraded or failed, and every service downstream of it threw its own error-rate and latency alerts as a direct symptom, at roughly the same moment, because they were all affected by the same upstream event simultaneously.

That reframes 200 alerts from "200 problems" to "one problem, with 200 symptoms." The alerting system did its job — it's just alerting on symptoms (this service's error rate) rather than root cause (the shared dependency is down), and without a dependency map in hand, that distinction is easy to miss under 2 a.m. pressure.

Step 3 — Investigation

• Build or pull up the service dependency graph — even a rough one. Which of the alerting services share a common upstream dependency?
• Look at alert timestamps precisely, not just "they all fired around 2 a.m." A shared-cause failure tends to show a tight clustering — seconds apart, not spread across minutes — while independent failures cluster far more loosely.
• Check the shared dependency's own health directly — its own metrics, not the forty services reacting to it. If it's a database, check its connections and query latency. If it's an internal auth or config service, check its own error rate and availability.
• Sample a few traces from the affected services and see where they spend their time — if they all show elevated latency at the same specific downstream call, that call is the actual failure point.

Step 4 — Recommended action

Don't split the incident into forty separate investigations. Assign one person (or a small group) to trace the shared dependency, and treat the forty service-level alerts as confirmed symptoms rather than forty separate mysteries to solve independently — that's both faster and avoids forty people converging on the same root cause from forty different angles.

Once the shared dependency is identified and restored, most of the downstream alerts should clear on their own without individual intervention — which is itself useful confirmation that the root-cause theory was correct, versus forty coincidentally unrelated failures that would need forty separate fixes.

Step 5 — Engineering lesson

Alert volume measures how visible a failure is, not how many failures there are. A well-connected architecture means a single upstream failure can legitimately look like a mass outage in the alert list, and the fix for that isn't fewer alerts — it's correlation. SLO-based alerting on the services that matter most, paired with a dependency map that's actually kept up to date, turns "200 alerts, panic" into "one root cause, forty confirmed symptoms" in the first five minutes instead of the first hour.