Skip to content

Day 40 of 100

The Alert With No Owner

SREOperationsAdvanced20–25 min

🔥 Problem Statement

An important production alert fires — and it's not a false positive, it's correctly detecting a real condition worth knowing about. The on-call engineer who gets paged has no idea what to do with it. There's no runbook. There's no clearly documented owner. The alert is technically correct and functionally useless in the moment it actually matters, because nobody knows what response it's supposed to trigger.

🏗️ Environment

  • Alert: fires on a specific threshold breach, correctly detects a real condition
  • Ownership: unclear which team is responsible for responding
  • Runbook: does not exist for this alert
  • On-call engineer: unsure what action, if any, to take

🤔 Your Challenge

What does an alert actually need, beyond correctly detecting a real condition, to be genuinely useful in production?

  • If an alert fires correctly but the responder doesn't know what to do, has it actually done its job?
  • What's the difference between an alert that's technically accurate and one that's operationally actionable?
  • Who should be able to look at an alert and immediately know who's responsible for it?
  • What would you want documented before this alert fires again?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

Detection and response are two different capabilities, and an alert only delivers value when both exist together. This alert nails detection — it correctly identified a real condition — but provides nothing for response: no indication of who should act, no guidance on what action to take, no context about why this condition matters or what its consequences are if left unaddressed. A technically accurate alert that leaves its responder stuck is, in practical terms, close to as unhelpful as no alert at all, just with extra anxiety attached.

Step 2 — Identify the likely bottleneck

This usually happens when an alert gets created to solve a narrow, immediate problem — "we need to know when X happens" — without the broader operational context being built in at the same time: who owns this, what the response actually is, why it matters, and where to look for more information. The alert ships technically complete (it fires correctly) but operationally incomplete (nobody can act on it), and that gap often stays invisible until the first time someone's actually paged by it under real conditions.

Step 3 — Investigation

Rather than investigating a root cause, this calls for defining what's actually missing:

  • Determine who is actually responsible for this condition — which team owns the system or component this alert is about, and does that team know this alert exists and pages toward them (or should)?
  • Determine what action is actually appropriate when this alert fires — is there a clear runbook step, or does responding require genuine investigation each time? If the latter, that's worth knowing explicitly rather than discovering under pressure.
  • Determine the actual severity and urgency — does this condition need immediate action, or can it reasonably wait until business hours? An alert with no severity guidance forces every responder to guess.
  • Check whether other alerts in this system have the same gap — an unowned, unrunbooked alert is rarely a unique case; it's often a symptom of how alerts get added generally.

Assign explicit ownership to this alert, tied to routing (it should page the team actually responsible, not whoever happens to be on the general on-call rotation if that's not the right team). Write a runbook — even a short one — covering what this alert means, why it matters, and what action to take, linked directly from the alert itself so a responder doesn't have to go hunting for context while a real condition is active.

Use this as the occasion to establish (or reinforce) a standard for what every production alert requires before it ships: an owner, a severity level, and either a runbook or a clear statement that investigation is genuinely required each time. Treating this as a template to apply going forward prevents the next unowned, unrunbooked alert from being created the same way this one was.

Step 5 — Engineering lesson

An alert's job isn't just to notice something — it's to get the right response from the right person, quickly. A correct detection paired with no ownership and no guidance puts all of the actual work of figuring out "who, what, why" onto whoever happens to be paged, at the worst possible moment to be figuring that out for the first time. Ownership and a runbook aren't optional extras on top of a good alert — they're part of what makes it an alert instead of just noise with good detection logic behind it.

🧠 Today’s Takeaway

An alert without ownership is just noise.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.