Skip to content

Day 39 of 100

The Error Budget Is Gone

SRESLOAdvanced20–25 min

🔥 Problem Statement

The service's error budget — the amount of unreliability its SLO allows before something has to change — is gone for this period. Fully spent. At the same time, the product team has a release ready to go, on the schedule they'd planned, with no particular awareness that the error budget situation should factor into that timeline at all.

Nobody's dealt with this exact situation before. There's no established precedent for what happens next.

🏗️ Environment

  • SLO: defined, with an associated error budget policy
  • Error budget: fully exhausted for the current period
  • Product team: has a release ready, wants to ship on schedule
  • No prior precedent for how this situation was handled

🤔 Your Challenge

What is an error budget actually for, and what should happen when it runs out?

  • If an error budget exists specifically to bound acceptable unreliability, what does exhausting it imply about further risk this period?
  • Is 'stop all releases' the only possible response, or are there other reasonable policies?
  • Who should be part of deciding how to respond — just SRE, just product, or both together?
  • What would make this decision repeatable next time, instead of being improvised again?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

An error budget exists precisely to make this moment a structured decision rather than an argument — it's meant to answer, in advance, "how much unreliability are we willing to accept before we deliberately slow down and prioritize stability." Reaching zero means that threshold has been crossed for this period, and it's a signal the system is asking for: either reduce risk (slow down releases, prioritize reliability work) or make an explicit, informed decision to accept more risk anyway, with eyes open.

Step 2 — Identify the likely bottleneck

The actual problem here isn't the exhausted budget itself — that's a known, expected possibility the moment a budget policy exists at all. It's that there's no established policy for what happens when it's reached. Without one, this becomes an ad hoc negotiation between SRE and product, potentially adversarial, decided under the immediate pressure of a specific release rather than as a calm, previously-agreed policy being followed.

Step 3 — Investigation

  • Understand why the budget was exhausted — a single major incident, or a steady accumulation of smaller issues over the period? This affects what the appropriate response actually is, not just that a response is needed.
  • Check whether an error budget policy exists anywhere, even informally, that was never fully operationalized or agreed to by both sides — sometimes the gap is enforcement, not the absence of any policy at all.
  • Assess the specific release's risk — is this a low-risk, well-tested change, or something with meaningfully higher potential to affect reliability further? Not every release carries equal risk against an already-spent budget.
  • Check historical incident data for this service to understand whether budget exhaustion has been a one-off or a recurring pattern — a recurring pattern suggests a deeper reliability investment is needed, not just a one-time policy decision.

If no error budget policy exists, this is the moment to create one — collaboratively, between SRE and product, before the next time this happens rather than improvised again under pressure. A common, reasonable policy: when the budget is exhausted, non-critical releases pause and the team's priority shifts to reliability work until the budget recovers; critical fixes or low-risk changes may still proceed with explicit review. The specific policy matters less than that it's agreed upon in advance and applied consistently, rather than negotiated fresh under the pressure of each individual release.

For the specific release in question, make the call using whatever policy (existing or newly agreed) applies — but make it a joint, explicit decision between SRE and product, not one side unilaterally overruling the other, since both have a legitimate stake in the tradeoff being made.

Step 5 — Engineering lesson

An error budget is only useful as a pre-agreed decision framework — if what happens at zero is improvised each time, it's not actually functioning as a framework, it's just a number people point to during arguments. The value of the whole error-budget approach is turning "should we prioritize speed or stability right now" from a recurring, contentious debate into a policy decided calmly in advance, applied consistently when the moment actually arrives.

🧠 Today’s Takeaway

Error budgets turn reliability into an engineering decision framework.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.