Skip to content

Day 36 of 100

The Region That Went Down

Disaster RecoveryCloudAdvanced20–25 min

🔥 Problem Statement

The cloud region hosting production becomes unavailable — not a single service failing, the whole region. Backups exist in a second region, which sounds reassuring right up until someone asks the obvious next question: what's the actual procedure for using them to bring the service back up, and how long is that supposed to take?

Nobody has a clear answer. There's no documented runbook, and this has never been tested end to end.

🏗️ Environment

  • Cloud: single primary region, backups exist in a second region
  • Recovery procedure: no documented, tested runbook
  • RTO/RPO: never formally defined for this service
  • Event: primary region becomes unavailable

🤔 Your Challenge

Having backups is clearly not the same as having a disaster recovery plan — what's actually missing here?

  • What do RTO and RPO actually define, and why does neither existing for this service matter right now?
  • If backups exist but recovery has never been tested, what confidence should anyone have that it would actually work?
  • Beyond data, what else does a service depend on that also needs to exist in the recovery region?
  • What would 'testing DR' actually mean, concretely, versus just confirming backups are being taken?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

Backups answer "do we have the data somewhere else." Disaster recovery answers a much bigger question: "can we actually restore full service, within an acceptable amount of time and with an acceptable amount of data loss, using a documented, rehearsed procedure." Having the first without ever having validated the second means, in a real regional outage, the actual recovery time and the actual data loss are both unknown — potentially very different from whatever anyone might optimistically assume.

Step 2 — Identify the likely bottleneck

RTO (Recovery Time Objective — how long recovery is allowed to take) and RPO (Recovery Point Objective — how much data loss, measured in time, is acceptable) never being formally defined for this service means there's no target to test against, and no way to know whether "backups exist in another region" actually meets whatever the business actually needs. Beyond data, a real regional failover also depends on DNS being repointed, dependent services and their own region-availability, configuration and secrets existing in the recovery region, and often manual coordination steps — none of which a backup alone addresses.

Step 3 — Investigation

Given the region is down right now, the immediate work is recovery, not analysis — but once service is restored, or in parallel if enough people are available:

  • Attempt the recovery using whatever exists today, treating it as the first real (if painful) test of the process, and documenting every manual step and every surprise along the way.
  • Identify every dependency required for a working service in the recovery region — not just the primary datastore, but downstream services, DNS, secrets, configuration, and anything else discovered to be missing or region-specific during the actual recovery attempt.
  • Measure actual time to restore, end to end, as it happens — this becomes the honest baseline RTO, however uncomfortable, against which a real target can later be set and worked toward.
  • Measure actual data loss between the last successful backup/replication point and the outage — this becomes the honest baseline RPO.

Once service is restored, formally define RTO and RPO for this service based on actual business requirements — not what current tooling happens to support, but what's actually acceptable, and then work backward to what infrastructure and process is needed to hit those numbers. Write a documented, step-by-step recovery runbook based on what was actually learned during this real recovery, filling in every manual step and gap discovered.

Then — this is the part organizations most often skip — schedule and run a regular, planned DR test (a "game day"), executing the documented runbook end to end, ideally without waiting for a real outage to be the first real test. A runbook that's never been executed outside of an actual crisis is still mostly theoretical, regardless of how thorough it looks on paper.

Step 5 — Engineering lesson

"We have backups in another region" feels like disaster recovery readiness, but it only answers whether data exists elsewhere — not whether anyone knows how to use it, how long that takes, or what else besides data is needed to actually serve traffic again. Real DR readiness is demonstrated by a tested runbook with a known RTO/RPO, not implied by the existence of a backup nobody's ever tried to restore from under real conditions.

🧠 Today’s Takeaway

A backup is not a disaster recovery strategy until recovery is tested.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.