Day 37 of 100
The Backup That Couldn't Restore
🔥 Problem Statement
Every night, for months, the backup job has run and reported success — green, every single time, no failures in the logs. It's exactly the kind of thing that builds quiet confidence: backups are happening, so recovery is covered.
Then, during a planned disaster recovery test — the first time anyone has actually tried to restore from one of these backups — the restore fails partway through. The backup that's been reporting success every night turns out to be unusable.
🏗️ Environment
- Database: automated nightly backups, running successfully for months
- Backup job status: green, every night, no failures logged
- Restore: attempted for the first time during a planned DR test
- Symptom: restore fails partway through, backup unusable
🤔 Your Challenge
How can a backup job report success every time while still being unable to actually restore anything?
- What does a backup job's 'success' status actually verify — that a file was created, or that the file is a valid, restorable backup?
- Could a backup be technically created but be corrupted, incomplete, or missing something restore depends on?
- What's the only way to actually know whether a backup would work, short of restoring it?
- Why does this failure mode specifically hide until the worst possible moment — an actual disaster?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
A backup job's "success" status typically confirms that the backup process ran to completion and produced an output file — it does not, by itself, confirm that the output file is complete, uncorrupted, and actually restorable. Those are different claims, verified by different checks, and it's entirely possible for the former to be true every single night while the latter has quietly never been true at all.
Step 2 — Identify the likely bottleneck
A handful of common causes produce exactly this gap: a backup that completes but is silently truncated or corrupted due to a transient issue that doesn't register as a job failure; a backup format or version incompatibility with the restore tooling being used, discovered only when someone tries the two together; a backup that succeeds for the data itself but is missing something else the restore process depends on (a schema, a configuration, an encryption key); or a restore procedure that itself was never actually tested and has its own bug or missing step, independent of whether the backup file itself is fine.
Step 3 — Investigation
- Examine exactly where the restore failed and why — the specific error is the fastest path to distinguishing a corrupted backup file from a restore-process problem from a tooling incompatibility.
- Attempt to validate the backup file independently — checksums, file integrity checks, or opening it with the same tooling used to create it, outside of a full restore attempt, to isolate whether the file itself is sound.
- Check whether backup validation (beyond job-completion status) has ever been part of the backup process — if the job only checks "did the process exit successfully," that's the structural reason success and restorability have been able to diverge unnoticed for months.
- Check for any related changes — a database version upgrade, a backup tool version change, a storage location change — that might have occurred sometime during the months this has been silently broken, narrowing down when the actual problem was introduced.
Step 4 — Recommended action
Fix the specific cause the investigation surfaces — repair or reconfigure whatever's producing an unusable backup, whether that's the backup process itself, a tooling mismatch, or a missing dependency for restore. Then, critically, change what "backup success" means going forward: add automated restore testing as a regular, scheduled part of the backup process itself — not a one-off DR test every so often, but a routine validation (even against a non-production environment) that confirms recent backups are genuinely restorable, on an ongoing basis.
This turns "the job exited with status 0" into "we know, with evidence updated regularly, that our backups actually work" — which is the claim that actually matters when a real disaster happens.
Step 5 — Engineering lesson
A green backup job status is reassuring and almost entirely unearned as confidence unless something has actually verified restorability, not just completion. This is precisely why disaster recovery testing exists — not to check a compliance box, but because it's often the only way anyone finds out, before a real disaster, that "backups have been succeeding for months" and "backups actually work" were never the same claim.
🧠 Today’s Takeaway
Backup success does not prove recovery success.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

