Day 58 of 100
The AI-Generated Runbook
🔥 Problem Statement
A team wants to speed up incident response by having AI automatically generate runbooks — step-by-step remediation guidance — based on patterns from past incidents. The appeal is obvious: faster access to guidance during a real incident, when every minute matters. The risk is also real: a runbook is instructions someone follows, often under pressure, sometimes including commands that directly affect production.
🏗️ Environment
- Proposal: AI generates incident runbooks automatically from past incidents
- Intended use: during real production incidents, potentially under time pressure
- No existing safety review process for AI-generated operational content
- Runbooks would include specific remediation commands/actions
🤔 Your Challenge
What should a safe AI-generated runbook actually contain, and what guardrails does it need before anyone uses it live?
- What's the difference between a runbook suggesting an action and a runbook everyone trusts blindly under pressure?
- What would you want an AI-generated runbook to cite, so a responder can verify its reasoning quickly?
- Should a runbook execute anything automatically, or always require human approval for consequential steps?
- How would a bad AI-generated runbook step be caught before it's followed live during a real incident?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
A runbook followed under real incident pressure is one of the highest-trust documents in an SRE practice — a responder, often stressed and moving fast, is meant to be able to follow it with a reasonable degree of confidence rather than independently re-deriving every step from scratch in the moment. That trust is exactly what makes AI-generated runbooks risky if applied carelessly: a runbook that's wrong, or subtly right for a different situation than the one actually unfolding, gets followed anyway if the whole point was reducing the amount of thinking required under pressure.
Step 2 — Identify the likely bottleneck
There's no current safety review process for AI-generated operational content — meaning a runbook could go from "generated by a model based on pattern-matching against past incidents" directly to "followed live in production" with no human check in between confirming it's actually correct, safe, and applicable to the current situation. Given these runbooks would include specific remediation commands, an incorrect or miscalibrated step isn't a documentation problem, it's a potential production-affecting action taken on bad guidance.
Step 3 — Investigation
Before this ships as a live tool:
- Determine what past-incident evidence each generated runbook step is actually based on — a runbook that can show "this step worked in incident X, here's the link" is verifiable; one that can't is asking for blind trust.
- Identify which steps, if any, are safe to suggest versus safe to execute automatically — reading a metric is low-risk; restarting a production service or modifying data is not, and these categories need very different levels of safeguard.
- Check how the system would handle a novel incident that doesn't closely match historical patterns — does it generate a plausible-sounding runbook anyway, potentially confidently wrong for a situation it doesn't actually have good precedent for?
Step 4 — Recommended action
Design AI-generated runbooks to include their evidence explicitly — links or references to the past incidents they're pattern-matched from, so a responder can quickly judge relevance to the current situation rather than trusting the output blindly. Require any consequential action (anything beyond read-only investigation) to be explicitly approved by a human before execution — the AI can propose and pre-stage a remediation step, but a person confirms it applies to this specific situation before it runs.
Before any AI-generated runbook reaches live incident use, run it through a review process — ideally validated against a real or simulated past incident by someone who actually handled it, confirming the guidance is sound. Version and audit every generated runbook, so if one turns out to contain a bad step, it's traceable and correctable, and the pattern that produced it can be reviewed.
Step 5 — Engineering lesson
Automation's value is making the correct, safe response faster to reach — not skipping the verification that made a runbook trustworthy in the first place. An AI-generated runbook without evidence, without a review process, and without approval gates on consequential actions optimizes for speed at the exact moment (a live incident, under pressure) where an unverified bad instruction is most likely to be followed without question and most costly if wrong.
🧠 Today’s Takeaway
Automation should make the safe path easier—not remove the safety checks.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

