Engineering runbooks, docs & workflow checklists

Most migration disasters look like "it worked everywhere except production." The cause is almost always that production has shapes your fixtures don't — volume, distribution, NULLs, foreign-key densities — so a migration that finishes in 2 seconds against a 12-row staging table hangs behind a row lock, drops a column something quietly depended on, or adds 90 seconds of latency at 2:14 AM.

Split the test into two phases. Phase 1 — schema validity (does the SQL parse, do constraints accept, do FK references resolve) is what CI already does against an empty schema. Phase 2 — behavior under real data is what CI can't do, and it's where the failures live. The workflow: pg_dump -Fc -j 4 a production snapshot, pg_restore it into a sandbox DB, run the migration with \timing on piped to a log, then run a post-migration-checks.sql file that asserts row counts match, new columns are populated, and no orphans appeared. For databases over ~100GB, snapshot a slice (WHERE created_at > now() - interval '90 days') or use pg_sample to keep FK relationships intact. After setup, each rerun is just restore + apply, about five minutes.

Five patterns a dry-run catches that staging never will: lock contention on large tables (fix: ADD COLUMN ... DEFAULT NULL, backfill in batches, SET DEFAULT separately); a NOT NULL constraint failing on pre-existing rows the default never touched; a non-concurrent CREATE INDEX locking writes (use CREATE INDEX CONCURRENTLY); FK tightening that trips over months-old orphaned rows; and data transforms (UPDATE users SET email = LOWER(email)) that corrupt the NULL edge cases production has and tests don't. Test the rollback in the same pass — apply, capture state, apply the down-migration, confirm it matches the pre-migration snapshot. If it doesn't, at least you learned the migration is irreversible before production, when the plan is still "restore from backup."

The classic failure: your incident folder has 47 files, the most-used is the second one ever written, and the other 45 might as well not exist. The fix isn't a better doc tool — it's a tighter format. Use a four-field troubleshooting card: Error, Cause, Fix, Verify.

• Error — verbatim, the exact string a developer would grep for, including the distinctive stack frame. Not paraphrased.
• Cause — one sentence on the root cause, not the symptom: "connection pool exhausted because the ORM doesn't release connections in error paths," not "too many connections." If you can't write it in one sentence, the diagnosis isn't finished.
• Fix — the actual change: code, config, or command, with the diff or PR link. "Wrap the handler in try/finally that calls pool.release()," not "use proper connection lifecycle management."
• Verify — the single copy-pasteable check that proves it worked: a SQL query, a curl, a log search. "SELECT count(*) FROM pg_stat_activity WHERE state = 'idle in transaction' — should be near 0 under load."

A complete card takes five minutes and saves the next person an hour. The format wins on three properties: it's greppable (future-you searches the error string and the fix is on the same screen), mergeable (new variants append under a VARIANTS heading instead of spawning new files), and AI-friendly (the four fields map exactly to what an LLM needs to reproduce the fix). Store them as a flat directory of .md files named by error class — django-too-many-clients.md, nginx-502-upstream-prematurely-closed.md — where the filename is the index. Keep the incident timeline, stakeholder comms, and postmortem process notes out; those belong in other documents. Don't backfill 30 cards in a sprint — write one the next time a fix takes more than 30 minutes to diagnose, while the cause is still fresh. The team norm that makes it stick: a fix isn't done until the card is written, and the PR doesn't merge without it.

Because they're shaped for the wrong access pattern. On Stack Overflow the accepted answer — six lines you copy, paste, and ship — gets 10x the upvotes of the thorough answer that explains the mechanism and walks three approaches. That's not shallow learning; it's a structural fact about how developers consume docs when something is on fire: you want the snippet, education comes after. Internal docs usually invert it, explaining the why in prose and burying the what, so people Slack each other for the snippet instead and the doc gets read once at onboarding.

Write runbooks snippet-first, in three parts and this order: the runnable command, then "Why this works," then "When NOT to use this." The command leads, always. A reader at 2 AM gets it in five seconds, runs it, and the explanation is right there if they want to go deeper — they choose the depth instead of bouncing off paragraph one.

Three rules make a snippet copy-paste safe. Variables are obvious — use placeholders that scream replace-me (<tenant-id>, ${POD_NAME}, your-bucket-name), never a subtle 42 someone leaves in prod. Side effects are visible — lead with the --dry-run or -n version when the tool supports it, then the destructive one. Output is verifiable — show what success looks like (deployment "api" successfully rolled out) so the reader knows immediately if they're on track. Avoid the three anti-patterns: multi-step pipelines with no intermediate verification, "customize as needed for your environment" (the panicking reader has no time — list the exact variables), and snippets with no one-sentence statement of what they do. You don't have to rewrite everything: pick the one doc your team actually opens, hoist its command to the top, push the prose down, add the expected output. Twenty minutes, and people start linking the doc in chat instead of retyping the snippet.

Because long playbooks reliably degrade into shorter de-facto checklists in the head of whoever's running them — steps 1-3 get read, the middle gets skimmed, the last one or two get remembered because they're the "verify" steps people want to do anyway. The clinical-checklist research is blunt about it: a 5-item pre-flight list gets ~95% adherence, a 25-item list drops under 50%, a 50-item list below 20% — and the skipped items aren't random, people skip the middle. When something breaks at step 39, the post-mortem says "the playbook wasn't followed." It's right, and it was never going to be followed. The gap between the aspirational written version and the operational one is where bugs live.

Compress to 5-7 load-bearing steps with three rules. Merge verification into the action it checks — "run migration; verify with SELECT column_name FROM information_schema.columns WHERE column_name='status'" is one atomic step, not three. Bundle setup into preconditions — SSH-in, cd, pull, activate-venv aren't steps, they're the environment; capture them in a PRECONDITIONS block so step 1 is the first real action. Move if-then branches into linked sub-runbooks — a linear happy path with [see: rollback-runbook.md] links beats "if X jump to step 23." The two mental modes, happy-path execution and something's-wrong-pivot, deserve separate documents.

The skill isn't writing fewer words; it's sorting the long playbook into four buckets — required actions (5-7), their verifications (merge in), setup (preconditions), edge branches (sub-runbooks) — and putting each where it belongs. If you struggle to name the 5-7 load-bearing actions, that's a signal your team doesn't actually agree on what's load-bearing; resolve that first. Length is justified for one-time first-time setup, compliance/audit trails, and training material for genuinely new engineers. But for day-2 operations you run weekly, a 5-step checklist executed completely beats a 47-step one executed partially.

A new user opens your config.yml, scrolls past 247 lines of cache_strategy, retry_jitter_ms, and feature_flag_x_enabled, guesses at the 10% they recognize, saves, restarts, and something breaks. Then it's three days of debugging or, more often, a fork off to find a different tool. The author's instinct is to add more documentation explaining each option. The right move is the opposite: ship a five-line minimal config that works for 80% of installs, and move the other 240 lines to "advanced overrides." Caddy did exactly this with the Caddyfile, and tools that copy the pattern onboard measurably better.

"Default-safe" means three things: the config file is small (5-10 lines, so the user doesn't have to read it); every option in it has a sane default (remove any line and the tool still starts); and every other option is documented elsewhere, so the new user never sees it. The structure is a layer split — config.yml holds only the absolute minimum, a separate advanced.yml (or a docs page) lists every option with its default, and the runtime merges them: the minimal file wins where set, everything else falls back to defaults. The new user edits two lines and runs; the power user copies the specific overrides they need. Nobody stares at 240 unfamiliar options on day one.

How do you know which options belong in the minimal file? Look at what your existing power users actually override — the top 5 most-overridden settings, which telemetry or a scan of "how do I set X?" forum threads will surface fast. Most tools find 2-3 settings cover 80% of installs; that's your minimal config, and everything below it is documented-but-not-enforced. The uncomfortable truth underneath: most tools that win adoption don't have better features than the alternatives, they have an easier first hour. Your onboarding cost is part of your product — a five-minute install beats a ten-feature install nine times out of ten.

The formats that make runbooks useful to humans turn out to make them useful to AI coding agents for the same reasons — and once an agent is in the loop, the rules tighten. Three patterns carry over directly. Four-field troubleshooting cards (Error → Cause → Fix → Verify) map exactly onto the structure an LLM needs to reproduce a fix: paste two or three relevant cards before describing a new error and the agent's first reply is sharper, because it has examples of how your team thinks about similar problems instead of generic Stack-Overflow-grade suggestions. Snippet-first runbooks beat prose-first ones for agents for the same reason they beat them for humans — the agent extracts the runnable pattern faster, and the constraint context ("when NOT to use this") sits right next to the command where it's actually applied. Compressed checklists matter more for agents than for people: agents have limited context windows, so a 50-step playbook burns budget that should go to the actual work; a 5-step runbook plus two or three linked sub-runbooks is the right shape for both.

The practical loop is to keep the cards close to the code, then feed the relevant ones in when debugging: "here's our existing runbook for similar errors: [paste 2-3 cards]. We're now hitting [new error]. Propose a diagnosis." The agent avoids generic suggestions because the cards demonstrate that your team prefers specific, verified fixes. And with more than one agent available, the workflow parallelizes — one terminal can be executing a fix while another searches the card library for a matching prior incident. Prose-first, 47-step, backfill-everything docs serve neither the human nor the agent; the same tight formats serve both.