Engineering runbooks, docs & workflow checklists

Most migration disasters look like "it worked everywhere except production." The cause is almost always that production has shapes your fixtures don't — volume, distribution, NULLs, foreign-key densities — so a migration that finishes in 2 seconds against a 12-row staging table hangs behind a row lock, drops a column something quietly depended on, or adds 90 seconds of latency at 2:14 AM.

Split the test into two phases. Phase 1 — schema validity (does the SQL parse, do constraints accept, do FK references resolve) is what CI already does against an empty schema. Phase 2 — behavior under real data is what CI can't do, and it's where the failures live. The workflow: pg_dump -Fc -j 4 a production snapshot, pg_restore it into a sandbox DB, run the migration with \timing on piped to a log, then run a post-migration-checks.sql file that asserts row counts match, new columns are populated, and no orphans appeared. For databases over ~100GB, snapshot a slice (WHERE created_at > now() - interval '90 days') or use pg_sample to keep FK relationships intact. After setup, each rerun is just restore + apply, about five minutes.

Five patterns a dry-run catches that staging never will: lock contention on large tables (fix: ADD COLUMN ... DEFAULT NULL, backfill in batches, SET DEFAULT separately); a NOT NULL constraint failing on pre-existing rows the default never touched; a non-concurrent CREATE INDEX locking writes (use CREATE INDEX CONCURRENTLY); FK tightening that trips over months-old orphaned rows; and data transforms (UPDATE users SET email = LOWER(email)) that corrupt the NULL edge cases production has and tests don't. Test the rollback in the same pass — apply, capture state, apply the down-migration, confirm it matches the pre-migration snapshot. If it doesn't, at least you learned the migration is irreversible before production, when the plan is still "restore from backup."

The classic failure: your incident folder has 47 files, the most-used is the second one ever written, and the other 45 might as well not exist. The fix isn't a better doc tool — it's a tighter format. Use a four-field troubleshooting card: Error, Cause, Fix, Verify.

• Error — verbatim, the exact string a developer would grep for, including the distinctive stack frame. Not paraphrased.
• Cause — one sentence on the root cause, not the symptom: "connection pool exhausted because the ORM doesn't release connections in error paths," not "too many connections." If you can't write it in one sentence, the diagnosis isn't finished.
• Fix — the actual change: code, config, or command, with the diff or PR link. "Wrap the handler in try/finally that calls pool.release()," not "use proper connection lifecycle management."
• Verify — the single copy-pasteable check that proves it worked: a SQL query, a curl, a log search. "SELECT count(*) FROM pg_stat_activity WHERE state = 'idle in transaction' — should be near 0 under load."

A complete card takes five minutes and saves the next person an hour. The format wins on three properties: it's greppable (future-you searches the error string and the fix is on the same screen), mergeable (new variants append under a VARIANTS heading instead of spawning new files), and AI-friendly (the four fields map exactly to what an LLM needs to reproduce the fix). Store them as a flat directory of .md files named by error class — django-too-many-clients.md, nginx-502-upstream-prematurely-closed.md — where the filename is the index. Keep the incident timeline, stakeholder comms, and postmortem process notes out; those belong in other documents. Don't backfill 30 cards in a sprint — write one the next time a fix takes more than 30 minutes to diagnose, while the cause is still fresh. The team norm that makes it stick: a fix isn't done until the card is written, and the PR doesn't merge without it.

Because they're shaped for the wrong access pattern. On Stack Overflow the accepted answer — six lines you copy, paste, and ship — gets 10x the upvotes of the thorough answer that explains the mechanism and walks three approaches. That's not shallow learning; it's a structural fact about how developers consume docs when something is on fire: you want the snippet, education comes after. Internal docs usually invert it, explaining the why in prose and burying the what, so people Slack each other for the snippet instead and the doc gets read once at onboarding.

Write runbooks snippet-first, in three parts and this order: the runnable command, then "Why this works," then "When NOT to use this." The command leads, always. A reader at 2 AM gets it in five seconds, runs it, and the explanation is right there if they want to go deeper — they choose the depth instead of bouncing off paragraph one.

Three rules make a snippet copy-paste safe. Variables are obvious — use placeholders that scream replace-me (<tenant-id>, ${POD_NAME}, your-bucket-name), never a subtle 42 someone leaves in prod. Side effects are visible — lead with the --dry-run or -n version when the tool supports it, then the destructive one. Output is verifiable — show what success looks like (deployment "api" successfully rolled out) so the reader knows immediately if they're on track. Avoid the three anti-patterns: multi-step pipelines with no intermediate verification, "customize as needed for your environment" (the panicking reader has no time — list the exact variables), and snippets with no one-sentence statement of what they do. You don't have to rewrite everything: pick the one doc your team actually opens, hoist its command to the top, push the prose down, add the expected output. Twenty minutes, and people start linking the doc in chat instead of retyping the snippet.

Because long playbooks reliably degrade into shorter de-facto checklists in the head of whoever's running them — steps 1-3 get read, the middle gets skimmed, the last one or two get remembered because they're the "verify" steps people want to do anyway. The clinical-checklist research is blunt about it: a 5-item pre-flight list gets ~95% adherence, a 25-item list drops under 50%, a 50-item list below 20% — and the skipped items aren't random, people skip the middle. When something breaks at step 39, the post-mortem says "the playbook wasn't followed." It's right, and it was never going to be followed. The gap between the aspirational written version and the operational one is where bugs live.

Compress to 5-7 load-bearing steps with three rules. Merge verification into the action it checks — "run migration; verify with SELECT column_name FROM information_schema.columns WHERE column_name='status'" is one atomic step, not three. Bundle setup into preconditions — SSH-in, cd, pull, activate-venv aren't steps, they're the environment; capture them in a PRECONDITIONS block so step 1 is the first real action. Move if-then branches into linked sub-runbooks — a linear happy path with [see: rollback-runbook.md] links beats "if X jump to step 23." The two mental modes, happy-path execution and something's-wrong-pivot, deserve separate documents.

The skill isn't writing fewer words; it's sorting the long playbook into four buckets — required actions (5-7), their verifications (merge in), setup (preconditions), edge branches (sub-runbooks) — and putting each where it belongs. If you struggle to name the 5-7 load-bearing actions, that's a signal your team doesn't actually agree on what's load-bearing; resolve that first. Length is justified for one-time first-time setup, compliance/audit trails, and training material for genuinely new engineers. But for day-2 operations you run weekly, a 5-step checklist executed completely beats a 47-step one executed partially.

A new user opens your config.yml, scrolls past 247 lines of cache_strategy, retry_jitter_ms, and feature_flag_x_enabled, guesses at the 10% they recognize, saves, restarts, and something breaks. Then it's three days of debugging or, more often, a fork off to find a different tool. The author's instinct is to add more documentation explaining each option. The right move is the opposite: ship a five-line minimal config that works for 80% of installs, and move the other 240 lines to "advanced overrides." Caddy did exactly this with the Caddyfile, and tools that copy the pattern onboard measurably better.

"Default-safe" means three things: the config file is small (5-10 lines, so the user doesn't have to read it); every option in it has a sane default (remove any line and the tool still starts); and every other option is documented elsewhere, so the new user never sees it. The structure is a layer split — config.yml holds only the absolute minimum, a separate advanced.yml (or a docs page) lists every option with its default, and the runtime merges them: the minimal file wins where set, everything else falls back to defaults. The new user edits two lines and runs; the power user copies the specific overrides they need. Nobody stares at 240 unfamiliar options on day one.

How do you know which options belong in the minimal file? Look at what your existing power users actually override — the top 5 most-overridden settings, which telemetry or a scan of "how do I set X?" forum threads will surface fast. Most tools find 2-3 settings cover 80% of installs; that's your minimal config, and everything below it is documented-but-not-enforced. The uncomfortable truth underneath: most tools that win adoption don't have better features than the alternatives, they have an easier first hour. Your onboarding cost is part of your product — a five-minute install beats a ten-feature install nine times out of ten.

The formats that make runbooks useful to humans turn out to make them useful to AI coding agents for the same reasons — and once an agent is in the loop, the rules tighten. Three patterns carry over directly. Four-field troubleshooting cards (Error → Cause → Fix → Verify) map exactly onto the structure an LLM needs to reproduce a fix: paste two or three relevant cards before describing a new error and the agent's first reply is sharper, because it has examples of how your team thinks about similar problems instead of generic Stack-Overflow-grade suggestions. Snippet-first runbooks beat prose-first ones for agents for the same reason they beat them for humans — the agent extracts the runnable pattern faster, and the constraint context ("when NOT to use this") sits right next to the command where it's actually applied. Compressed checklists matter more for agents than for people: agents have limited context windows, so a 50-step playbook burns budget that should go to the actual work; a 5-step runbook plus two or three linked sub-runbooks is the right shape for both.

The practical loop is to keep the cards close to the code, then feed the relevant ones in when debugging: "here's our existing runbook for similar errors: [paste 2-3 cards]. We're now hitting [new error]. Propose a diagnosis." The agent avoids generic suggestions because the cards demonstrate that your team prefers specific, verified fixes. And with more than one agent available, the workflow parallelizes — one terminal can be executing a fix while another searches the card library for a matching prior incident. Prose-first, 47-step, backfill-everything docs serve neither the human nor the agent; the same tight formats serve both.

The moment more than one role touches AI-agent work, you get the failure a recent r/ClaudeAI thread describes: the spec lives in Notion, acceptance criteria are in Jira/Linear, the agent reads CLAUDE.md, QA works off a task file, and the tech lead's real intent is in a Slack thread nobody linked. Six surfaces, six versions of "the truth," and the agent confidently builds against whichever one it was handed. Nobody is wrong; the docs just drifted.

The fix is to pick ONE canonical surface per kind of truth and make everything else a pointer to it. Split truth into three layers and give each an owner:

• Intent — why, and what "done" means. Lives in your ticket system (Jira/Linear/GitHub Issues), owned by the PM. One issue = one unit of agent work, with acceptance criteria written as checkable statements, not prose.
• Constraints — how this repo wants to be built. Lives in a checked-in CLAUDE.md / agent-rules file, owned by the tech lead, versioned with the code so it can never be more current in Notion than in the repo.
• Task state — what the agent is doing right now. Lives in a task file next to the code, disposable, regenerated per unit of work.

Then enforce a one-way rule: Notion and design docs may explain, but they must link to the ticket, never restate its acceptance criteria — the moment they restate, they drift. QA reviews against the ticket's criteria, not a private checklist. The test that this is working: pick any in-flight change and ask "which single artifact says what done means here?" If two people answer differently, that's your next drift to close — not a tooling problem.

(Mapping cost to that work is its own problem — see model routing & AI coding costs; for proving the work is scoped and safe, see agent guardrails & proof loops.)

A working demo and a tool twelve people depend on every morning are different products, and the gap is exactly what a logistics intern hit in a recent r/ClaudeAI thread: a vibe-coded "loading cockpit" replaced the spreadsheets and whiteboards fine, but "it works on my laptop" isn't a system the morning shift can trust. Promotion isn't a rewrite — it's a checklist, run once, in this order:

1. Data safety first. Where does it write, and what happens if two people save at once? A single shared SQLite file on someone's laptop is the classic vibe-coded trap. Move to a real database, add a UNIQUE constraint or a transaction where concurrent writes collide, and confirm nothing important lives only in browser localStorage.
2. Backups before access. A nightly pg_dump (or a hosted DB with point-in-time restore) and one tested restore — an untested backup is a hope, not a backup. Do this before you invite users, because that's when the data starts mattering.
3. Access control. Even a shared-password gate beats an open URL on the office network; if it touches anything sensitive, per-user login. Decide who can edit vs view.
4. Deployment off the laptop. It has to survive the author closing their machine — a small VPS or internal host, started by a process manager, not npm run dev in a terminal.
5. A five-minute smoke test covering the two or three flows the shift actually runs, so a bad deploy is caught before the shift, not by it.
6. Handoff state. Write four-field cards for the failures you already hit and a one-page "how to restart it" runbook — the intern won't own this forever.

Ship these six and the demo becomes dependable. On the cost-sustainability worry another thread raised — heavy Claude/Fable usage versus API-rate economics — that's a real ceiling; track it rather than discover it (see model routing & AI coding costs). For the four-field cards in step 6, reuse the troubleshooting card format entry above.

What changed is that you no longer control the inputs. Before launch the only path through the code was the one you demoed. Now there are dozens, driven by people who never read your assumptions, over data you didn't imagine. Build speed bought you nothing here, and in one specific way it cost you: code you didn't write line by line is code whose invariants you don't know, so you can't predict what a change will break. That's the maintenance cliff — not bad code, unknown code.

Three loops to build, in this order:

1. Reproduce before you fix. A production-only bug is a difference in input, state, or environment, and the fix rate collapses when you guess at which. Capture the actual failing input and get it running locally as a fixture. Then make the habit stick: every incident ends with a test that fails against the old code. That's how a bug converts into a regression net instead of a memory.

2. Make regressions structural rather than remembered. "Change this without breaking that" is unanswerable when the behaviours were never specified and the agent that wrote them is long gone. So specify them retroactively: write the acceptance criteria for the behaviour you already have, prioritised by what would embarrass you if it broke, one incident at a time. You're converting an unknown-shaped system into a specified one, incrementally, funded by the failures as they arrive.

3. One change per deploy. AI-built codebases tend to carry wide implicit coupling, so a batched deploy that breaks something gives you no signal about which change did it. Small deploys are the cheapest debugging tool available, and they get cheaper the more you do them.

Then budget for the part that surprises solo builders most: the support load itself is now the work. Triage, a channel for users to tell you what they saw, and a standing decision about which reports you act on. That isn't a tooling problem — it's time allocation, and it competes directly with building.

The prototype-to-production checklist above is the entry fee. This is the recurring cost, and the evidence loops in AI coding agent guardrails & proof loops are what keep it from growing.

The meeting fails because you're asking people to predict a behavioural change by reading a diff of English. Nobody can do that. Prompts are code with a stochastic runtime, and the only thing that settles a disagreement about code is a run.

A gate that fits a small team has five parts:

1. Version the candidate with a hypothesis attached. Every change names the failure mode it's meant to fix — "reduces wrong tool selection on ambiguous requests." A change with no stated failure mode can be rejected on that basis alone; it's a preference, and preferences are what the meeting is drowning in.

2. A fixed regression set that grows out of real failures. Twenty to fifty cases, each one something that actually went wrong in production, with the expected outcome labelled by a human at the time. Never edit a case to make a candidate pass. This set is the only asset in the whole process that compounds, and it's worth more than any framework you put around it.

3. Score cases individually and look at what moved. One average is how you ship a change that fixed four cases and broke two more important ones. The review question becomes "which cases changed direction, and do we accept that trade?" — which is answerable, unlike "is this prompt better?" Keep the specific production examples where the candidate reliably does the right thing; a concrete case beats a decimal place in a group discussion.

4. Guardrail metrics, tracked separately from quality. Latency, token cost, tool-call count, format-failure rate. A candidate that scores better and costs three times as much is a business decision, and it should arrive at the meeting labelled as one.

5. A canary with the rollback rule written before rollout. A small traffic share, a fixed observation window, and a pre-committed threshold on one guardrail that triggers revert. Deciding the rule in advance is precisely what prevents the post-hoc argument about whether the numbers were bad enough.

The division of labour most teams settle on: the offline regression set gates the merge, the canary gates the rollout, and manual spot checks exist to find new cases for the set — not to approve changes. Keep the human labels. Automated scoring drifts in the same direction as the thing it's scoring, and a small human-labelled set outlives a large machine-graded one.

Keep prompts and their versions in the repo, where code review already happens, so the diff, the hypothesis, and the case results land in one place.

The instinct that this needs solving is right, and the reason it stays unsolved is that these scripts fall between two accepted categories: not product code, but not disposable either. So they end up in a personal folder, a chat thread, or a wiki page — which is precisely how the same dangerous operation gets rewritten from memory two years later by someone who wasn't there the first time.

Treat it as a small archive with a lifecycle rather than a folder:

• Versioned in the repository, in a directory that is not deployed. Same review, same history, no runtime exposure. The reviewability is most of the value here — a bulk data operation deserves a second pair of eyes far more than an ordinary feature does.
• A dry-run mode that is the default. Report what would change, with counts, before anything writes. For bulk work the count is the check: "this will update 3 rows" when you expected 30,000 is the signal that saves you.
• Fixtures or a restored snapshot to run against, so "I tested it" means something specific rather than "it didn't crash on my machine."
• A header block answering the questions you'll have later: what it does, who approved it, when it ran, against which environment, and what the outcome was. Ownership and dates matter more than elegance.
• Recorded execution evidence — the invocation, the output, the row counts. This is the difference between a script you can trust next time and one you'll rewrite because nobody can say whether it worked.
• An explicit expiry. A script referencing a schema that has moved on is worse than no script, because it reads as tested. Review the directory whenever the schema changes and delete what no longer applies.

None of this needs a platform. It needs the scripts to stop being invisible.

Because those tools answer "what is the state of X" and an incident asks "what changed, what does it touch, and who owns it" — a question that spans all of them and lives in none. Every dashboard is a correct answer to a question you didn't ask at 2am.

Where the time actually goes, roughly in order:

• Assembling a timeline across systems. The deploy is in CI, the config change is in Git, the feature flag is in a third tool, the alert is in the pager. Ordering those by time is manual, and it's the first thing anyone does.
• Establishing blast radius. "Checkout is broken" doesn't say which dependency, which region, or which cohort. Service maps go stale, so people reconstruct it from memory and search.
• Finding the owner. Frequently the single longest step, and almost never a technical problem.
• Deciding whether the obvious recent change is the cause. Usually there are three recent changes and the correlation is ambiguous, so someone has to reason about it under time pressure.

What helps is unglamorous: a single ordered change feed — deploys, config, flags, infrastructure, migrations — across systems, queryable by time window and service. Most teams can assemble one from what they already emit, and it removes the largest chunk of that half hour without adding another dashboard to check.

Two things help nearly as much. Ownership resolvable from a service name without asking in chat. And a decision log kept during the incident rather than reconstructed after — which is also why postmortems tend to read as though the path was obvious: they're written from the end.

You're not missing a tool. You're missing the join between the ones you already have.

The core mistake is treating a production pipeline as a long-running agent task. An agent asked to hold a hundred interacting rules in context will silently drop some on every run, and that is what the returning defects are — not a rule that broke, a rule that was not applied this time. Non-determinism is fine while drafting and disqualifying in a pipeline.

So move everything that can be deterministic out of the model:

• Split generation into stages with defined artifacts between them. Content assembly, structure, formatting, document assembly. Each stage takes a file and produces a file, so a defect belongs to a stage instead of to "the workflow".
• Make formatting code, not prose. Layout rules expressed as instructions get re-interpreted every run; the same rules as a template or a function do not. This one change usually removes most recurring layout regressions.
• Freeze reference outputs. A known-good artifact per stage, diffed after every change. This is the regression suite, and it is exactly what is missing when structural checks pass but the output is wrong.
• Add visual comparison of rendered pages. Page images against the reference catch overflow, pagination and broken tables — semantic failures no schema check will ever find.
• Regenerate only what changed. With staged artifacts a formatting fix re-runs the formatting stage against cached content. That is what stops a rule fix costing a full run.
• Give every rule an owner and a check. A rule not attached to a check is a rule that will regress unnoticed.

The direction to move: the model's job shrinks to what needs judgement — drafting prose, interpreting a data anomaly — and everything mechanical becomes code you can test. At 100 pages and 800 formulas you are running a build system. Build systems are staged, cached and deterministic, and the reason they are is precisely the problem you have.

Ten generated scripts is not a migration plan; it is ten opinions about a migration, written independently, with no shared idea of ordering or failure. The inconsistency you noticed is the important signal — it means there was never one model of the operation, so the scripts cannot be reviewed as a unit even though they will run as one.

Rather than reading them line by line in the order they arrived, impose a structure and make the generated material fit it:

• Write the sequence first, in plain language. Every step, its precondition, its expected effect, and how you would know it worked. Steps no script implements, and scripts fitting no step, are both findings — usually the most useful ones.
• Separate read-only from mutating. Anything that only inspects can be run now against the real system, and running it is the fastest way to find out that a script's assumptions about the environment are wrong. Everything mutating stays behind an explicit decision.
• Demand idempotence, or an explicit statement that it is absent. Migrations get re-run after partial failure. A script that only works against a clean system needs to say so, or you find out the hard way.
• Identify the point of no return — usually the first write to the new system, or the traffic cutover — and define what rollback means before and after it. The step where rollback stops being possible earns the most review.
• Rehearse against a copy. A restored snapshot, timed. This surfaces environment differences and duration surprises, which are the real risks and neither of which appears in a code review.
• Verify data, not completion. Row counts, checksums on key tables, a query returning identical results on both sides. "The script exited zero" is not evidence that the data arrived.

On the verbosity: prefer fewer scripts that read clearly over many that are individually short. You have to understand this well enough to debug it at an unpleasant hour, and that is the standard — not whether it looks tidy.