How to Document Your Backup and Recovery Plan Before an Emergency

A backup and disaster recovery plan is only as good as the document that tells someone what to do at three in the morning when the primary database will not start. Write that document before you need it, keep it outside the systems it describes, and make it specific enough that a competent engineer who has never touched your stack can follow it under pressure.

The document has one job: state what to restore, from where, in what order, and how to prove the restore worked. Everything else is commentary. If a step requires judgement, say so and say who is allowed to make the call.

Write the header first, because it answers the panic questions

Start every runbook with a header block. The header is not decoration. It tells the reader whether they are even allowed to start, and who to wake up. Keep it in the same file as the procedure so it cannot drift out of sync.

service: checkout-api
owner: platform-team
escalation: oncall-platform (pager), then eng-manager
rpo: 15m
rto: 2h
last-restore-test: recorded in test log, see section 7
source-of-truth: primary managed database, region A
restore-target: staging cluster first, never straight to prod

The rpo and rto values are the two numbers that decide everything downstream. Recovery point objective is how much data loss is tolerable, which sets your backup frequency. Recovery time objective is how long the service may be down, which sets how much of the procedure can be manual. If your backups run hourly but your recovery point objective is fifteen minutes, the plan is already broken and no amount of careful writing fixes it.

Describe the sources, not the vendor

Name the mechanism, not the brand. "Nightly logical dump written to object storage, plus continuous write-ahead log shipping" tells the reader what to look for. A product name tells them nothing about where the bytes actually are.

For each source, record the storage location as a path or a URI pattern, the retention window, the encryption key location, and who holds the key. A backup you cannot decrypt is not a backup. Write down the key reference, not the key itself.

sources:
  - type: logical-dump
    schedule: every 6h
    path: s3://example-backups/checkout/dump-{date}.sql.gz.gpg
    retention: 30 daily, 12 monthly
    key: referenced in secret store as backup/gpg-key
  - type: wal-archive
    path: s3://example-backups/checkout/wal/
    retention: 14 days
  - type: config-and-secrets
    path: git repo, tagged releases
    note: infra config is restored from git, not from the database backup

That last line matters more than people expect. Application config, environment variables, proxy rules and DNS records are usually not inside the database dump. If they live only in a running container, you have no backup of them at all. Keep them in version control and tag the release that matches each deployment.

Order the restore steps and state the dependencies

Restores fail on ordering, not on commands. Write the sequence as a dependency chain: what must exist before the next step can run. A typical order is network and DNS, then secrets, then the database, then the application, then background workers, then the queue, then external integrations.

For each step, give the exact command, the expected output, and the failure signal. Do not write "restore the database." Write the command.

# pull the most recent dump and confirm it decrypts
aws s3 cp s3://example-backups/checkout/dump-latest.sql.gz.gpg .
gpg --decrypt dump-latest.sql.gz.gpg | gunzip | head -n 5

# expected: valid SQL, CREATE TABLE / SET statements
# failure: gpg error means wrong key, stop and escalate

Notice the verification built into the step. Piping through head proves the file decrypts and decompresses before you spend twenty minutes loading it into an empty cluster. Cheap checks first, expensive operations last.

Verification is a separate section, not an afterthought

The plan is not complete until it says how you know the restore worked. Row counts against the last known good snapshot, a checksum on a critical table, a smoke test that exercises login and one write path. Write the query, not the idea.

SELECT count(*) FROM orders WHERE created_at > now() - interval '24 hours';
-- compare against the number recorded in the incident log

Record the expected value at the time of the incident, because after a restore the "expected" number is whatever you decide it is. A verification step with no reference number verifies nothing.

Test the runbook and date the tests

A plan that has never been executed is a hypothesis. Run the restore into a scratch environment on a schedule, and have someone other than the author follow the document without asking questions. Every place they hesitate is a gap. Fix the document, then run it again.

Keep the test log in the same file. A short entry recording what was restored, whether the verification passed, and how long the whole thing took is enough. If the measured time exceeds the recovery time objective, the objective is wrong or the procedure is too slow, and you need to know which before an incident decides for you.

Store the runbook where it survives the outage. Version control is a good default, but a copy in an offline location or a printed sheet matters when the failure takes your repository host with it. The document describing how to recover must not depend on the thing being recovered.

What to do next

Pick one service, the one whose loss would hurt most, and write its runbook this week using the structure above. Then schedule a restore into a scratch environment and have a colleague follow your document while you watch without helping. Every question they ask is a line you need to add. Repeat that loop for the other services only after the first one survives the test, because a plan nobody has executed is just a wish with headings.