How to Roll Back a Bad Deploy Without Losing Data

To roll back a bad deployment without losing data, you revert the application code to the last known good release while treating the database as a separate, mostly forward-only problem. The code is easy: a symlink flip, a container image tag change, or a redeploy of a tagged commit. The data is where people get hurt, because a rollback that reverts a schema migration will happily drop a column that has been collecting writes since the deploy went out.

The core discipline is to separate two questions. Can the previous release run against the current database state? If yes, rollback is a code operation and takes seconds. If no, you are not rolling back, you are doing a forward fix or a data migration, and you should say so out loud before anyone touches anything.

First, stop the bleeding and freeze writes

Before you revert anything, decide whether the bad release is still writing. A deploy that corrupts records on every request will keep corrupting them while you read the release notes. If the damage is ongoing, take the write path out of service first. That usually means a load balancer change, a feature flag, or a read-only mode in the application.

On most reverse proxies you can drain the backend without editing application config. With nginx you would comment out the upstream server and reload, which lets in-flight requests finish instead of killing their connections.

upstream app {
    # server 10.0.4.11:8080;   # drained during incident
    server 10.0.4.12:8080;
}

Reload rather than restart: nginx -s reload keeps the listening socket open and finishes current requests. If you are behind a cloud load balancer, deregister the instance and wait for the connection count to fall to zero before you stop the process. Cutting connections mid-transaction is how you turn a bad deploy into a corrupted table.

Revert the code with a version you can name

A rollback is only fast if you can point at an exact artifact. Tags and digests, not branch names. If your deploy pipeline produces an immutable image or a release directory, reverting is a one-line operation and you should rehearse it before you need it.

With a symlink-based release layout, the whole rollback is a link swap and a restart of the application server, not the web server.

ls -l /srv/app/current
# current -> /srv/app/releases/20240114-0930

ln -sfn /srv/app/releases/20240113-1745 /srv/app/current
systemctl restart app.service
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/healthz
# 200

If you deploy containers, pin by digest rather than by a mutable tag, because a tag can be repointed and then your rollback target is not the thing you tested. If you deploy from a git repository, check out the previous release tag and rebuild, but be aware that a rebuild is not guaranteed to be byte-identical to the artifact that ran before, which is why keeping the built artifact is worth the storage.

Once the old code is serving, watch two things: error rate and the shape of the data it is writing. Old code can be perfectly healthy and still write rows the new schema expects to be populated. That is the failure mode that turns a five minute rollback into a two day cleanup.

Rolling back code against a migrated database

This is the part that decides whether you have a rollback or a rescue. Ask one question: did the deploy apply a schema change? If it did, the previous release may not be able to run at all, and reverting the code can make things worse by writing through a schema it does not understand.

The safe pattern is expand and contract. Add new columns as nullable, backfill, deploy code that writes both old and new shapes, then remove the old column in a later, separate deploy. Under that discipline every release is compatible with the one before it, and a rollback is genuinely a code-only operation. If your team does not work that way, the honest answer is that rollbacks are limited and you should plan for forward fixes instead.

When you must revert a migration, do it with a tested down migration and a backup taken immediately before, never by hand-editing a schema on a live primary. Confirm the tool's version table so you know exactly which revision you are on and which one you are going back to.

SELECT version, applied_at FROM schema_migrations ORDER BY version DESC LIMIT 5;
#  20240114_add_currency   | 2024-01-14 09:31:02
#  20240109_add_index      | 2024-01-09 11:02:44

If the migration dropped or renamed a column, the down migration cannot restore the data that was in it. Restoring from a backup means losing every write since that backup, which is usually worse than the bug. In that situation, restore the column as a new one, backfill from the backup into a staging copy, and reconcile the gap by hand. Slow, but it does not throw away customer writes.

Files, uploads and caches

Code rollback does not roll back user uploads, generated thumbnails, or cache entries. If the bad release wrote files in a new layout, the old release may not find them. Check whether your storage path is versioned. If object keys changed format, leave the new keys in place and let the old code read them, or copy them into the old layout during a maintenance window rather than deleting anything.

Caches are the opposite: you want them gone. A stale cache entry written by the new code can be served by the old code and look like the rollback failed. Flush the application cache and any CDN path that changed, and remember that a CDN purge is not instant everywhere.

Finally, write down the exact commit, image digest, migration revision and cache state you rolled back to. That record is what lets you redeploy the fix later without guessing, and it is what you attach to the incident write-up.

What to do next

Rehearse the rollback on a staging copy of production data, with the same commands, before you need it in anger. Time it, and find out where it stalls. Then go and check whether your last few releases were backward compatible: if any of them dropped or renamed a column in the same deploy as the code change, that is the release that will hurt you, and the fix belongs in your deploy process, not in your incident runbook.