# 1-Second Rollback From Phone (PM2 + Git)

> Bad deploy at midnight? One symlink flip plus PM2 reload restores the last good release in seconds, from your phone. The pattern inside.

Source: https://boringpush.com/blog/rollback-from-phone-pm2

## The deploy that worked everywhere except prod

The hook built clean. Lint passed.

Staging looked perfect. And production checkout 500s on every third order — a data-shape difference staging never had.

It is midnight. The fix needs an hour of thought.

Your users need thirty seconds of action. Forward-fixing from a phone keyboard at midnight is how one incident becomes two.

Rollback from phone PM2 thinking says: retreat first, think later. The last good release is still on the box — point the server at it and go back to sleep.

## Retreat options ranked by seconds

| Retreat | Needs | Time | Risk |
|---|---|---|---|
| Fix forward from phone | Insight + typing | 30–90 min | Second bug, tired thumbs |
| Rebuild previous commit | Full rebuild (~3 min) | Minutes | OOM spike mid-crisis |
| CI redeploy old SHA | Dashboard + queue | 5–15 min | Queue behind others |
| Symlink flip + reload | One command | Seconds | Near zero — it already ran |

Three details make rebuild-based retreats dangerous at midnight:

1. **Rebuilds can fail.** The old commit built fine last week against last week's registry; tonight a transitive dependency moved and the "safe" rebuild dies. The kept release already built, already booted, already served traffic — certainty, not hope.
2. **Queues do not care about your outage.** Shared CI runners process your rollback behind someone else's matrix build. Seconds become quarter-hours while users refresh errors.
3. **Memory spikes during crisis.** A rebuild beside a struggling app is the OOM post's worst case wearing a trench coat. The flip touches no compiler and allocates nothing.

Platforms sell "instant rollback" as a premium feature. Keeping the previous build on disk is the same feature for free.

## Keep, flip, confirm

BoringPush bakes this into every deploy. Each one keeps its predecessor on the server, so going back never needs a rebuild.

**Step 1: Keep the last good build on disk.** When `bp deploy` ships a new version, the previous build stays next to it on the server. A rollback does not rebuild anything, so it never meets a failing registry, a busy CI queue or an out-of-memory spike. The kept build already built, already booted and already served traffic, which is certainty, not hope.

**Step 2: Flip back from the phone in one command.** From your SSH app, run `bp rollback --yes`. It swaps the previous build back in and reloads the app. Without `--yes`, it refuses, so an agent or a half-awake thumb cannot roll back by accident. If your server is not set up or cannot be reached, the command tells you so and stops.

**Step 3: Confirm silence, then sleep.** Open the live site and check that it loads. Watch your Telegram or Discord alert channel for five minutes. A healthy page plus a quiet channel means the retreat held. The forward fix waits for morning, a desk and coffee, written against the broken release, staged on a branch and approved by the daylight process. Midnight-you did exactly one wise thing and stopped.

That's the design earning its keep, including shipping backwards at midnight.

## The midnight drill card

| Check | Action | Pass looks like |
|---|---|---|
| Previous build kept | Look at the server after a deploy | The last good build is still there |
| Flip works | Run `bp rollback --yes` on staging | The old version serves again in seconds |
| No rebuild | Watch the server during the flip | No compiler activity |
| Forward path intact | The broken build is not destroyed | Morning debugging has something to read |
| Guard works | Run `bp rollback` without `--yes` | It refuses |

Drill the flip every quarter on staging: deploy, break it deliberately, retreat from the phone and time it. A rollback you have never run is a rumor. A rollback you timed is a plan.

The other column, filled in: BoringPush keeps only the previous build, one step back, not a long history of releases. And a code retreat cannot undo database changes.

Additive-only migrations mean old code runs fine against new columns, which is exactly why the law exists. Retreat covers code, and the migration law covers data.

## FAQ

### Does a rollback work with database changes too?

Only forward-compatible ones, which is all the migration law allows. Additive columns with defaults are invisible to old code, so the flipped-back release runs happily. If a deploy ever needs a destructive change, it ships as three deploys across days, and rollback stays safe at every step.

### How is a rollback different from reverting and pushing?

Speed and certainty. Revert and push rebuilds, which takes minutes and can run out of memory or hit registry drift. The flip serves known-good bytes in seconds.

Revert is the right daylight cleanup, so commit it in the morning and history tells the truth. At midnight, flip first and revert second.

### What if the bad deploy also poisoned the database?

Then code retreat is step one of two. Flip first to stop the bleeding, then repair data with a targeted script against backups.

Never restore a full old snapshot over live writes, because that deletes tonight's good orders along with the poison. The daily snapshots are ingredients, and judgment picks the rows.

### Can the flip itself fail?

Rarely, since it replays a boot that already succeeded once. The residual risk is settings drift, such as a new required key the old release never read.

Syncing settings with `bp secret sync` keeps one file both releases share. Drill quarterly and even that residual gets witnessed.

### Should rollbacks alert the team like deploys do?

Tell your team in your usual channel when you roll back. A silent retreat confuses morning-you with "why is the fix gone?". Write the reason in one line and link the forward fix by noon.

### When is rollback the wrong call versus fixing forward?

When the blast radius is tiny and the fix is obvious, such as a typo in copy, a wrong redirect target or a missing setting you just synced. The flip costs seconds but also discards good changes riding along. Rollback is for "I don't fully understand this yet", which at midnight is most things.

## Keep reading
- [git push production deploy: Skip the 9-Minute CI Queue](/blog/git-push-production-main-deploy) — the deploy that keeps your last release
- [Fix Production From Your Phone: The Grocery-Line Drill](/blog/fix-production-from-phone-telegram) — the ping that tells you to retreat
- [PM2 Reloads Nobody Feels: Next.js With Zero Downtime](/blog/pm2-zero-downtime-nextjs) — the reload that keeps flips invisible to users