Add Exponential Backoff Retries to Next.js Backgrounds

At two o'clock on a Tuesday afternoon, your transactional email provider experiences a minor network blip.

For exactly four minutes, their API returns HTTP 500 errors.

During those four minutes, six people purchase your software product.

If your background tasks do not have automatic retries with exponential backoff, those six customers never receive their license keys. They assume your app is a scam, open credit card disputes, and send angry support emails.

Building reliable background systems is not about avoiding failures. It is about how your code behaves when external systems inevitably fail.

The failure modes of naive retry logic

When developers realize they need retries, they usually implement one of two flawed approaches.

The first flaw is the immediate retry loop:

// Naive approach: immediate retry
for (let i = 0; i < 3; i++) {
  try {
    await sendReceipt(order);
    break;
  } catch (err) {
    // Retries instantly with no delay
  }
}

If the upstream service is experiencing a network partition or database failover, retrying three times in fifty milliseconds will fail three times in fifty milliseconds.

Worse, if thousands of customers hit this loop at once, your application creates a self-inflicted denial-of-service attack, hammering an already struggling provider.

The second flaw is the infinite unmonitored retry.

Tasks fail, re-queue immediately, and spin your server CPU into a runaway tailspin while hiding the underlying error from your team.

You need controlled backoff paired with a dead-letter state.

What is exponential backoff with jitter?

Exponential backoff is a mathematical strategy that increases the waiting time between consecutive retry attempts exponentially.

The formula is straightforward:

delay = base_delay * (multiplier ^ attempt_number)

If your base delay is five seconds:

• Attempt 1 fails: wait 5 seconds before retrying.
• Attempt 2 fails: wait 25 seconds before retrying.
• Attempt 3 fails: wait 125 seconds before retrying.
• Attempt 4 fails: wait 300 seconds (capped at a sensible maximum).

This spacing gives the upstream service time to recover, reboot, or finish its failover routine.

By the time your fourth retry fires five minutes later, the provider is healthy again, the email delivers cleanly, and the customer never knows a blip occurred.

The safety net: Dead-letter queues

What happens if an API key is revoked, or a customer typed an invalid email format that will never succeed?

You cannot retry forever. After a fixed number of attempts (typically three to five), the task must move to a dead state.

A dead-letter queue is not a trash can. It is an quarantine holding pen.

It preserves three critical pieces of evidence:

1. The original payload data so work is never lost.
2. The exact error message and stack trace from the final failure.
3. The timestamp of the failure for auditing.

When your team fixes the root cause (such as updating an expired API secret), you run a single command to replay all quarantined jobs cleanly.

Implementing backoff and dead letters on SQLite

You do not need an enterprise service bus to implement this pattern.

In BoringPush, boringqueue handles this state machine directly inside a local SQLite table (db/queue.db):

CREATE TABLE IF NOT EXISTS queue_jobs (
  id TEXT PRIMARY KEY,
  path TEXT NOT NULL,
  payload TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  attempts INTEGER NOT NULL DEFAULT 0,
  max_attempts INTEGER NOT NULL DEFAULT 3,
  run_at INTEGER NOT NULL,
  last_error TEXT,
  created_at INTEGER NOT NULL
);

When you dispatch a task from a Next.js route, you specify the retry policy:

import { dispatchJob } from "@/libs/boringqueue";

export async function POST(req) {
  const { orderId } = await req.json();

  // Retry up to 5 times on failure
  dispatchJob("billing/send-receipt", { orderId }, {
    maxAttempts: 5,
  });

  return Response.json({ ok: true });
}

Inside your job handler in scripts/jobs/billing/send-receipt.mjs, you write standard async JavaScript.

If your code completes successfully, the worker marks the row as completed.

If an error is thrown, the worker catches the exception and computes the next execution time automatically:

// libs/boringqueue/core.mjs backoff computation
const nextDelaySeconds = Math.min(300, 5 * Math.pow(2, attempts - 1));
const nextRunAt = Date.now() + (nextDelaySeconds * 1000);

If attempts >= maxAttempts, the worker marks the job as dead and logs the full error details to disk.

No customer order is ever dropped.

Rescuing dead jobs with one command

When an incident occurs, you want fast resolution from your terminal.

BoringPush provides built-in CLI commands to inspect and recover dead jobs:

# View all dead jobs across your application
npm run bp -- jobs list --status dead

# Inspect the exact failure error for a specific job
npm run bp -- jobs show <jobId>

# Replay all dead jobs after resolving the issue
npm run bp -- jobs retry-dead

When you execute retry-dead, the queue resets the attempt counters, schedules the jobs for immediate execution, and your worker pool processes the backlog safely.

Autonomous AI agents like Claude Code can run these exact commands when responding to production alerts.

Reliability is a design decision

Software breaks. Third-party APIs go down. Network cables get cut.

Brittle software assumes external services work perfectly one hundred percent of the time. Resilient software expects failure and builds automated recovery into the foundation.

By adding exponential backoff and dead-letter queues to your Next.js application, you turn production catastrophes into minor delays that resolve themselves while you sleep.

FAQ

Does an unhandled rejection in a background job crash the Next.js server?

No. In BoringPush, background workers run in separate child threads managed by a thread pool. A crash or thrown error inside a job script is isolated to that specific worker thread and never impacts the main Next.js web process.

Can I set different retry counts for different tasks?

Yes. Fast, idempotent tasks (like sending a Slack webhook) can be configured with maxAttempts: 3, while critical financial operations (like provisioning user access) can use maxAttempts: 10.

What happens if the server loses power during a retry delay?

Because the run_at timestamp is written to the SQLite database file on disk, scheduled retries survive server reboots and power outages without data loss.

Should I retry on HTTP 400 Bad Request errors?

Generally, no. Client errors like 400 (Bad Request) or 404 (Not Found) indicate permanent input problems that will not succeed on retry. In your job code, validate inputs first and only throw on network or server errors (500s).

How do I know if jobs are accumulating in the dead-letter queue?

BoringPush integrates with Telegram and Discord alerts via libs/telemetry.js. You can configure alerts to notify your phone whenever a job enters the dead state.

Keep reading

• Next.js Background Jobs Without Redis: $0 BullMQ — the full queue setup
• Why Next.js Background Tasks Die in Production — solving serverless freezes
• Cron vs. Background Queues in Next.js — understanding task boundaries