Run Long Background Tasks in Next.js Without Timeouts

You added a feature to let users import a contact list from a CSV file. Or you connected an OpenAI endpoint to generate a full marketing report.

In local testing with ten rows, it completes in three seconds.

In production, a customer uploads a file with five thousand rows. Forty seconds later, their browser shows an ugly error page: 504 Gateway Timeout.

The import failed halfway through, some rows were saved while others were lost, and your user is frustrated.

Why HTTP connections hate long compute jobs

Web browsers and serverless reverse proxies are designed for rapid request and response cycles.

When a browser opens an HTTP connection, every proxy, load balancer, and edge gateway along the path maintains an active TCP socket. Every network hop enforces a strict idle timeout.

Cloudflare drops connections after one hundred seconds. AWS API Gateway terminates requests after twenty-nine seconds. Vercel terminates serverless routes after ten seconds on Hobby plans, or sixty seconds on standard Pro plans.

If your code performs heavy data manipulation, video encoding, web scraping, or multi-step AI inference, you will hit these walls.

Trying to stretch HTTP timeouts is a losing battle. The correct architectural pattern is to accept the request immediately and execute the compute asynchronously.

The Accept-Fast and Poll-Slow pattern

The industry standard pattern for long-running operations splits the work into two distinct phases:

First, the synchronous HTTP handler accepts the file or parameters, records the user's intent in persistent storage, and returns an immediate response:

{
  "status": "queued",
  "jobId": "job_94821a",
  "checkUrl": "/api/jobs/job_94821a"
}

This HTTP response finishes in less than one hundred milliseconds. The browser never hangs, and Cloudflare never fires a 504 error.

Second, a background worker process pulls the job from storage, performs the compute outside the web server request lifecycle, and marks the task as completed when finished.

The client can then check the job status periodically or receive an instant push notification when the results are ready.

Why serverless struggles with heavy tasks

When developers attempt to implement background workers on serverless platforms, they run into three major friction points:

First, duration limits still apply to background lambdas. Even if you decouple the web route using SQS or QStash, your consumer lambda still faces execution caps.

Second, memory limits are expensive. Processing large video files or running headless browsers requires significant RAM. Scaling serverless function memory balloons your monthly cloud bill.

Third, cold starts delay execution. If your worker has been idle, spinning up a heavy compute environment introduces several seconds of latency before processing even begins.

For heavy compute, long-running processes on a dedicated server or VPS beat serverless every single time.

Running multi-minute workers on a $5 VPS

When you host your application on a standard Linux VPS, you are not bound by serverless duration meters.

Your machine runs continuously. A Node.js worker process can run for five minutes, thirty minutes, or three hours without any external gateway terminating its execution.

In BoringPush, this workflow is built into boringqueue:

Step 1: Dispatch from your Next.js route

Your API route receives the upload or payload, dispatches a task to local SQLite, and returns immediately:

import { dispatchJob } from "@/libs/boringqueue";

export async function POST(req) {
  const { fileKey, userId } = await req.json();

  // Instant dispatch: durable in 1ms
  const job = dispatchJob("imports/contacts", { fileKey, userId });

  return Response.json({
    status: "processing",
    jobId: job.id,
  }, { status: 202 });
}

Step 2: Process in a detached worker thread

The worker script lives in scripts/jobs/imports/contacts.mjs.

It runs under Node's native worker thread pool, managed by PM2:

export default async function (payload, { job, signal }) {
  const { fileKey, userId } = payload;

  // Stream and process the heavy CSV file
  const stream = await getStorageStream(fileKey);
  
  for await (const row of parseCsv(stream)) {
    if (signal.aborted) break;
    await insertContact(userId, row);
  }

  // Finished cleanly: mark completed
  return { importedCount: totalRows };
}

Because this worker runs independently of Next.js web requests, it consumes zero HTTP connections.

If the server reboots during processing, boringqueue detects the interrupted task on startup and reclaims it for execution.

Updating the user interface when finished

Once a five-minute job finishes, how do you notify the user without making them refresh the page?

Many developers reach for heavy WebSocket servers like Socket.io or rent third-party services like Pusher or Ably. Both add unnecessary moving parts.

BoringPush uses lightweight Server-Sent Events (SSE) via libs/live.js.

When your worker completes its task, it broadcasts an update to the user's private topic:

import { broadcast } from "@/libs/live.js";

// Inside your worker script when work finishes:
await broadcast(`user:${userId}`, "import_complete", { count: totalRows });

In the browser, the <Live topic="user" /> component listens over standard HTTP/2 and triggers a re-render the instant the broadcast fires.

No open WebSocket socket leaks, no third-party connection bills, and zero polling loops.

The architectural takeaway

Stop fighting HTTP gateway timeouts.

Do not try to hack five-minute tasks into sixty-second serverless handlers. Do not make users wait on white loading screens while your server crunches data.

Accept the request in milliseconds, process the work in a dedicated background worker, and notify the client when the output is ready.

Simpler code, happier users, and zero 504 errors.

FAQ

What happens if an async task uses all my VPS CPU?

In BoringPush, worker threads are configured with bounded concurrency. You can pin background workers to specific thread counts in libs/config.js so background processing never starves incoming web requests of CPU cycles.

Can I run headless Chrome (Puppeteer) in these background workers?

Yes. Because you run on a full Linux VPS rather than a stripped-down serverless environment, you can install Chromium and run automated PDF generation or web scraping tasks natively.

How do I know if a background task got stuck in an infinite loop?

Every job dispatched in boringqueue supports a timeout parameter. If a task exceeds its configured maximum execution duration, the worker pool aborts the job's AbortSignal and flags the task as failed.

How large of a CSV can I process with this pattern?

By streaming the file directly from disk or cloud storage (like Cloudflare R2) row-by-row, memory usage remains flat regardless of whether the file has one hundred rows or one million rows.

What should my frontend show while a long task runs?

Show a subtle processing indicator or progress pill. Because the job ID is returned immediately, the user can continue navigating the dashboard while the background worker finishes.

Keep reading

• Next.js Background Jobs Without Redis: $0 BullMQ — the core queue engine
• Why Next.js Background Tasks Die in Production — understanding serverless limits
• Cron vs. Background Queues in Next.js — choosing the right tool