Background Jobs

Sooner or later your app needs to do things nobody asked for: back up the database at 3am, sweep old logs, clean dead telemetry rows, defragment. The industry answer is Redis + BullMQ + a worker dyno — three new services to host, monitor, and pay for, all to run five scripts. Here's the mentor's shortcut, and I want you to really hear this: you don't have a queue problem. You have five cron entries. This doc shows the system and teaches you to add the sixth job yourself.

The philosophy: the filesystem is the scheduler

Open scripts/workers/ and look at what's there: five folders named minute/, hourly/, daily/, weekly/, monthly/, and one dispatcher, dispatch.mjs. That's the entire job system. To schedule work, you drop an executable script into a folder. There is no queue, no broker, no dashboard, no "worker failed to pick up the job" incident at 4am — because there's nothing to fail except your script and cron itself, both of which log loudly.

The dispatcher (node scripts/workers/dispatch.mjs daily) runs every script in that period's folder, in order, logging each to logs/worker.log with timestamps. If any script exits non-zero, the dispatcher exits 1 with full details — cron mails you (or your alert channel fires) instead of swallowing it. Silent success, noisy failure. That's the whole contract.

The lock that prevents the 4am pileup

Here's the failure this system was built around: the hourly job takes 70 minutes (big database, slow disk), cron fires the next run on schedule, and now two VACUUMs fight over the same file. Locks pile up, disk fills, the server falls over — caused by the maintenance system itself. Ironic, common, awful.

The dispatcher prevents it with an advisory file lock: before running, it atomically creates logs/worker_<period>.lock (the wx flag fails if the file exists — atomic at the kernel level, no race possible). Lock exists? The run skips with a log line instead of stacking. The PID goes inside the lockfile so a human can check whether the holder is actually alive. Stale lock from a killed process? Delete the file and the next run proceeds — one command, documented right in the dispatcher header. No fencing tokens, no Redis Redlock, no distributed consensus for a single VPS that runs one copy of everything.

The five jobs already working your night shift

• Every minute: yours. minute/ ships empty. It's for tiny jobs that must finish well inside 60 seconds (poll for a missed webhook, flush a small outbox); the lock makes an overrunning minute skip instead of pile up.
• Hourly: wal-checkpoint — truncates SQLite WAL sidecars (PRAGMA wal_checkpoint(TRUNCATE)). Without it the -wal file grows forever beside your database; with it, disk stays flat. This is the highest-frequency job because it's the cheapest and the most forgotten.
• Daily: backup-sqlite — atomic snapshot via VACUUM INTO into db/backups/. Atomic means readers never see a half-written backup; the previous section's lock means two backups never fight. Your disaster recovery is a file copy away, every morning, without you thinking about it.
• Daily: analytics-cleanup — purges telemetry older than 90 days. Analytics tables are write-only appendages that grow forever; unbounded growth is how a $5 disk fills in month eleven. Ninety days is plenty for trend charts and mercy for the disk.
• Weekly: log-rotate — rotates logs/ so deploys, workers, and request logs don't eat the volume. Logs are for reading this week, not archiving forever.
• Monthly: vacuum-databases — full defragmentation and query-planner optimization. The deep clean: reclaims scattered pages, refreshes statistics, keeps month-twelve queries as fast as week-one queries.

Notice the rhythm: cheap + frequent for growth control, expensive + rare for deep maintenance. When you add jobs, match that rhythm — hourly things must finish in minutes, monthly things may take the night.

Add your job in five minutes (do it with me)

Say you want a daily digest email of new signups. Here's the whole ceremony:

1. Write the script at scripts/workers/daily/signup-digest.mjs. Shebang #!/usr/bin/env node, exit non-zero on failure (the dispatcher treats that as an alarm), log progress with plain console.log (the dispatcher timestamps it into worker.log). Keep it under ~60 lines — if it's bigger, it's a feature with a script attached, not a job. Split it.
2. Make it executable: chmod +x scripts/workers/daily/signup-digest.mjs. Cron runs it directly; non-executable means silently skipped. Check this first when a new job "doesn't run."
3. Test it by hand: node scripts/workers/dispatch.mjs daily. Watch your script run inline with the others, check the exit code, check logs/worker.log. Never install an untested job into cron — the scheduler is not your debugger.
4. Wire the schedule (one line in crontab): the repo's crontab installer covers the five periods, and every bp deploy adds any period line your server is missing (it only adds, never edits yours; SKIP_CRON=1 opts out); your script rides the daily run automatically. No registration step, no config file — presence in the folder is the registration.

That's it. Four steps, no dashboard, no deploy of worker infra. The next founder who reads your repo finds the job exactly where the rhythm says it should be.

What this system is (and what it isn't)

This folder-based cron system is built for scheduled calendar maintenance: hourly WAL checkpoints, nightly database snapshots, weekly log rotations.

It is not an event queue:

• It doesn't trigger immediately on user actions (it runs on its folder rhythm).
• It doesn't retry with exponential backoff (failed runs wait for the next period).

Need immediate event-driven tasks?
When someone buys, signs up, or triggers a 45-second AI workflow, you need work dispatched instantly with automatic retries if an external API flaked. You don't need to install Redis or pay for Inngest or Upstash for that — BoringPush includes a built-in event-driven background worker engine right inside the app. See 19. Event-Driven Background Workers.

For scheduled calendar maintenance, five folders and one dispatcher do the job with zero bills and zero 4am pages. That's the deal.

Verify the night crew (thirty seconds, daytime)

node scripts/workers/dispatch.mjs daily   # runs today's jobs now, watch the log
tail -n 20 logs/worker.log                # every run timestamped per period
ls -la logs/worker_*.lock                 # no stale locks lying around

If the log shows all five jobs green and no lock files linger, your nights are covered. Go build features — the crew has the dark.