# Don't Ship It. Prove It First.

> Agent claims need evidence: gate output, curled endpoints, adversary tests. The prove-it ritual that ends blind merges. Inside.

Source: https://boringpush.com/blog/prove-it-not-ship-it-agent-verification

## Merging a paragraph, not a proof

"Implemented rate limiting — tested, working." Merged on the sentence. Production taught the lesson the sentence skipped: no test existed, the limiter counted per-process (useless in cluster), and the first scraper proved it within hours.

The transcript had confidence. It lacked the one thing confidence can't replace: output of commands run against reality. Literature had been accepted as engineering.

Don't ship it. Prove it first. Four words that would have saved that week — now taped above every merge this team makes.

## Claims vs evidence at merge time

| Merge basis | What you know | Incident rate |
|---|---|---|
| Agent paragraph | They believe it works | High (belief ≠ truth) |
| Pasted gate output | Syntax + standards hold | Medium (form only) |
| Gate + curls | Paths behave live | Low |
| Gate + curls + adversary | Forbidden fails too | Near zero |
| Re-run by you/CI | Independently true | Lowest measurable |

Three things transcripts teach the hard way:

1. **Confidence is a style, not a signal.** Models write "definitely fixed" and "might be fixed" with equal fluency — tone carries zero information about correctness. Merges keyed to confident language approve fiction at the same rate as fact. Read claims as weather (atmosphere), proof as ground (footing).
2. **Untested paths are guilty until proven innocent.** The rate limiter failed on the exact path nobody curled (multi-process reality vs single-process tests). Every unexercised path is a suspect; merges are verdicts. No curl, no verdict — the case stays open and the suspect ships free.
3. **Re-running beats re-reading.** Re-reading that transcript twice convinces both times; re-running its claims once catches it in seconds. Reading evaluates rhetoric; running evaluates reality. Thirty seconds of commands outperforms thirty minutes of careful reading, every single merge, forever.

The done-but-500 post showed the corpse. This one runs the autopsy as policy.

## Ai verification gate ritual

Claim, evidence, independent re-run — the three-part merge requirement in the lock file.

**Step 1 — Demand the proof bundle with every handoff.**

```md agent handoff template (required, no exceptions)
CLAIM: rate limiting enforced per user across cluster.
PROOF:
  gate: [pasted: node --check + lint, exit 0]
  happy: [pasted: curl 200 with normal traffic]
  unhappy: [pasted: curl 429 past limit from two workers]
  adversary: [pasted: spoofed user id → still limited]
```

Four pastes, no prose accepted as substitute. The template converts "tested, working" from literature into exhibits — numbered, timestamped, re-runnable. Agents adapt within sessions (proof becomes reflex when merges depend on it); transcripts get shorter and truer simultaneously.

**Step 2 — Re-run the bundle yourself (or let CI).**

```bash
# your terminal, their branch, two minutes:
npm run lint && curl <their happy> && curl <their unhappy> && curl /api/health
# match? merge. mismatch? the transcript lied — reject regardless of prose quality
```

Re-running is the whole game: pasted output proves the agent ran commands once (in some environment, at some time); your run proves they pass here, now. Fabrication dies at re-run (the 13-agent lesson); optimism dies too (stale green from pre-change code). Trust the re-run, file the transcript as supporting color.

**Step 3 — Adversary-test the forbidden per PR.**

```bash
# one new test per PR asserting what must NOT happen:
# unpaid export → 403. user B reads user A → null. tampered price → product-row wins.
# name it ADVERSARY_<threat>.test.js — visible, countable, non-skippable in review
```

Adversary tests are the only tests that prove safety rather than function — everything else asserts the happy path the author already believes. One per PR compounds into a security suite written incident-free (each test names a disaster that never happened because the test exists). Green suites prove features; adversary suites prove sleep.

That's the architecture talking — merges backed by exhibits.

## The merge ledger, literature vs exhibits

| Quarter of merges | Paragraph accepted | Proof bundle required |
|---|---|---|
| Post-merge incidents | 5 (limiter, webhooks, paywall) | 0 |
| Fabricated proofs caught | 0 (never checked) | 2 (re-run mismatches) |
| Avg merge delay | 0 min | ~8 min (proof + re-run) |
| Adversary suite size | 0 tests | ~35 and growing |
| Weekend pages | 3 | 0 |

The real cost, stated: eight minutes per merge plus the social friction of rejecting confident prose ("but it said tested!"). Priced against five incidents and three ruined weekends: proof costs minutes, blind trust costs evenings. Merges are the narrow gate everything flows through — guard the gate and the codebase guards itself. Exhibit A: uninterrupted sleep.

## FAQ

### Is ai verification gate just CI with extra steps?

CI is half of it (machine re-runs, tireless, dumb). The gate adds human judgment at the right points: designing adversary cases (what must fail), reading proof bundles for completeness (what's missing), and holding the merge pen (accountability lives somewhere). CI automates verification; the gate ritualizes responsibility. Automation without ritual produces green pipelines shipping guesses.

### What if the agent genuinely can't produce proof (flaky env, missing data)?

Then the work isn't mergeable yet — say so explicitly ("blocked: needs staging webhook replay") instead of accepting literature. Blocked-with-reason beats merged-on-hope every time; the branch waits cheaply while prod stays safe. Pressure to merge unproven work is how incidents get scheduled. Name the blocker, calendar the unblock, protect main ruthlessly.

### Should small docs/copy PRs need the full bundle?

Scaled, never zero: docs get lint plus link-check (two minutes), copy gets preview screenshot (one minute). The principle (evidence over claims) scales down to a breath; only the ceremony shrinks. Skipped-small-PRs teach the pipeline that some merges need no proof — and pipelines generalize exceptions faster than rules. Small proof, always.

### How do I review proof bundles without becoming the bottleneck?

Template literacy: reviewers scan four pastes in two minutes (gate exit, happy code, unhappy code, adversary result) versus twenty minutes reading prose. Rejections get faster too (missing paste = instant bounce, no deliberation). Standardized proof compresses review; freeform claims expand it. Bottlenecks come from reading literature, not checking exhibits.

### Can proof requirements coexist with fast-moving teams?

They're how fast teams stay fast: unproven merges create incident debt that compounds into velocity-killing firefighting. Teams measuring merge-to-incident (not just merge speed) find proof-gated throughput higher within a quarter. Speed is features shipped per incident, not per day. Gates raise the denominator that matters.

### What about agent-to-agent handoffs (no human in loop)?

Same bundle, machine-checked: downstream agent re-runs upstream's pasted commands before building on them (verify, then extend). Chains of unverified agent work amplify hallucinations (the subagents post shows the math); verified chains contain them per link. Every handoff — human or silicon — carries exhibits or it doesn't happen. Trust is per-link, never transitive.

## Keep reading

- [Agent Said Done. Checkout 500s. (The 10-Second Check)](/blog/ai-agent-says-done-but-500-fix) — the corpse this gate autopsies
- [Mobile AI Guardrails: Couch Fixes That Can't Break Prod](/blog/mobile-session-guard-zero-breakage) — the gate at phone scale
- [AGENTS.md Is a Wish List Until You Add Teeth](/blog/agents-md-vs-conventions) — the file holding this ritual