Nightwatch: what it takes to let an agent fleet run while you sleep
Planning in the afternoon and waking up to a reviewed, gate-passed pull request sounds like a demo. Making it survive eight unattended hours is an exercise in designing for death, and the hard parts are not the ones you expect.
The pitch is easy to say and hard to earn: spend an afternoon shaping work into tickets, go to bed, and wake up to a single reviewed pull request plus a step-by-step plan for testing it. We call the loop nightwatch. It has run real multi-ticket features end to end — one of them produced a merge-ready master PR in about seven and a half hours of wall clock, unattended.
Getting there took several nights of failure, and almost none of the difficulty was in the part that sounds difficult. Prompting agents to write code is basically solved. Keeping eight hours of unsupervised work from quietly evaporating is not.
The structural problem: the human is the bottleneck
The obvious design is one orchestrator that spawns workers. It fails for a boring reason: the orchestrator runs on the laptop, and the laptop's owner goes to sleep. Every worker that finishes a ticket at 02:00 and needs a decision blocks until morning. You get parallel agents with serial throughput.
So nightwatch runs two orchestrators that are deliberately co-equal:
- The Helm — the laptop-side agent. Turns a conversation into a plan, shapes tickets, bootstraps the remote side, then goes dormant when its human does.
- The field orchestrator — a persistent agent on a big remote box. Self-plans its build order within the Helm's constraints, spawns and supervises workers, gates every PR, runs all night.
The keystone is that workers listen to both orchestrators equally. Not a primary with a failover — genuinely co-equal authority. That single property is what lets the laptop go dark without stalling anything. If workers had to escalate to the agent that created them, the whole thing would degrade into a queue waiting on a sleeping human.
The shape in that graph is the argument. Each worker sits between the two orchestrators rather than hanging off one, so either can answer it and neither is a single point of failure.
The highest-leverage thing happens before any agent starts
If we could keep one rule from all of this, it would be this one: every ticket must carry a closed verification loop the worker can run by itself, many times, with no human in it.
Not "build X." Instead:
Build X. Verify with
pnpm tsx scripts/verify-x.ts, which should print Y. Gate to a clean review.
The difference in convergence speed is not subtle. A ticket with a closed loop lets an agent try, observe, and correct on its own for as many rounds as it needs. A ticket without one produces an agent that writes plausible code, cannot tell whether it worked, and either stops early or asks a question into an inbox nobody is reading until morning.
Two loops per ticket, and they answer different questions:
| Loop | Question | Form |
|---|---|---|
| Functional | Does it do the thing? | A unit/integration test, or a purpose-built script the agent runs on demand |
| Non-functional | Is it clean and safe to ship? | The gate — CI plus automated code review |
For work with no natural test surface — sandboxed execution, generated artifacts — the answer is usually a purpose-built harness: a script that spins up the thing and exercises exactly one behaviour. Writing that script is part of the ticket, not overhead on top of it.
There is a cheaper rung worth defending, too. Scripts and component-level stories need no auth, no database, no running product. Full browser verification is the second rung, reserved for tickets that already need the app up. Making screenshot-driven verification the default turns every ticket into an infrastructure problem.
The gate, and why three quarters of it is traps
"Merge-ready" has to mean something a machine can check, because at 03:00 no human is checking. Ours means all of:
- A genuine 5/5 confidence score from the review bot, in the PR body, with the footer's last reviewed commit equal to the current head.
- Every check run green.
- Zero unresolved review threads and zero outstanding comments-outside-diff.
- The body read for blocker, deviation, or deferred flags.
Points 1–3 each encode a night we lost.
A review score can be stale. The bot posts a score, the agent pushes two more commits, and the PR still displays a proud 5/5 that describes code that no longer exists. The footer commit is the only honest field. Compare it to head or you are reading a review of the past.
A conflicting PR silently skips CI. This one is genuinely nasty. When a pull
request goes into a conflicting state, the pull_request workflows never fire —
so the PR is not red, it is thin. The tell is a checks list that is shorter than
usual and entirely green. An automated gate that asks "are all checks passing?"
answers yes with total confidence about a branch that ran nothing.
Review comments arrive on more than one channel. Threads attached to diff lines are the obvious one. Comments outside the diff are a separate channel with separate state, and a gate that counts unresolved threads will cheerfully report zero while real objections sit unread.
Batch re-gating finds what per-component review cannot
When several gated branches need to land together, the temptation is to merge them and ship, on the theory that each was already reviewed. That theory is wrong, and the evidence was unambiguous.
On one run, four component PRs were each individually gated to a clean 5/5 with green CI. We merged them into an integration branch as real merge commits — no squash, so attribution survives — and re-ran the full review on the combined head. It found roughly half a dozen real bugs that all four individual reviews had passed.
They were not sloppiness. They were the class of bug that only exists in the combination: one branch adds a code path, another changes a permission check, and neither diff on its own contains the mistake. Several were access-control gaps visible only once both changes were present. One would have broken a customer-facing demo.
Design for death, because they die
Agents die. Processes get swept, hosts hiccup, a turn ends at the wrong moment. If your design treats death as an exception, every death costs you the work since the last checkpoint.
Three rules, each bought the hard way:
- Push first, always. Commit and push before anything that might not come back. An agent's local worktree is not durable storage.
- Respawn into the existing workspace, never a fresh one. Recreating the workspace resets the worktree and orphans the agent's identity — you lose the work and the ability to talk to whatever replaces it.
- Treat a death event as park-and-replace. Re-dispatch the ticket to a fresh agent rather than trying to resurrect the old one.
One failure mode remains unsolved: an occasional simultaneous die-off of the entire fleet, most likely an external sweep on the host. Push-first recovery keeps it survivable; we still cannot prevent it. Saying so is more useful than pretending the pattern is airtight.
Liveness is not reachability
The roster says an agent is online. That is a claim about a heartbeat, not proof that a message you send will arrive and be acted on. We now trust exactly one thing: a real round-trip.
This matters most at the checkpoint before the human sleeps, because the morning hand-off depends on state that is easy to get silently wrong. Our preflight script checks the mechanical things — credentials active, daemon healthy, remote orchestrator online, filesystem trust pre-seeded — and then does the part a local script fundamentally cannot: it dispatches a request the remote orchestrator must answer, where a correct answer requires the remote side's own credentials to be working.
That one round trip proves two things at once: the message path is real, and the remote agent can actually perform the privileged action it will need at 07:00. No amount of local checking substitutes for it.
The typed, addressed message log doubles as the audit trail. When something goes wrong at 04:00, the question is never "what was the agent thinking" — it is "what did it actually send", and that has an answer.
The failures are boring, and that is the point
A representative sample of what actually cost us nights:
- A headless agent spawn stalling forever on an interactive first-run trust prompt. Nothing in the logs; the agent simply never started.
- Node's default heap being too small for the workload, producing an out-of-memory death that reads as an unexplained disappearance.
- A newer Node's io_uring path hanging under the sandbox's seccomp filter. Every filesystem operation just stopped.
- A worktree that silently forked from the default branch because the host's clone had never fetched the base branch. Everything built cleanly, on top of the wrong commit.
None of these are interesting. All of them are fatal, all are invisible until morning, and all are one mechanical assertion away from being caught in seconds. That is the real lesson: the intelligence in the system is not the scarce resource. The scarce resource is verified state at the boundaries — before you sleep, before you spawn, before you trust a green check.
What the morning looks like
The field orchestrator watches for its human to come back online, then spawns a reviewer agent on the laptop with a pre-written test plan and starts a dev server in its own terminal — deliberately separate, so the server outlives the agent that set it up. The human signs in, walks the plan, and drops pinned-screenshot comments that route straight back to an agent that acts on them live.
That right-hand panel closes the loop. The comment is not filed for later — it becomes a message to an agent that is still awake and still holds the context for the code you are complaining about.
The fleet does not decide whether to merge. It produces something a human can evaluate in twenty minutes instead of reconstructing over a morning. That distinction is the whole design: the goal was never autonomy, it was compressing the distance between an idea in the afternoon and a reviewable artifact at breakfast.
Everything above is the cost of that compression being real rather than a demo — and nearly all of it is bookkeeping, verification, and taking failure seriously. The agents were the easy part.