I put Temporal around my software factory. Here’s what it fixed, and what it didn’t.

What I learned adding durable execution to a fleet of coding agents built on Gas City

A durable work record is not the same thing as a durable procedure.

Cabo and Beads

In the fall of 2025, I sat next to Steve Yegge at dinner during a Sourcegraph offsite in Cabo.

Steve had brought his laptop to the table to continue working with his agents (I was just running mine from my phone instead, so I understood the impulse). Terminals running agents were scattered across his screen in a rainbow of different colors, and he kept moving between them while everyone else was doing more normal dinner activities.

I asked what he was building, and I remember the answer being something like, “Something that’s going to be big.”

A few weeks later, in October 2025, he published Introducing Beads, a task and dependency system designed around how coding agents actually work. So the thing on the laptop at dinner really did become big. It was Beads.

Typically, a new coding-agent context begins without episodic memory of the session before it. The earlier session may have investigated the codebase, discovered hidden dependencies, ruled out two approaches, found a broken test, and developed a reasonable plan. Unless the system restores that history, the next context begins by reconstructing it.

Beads gave me a dependency-aware work record that survives those boundaries. An agent can record what it discovered, connect related work, mark blockers, and resume without rebuilding the project from a transcript.

On January 1, 2026, Steve published Welcome to Gas Town: an opinionated coding-agent factory built on Beads, with persistent agent identities, bounded sessions, reusable workflows called formulas, coordination through mail and nudges, and a Mayor agent, the top-level coordinating agent and human handoff point.

It also came with Polecats, a Deacon, a Witness, a Refinery, Rigs, Convoys, Molecules, Wisps, and other abstractions, that while fun and flavorful, I struggled a bit to keep straight in my head.

I forked Gas Town in early January with lofty goals of making it more vendor-agnostic, and briefly mentioned this to Steve (to which he responded “send PRs!”). There had to be a better way to manage my projects than flitting between 10+ tabs in Ghostty, and that was the promise of Gas Town. But I never quite got my first town working the way I wanted, and I had a critical project at the time (CodeScaleBench) that couldn’t afford to stall on dropped queues and other random orchestrator nonsense, so I reluctantly abandoned my ghost town. I could run a bunch of coding agents; I didn’t know how to keep a town of them from catching fire (or, more accurately, from agents sitting around doing mostly nothing while passing around poorly translated tasks in a game of Agent Telephone).

Many of these issues were in theory resolved by a philosophy Steve called Nondeterministic Idempotence, or NDI: if a session ends, a later session, potentially continuing the same persistent role, can inspect the durable work record and choose another valid route to the same outcome. NDI is one of the two kinds of recovery this article is about. But before I really understood it, I had to piece it together through a deconstructed version of the system.

In March 2026, before its public launch, I found that deconstructed version of the system in Gas City. Gas City, created by Julian Knutsen and Chris Sells, took the primitives behind Gas Town and turned them into a composable orchestration SDK. Gas Town was one specific city design. Gas City gave you the building materials to create your own. That separation immediately made sense to me, and I wanted it (and by extension my own software factory) to succeed. I began contributing heavily in early April, and Julian invited me and a fellow avid contributor to become Gas City’s first outside maintainers. I’ve since authored over two hundred pull requests across session lifecycle, reconciliation, Beads integration, formulas, dispatch, the API, repair controls, and external messaging, plus a Slack integration and most of the formative commits to its revamped dashboard.

As a Gas City maintainer, I use the city to build and repair Gas City itself. That includes the queueing, dispatch, recovery, and orchestration paths I help maintain. It is a useful (if slightly hostile) test environment because every bug in the machinery eventually becomes a bug in the system trying to fix the machinery, which means I experience the pain quickly and become motivated to address it.

That work taught me how a city actually stays humming (ish), patch by patch by patch. A claim goes stale, so I add a stale-claim reaper. A workflow root stays open after its steps finish, so I add an orphaned-molecule reaper. Sessions wedge on a provider error, so I add a scanner that checks every session every two minutes. Each fix ends up small, reasonable, and correct about the failure directly in front of it.

By mid-July, my orders directory held roughly 100 of these patrol and repair jobs, firing about 850 times an hour. When I later had my agents audit the city’s own reliability, they flagged the cadence itself: the repair jobs were consuming the machine they were supposed to defend, oops.

The failures we wound up chasing were fairly consistent.

One night, the store file grew to 168 MB: 79,210 beads, 96 percent of them already closed. Because every dispatch reparsed the entire file under a lock, order firing froze. One wedged session claimed the same bead ten times over 5.6 hours. A maintenance order sat dormant for ten days before anyone noticed, because the thing responsible for checking whether orders fired was itself an order.

Another artifact from this era was a recovery flag written by one workflow so that “a reaper” could query it. But no reaper ever did. The session-recovery scanner ran every two minutes for two months, around forty thousand scans, and detected zero instances of the condition it existed to find.

Meanwhile, the actual repairs fell to the Mayor, who really is supposed to have better things to do than track down city infra issues. Their prompt slowly became a field manual for infrastructure archaeology: drain duplicate sessions every cycle, classify the stalled-delivery fault, bypass the broken delivery path, never let a dead nudge sit ahead of a P0. It got bad enough that I chartered a dedicated infrastructure agent to be the Tier-1 first responder, so the Mayor would stop being the single point of contact for mechanical breakage.

Each new reaper fixed one failure mode. It also spread ownership of the procedure across another process. The patrols checked liveness: is the process still there? But the failures were about outcomes. Did this claim launch exactly one agent? Does this result belong to the current generation? Who owns the next transition?

And every layer of protection was itself an order, watched byyyy… nothing.

I really wanted Gas City to just work. I did not want to spend my time doing detective work every time something stopped moving, poking at agents to determine why they weren’t working or chasing down reasons why a second agent started editing an already-claimed branch.

The task survived. The procedure did not.

The problem became a bit clearer through one recurring failure. A work item becomes ready. The Mayor claims it and launches a coding agent. The agent gets a worktree, edits files, runs tests, and works for an hour. Then the Mayor’s session abruptly ends before recording the handoff.

After restart, the system basically has three bad options. It can launch another agent and leave two processes editing the same worktree. It can wait indefinitely for a process it no longer knows how to contact. Or it can release the claim and ask a repair loop to reconstruct what happened from whatever evidence the first process left behind. At different points, I ran versions of all three.

The procedure existed only as clues scattered across controller ticks, session records, mail, hooks, process state, and repair logic.

At enough scale, those clues become useless. I saw stranded claims, duplicate sessions after retries, stale completions arriving after a newer generation had started, lost notifications, and correlation metadata pointing at executions that did not exist.

The work record survived. What the system lost was the procedure connecting one state to the next.

Two kinds of recovery

So how do we handle work transition and continuity in the face of inevitable failure and nondeterminism? We need to figure out where to draw the boundaries.

NDI is one answer, and it is a good one where any reasonable path to an acceptable outcome is valid, which describes a lot of agent work. An agent can inspect a bug, choose a repair, run tests, realize it was wrong, backtrack, and try something else. I do not need or want a system that has to replay every thought and tool call. Coding itself is nondeterministic. Two agents can solve the same issue in very different ways, and either result may be acceptable (unless you’re a SWE-Bench verifier).

But some parts of an orchestration system are promises rather than work:

• Did this claim already start an agent?
• Does this result belong to the current generation?
• Has this exact outcome been acknowledged?
• Is a retry resuming the same execution or creating a competing one?
• Which recovery action owns the next transition?
• Should a late completion still be allowed to mutate the work record?

Those are procedural questions, and “a capable agent will inspect the clues and figure something out” is not the guarantee I want around them. The flaw was asking an NDI-style recovery model to own guarantees whose ordering, retries, waits, identities, and acknowledgements were themselves the contract. For those parts, I need more determinism.

By this point, my research on coding-agent failures had already convinced me that reliability depends as much on the system around the agent as on the agent itself. This was another instance of the same problem, so I went looking for systems that had solved adjacent parts of it.

Gas Town can kinda be described as “Kubernetes for agents,” so I started with Kubernetes controller patterns. I picked up Designing Distributed Systems and read it against Gas City’s controller: reconciliation loops, leases, transactional outboxes, stable identities, generation fencing, explicit acknowledgements (I also photographed pages and sent them to my agents to brainstorm with me). Kubernetes can see that a process disappeared and start another one; it does not know whether a claim already launched this particular agent session, whether a late result belongs to the current generation, or whether a human acknowledged the exact outcome the system is waiting for. The distributed-systems toolbox supplied necessary pieces, but no durable owner for the sequence connecting them.

Then I remembered Steve saying Gas Town looked like what you’d get if Kubernetes and Temporal had a very ugly baby together, so there was some additional heritage to look into. He had described NDI as similar to Temporal’s deterministic durable replay, implemented through completely different machinery, and he was explicit that Gas Town was not a replacement for Temporal.

Temporal’s model works like this. A Workflow records a deterministic procedure in an Event History. When a Worker, the process executing Workflow code, crashes, another Worker can replay that history, reconstruct the Workflow’s state, and continue from the recorded boundary. External and nondeterministic effects run in Activities.

That gave me two recovery models to consider side by side. NDI: a new agent session may reach an acceptable outcome by a different route than the one that was interrupted. Temporal: If the system becomes interrupted, you can pick up exactly where you left off.

Recovery is not the same thing as continuity

The two models are complementary, because there isn’t a uniform answer to what should survive when an interruption occurs. You need to be able to answer when does it matter that the steps start exactly from where they were, vs. reconstructing the next steps using judgement from available relevant information.

A durable work record preserves what was learned and what remains to be done. A durable procedure preserves the system’s outstanding promises. Neither one, by itself, defines whether the next model context is a continuation of the same agent role or a replacement assigned to the same task.

That identity belongs outside Temporal. In my system, Gas City owns the persistent agent role and Beads preserves its work history. Temporal preserves the obligation to return to the correct session when it still exists, or to make an explicit recovery decision when it does not.

This changes how I describe start-or-attach: it is continuity-preserving recovery, not just duplicate prevention. The infrastructure first asks whether the session that owns the work is still available. It creates a replacement only when the system has deliberately decided that continuity is no longer possible or desirable. Context, attribution, and ownership are what make the handoff clean and the system legible.

So, could the setup I run with Gas City use both models at once?

My agents voted against Temporal almost everywhere at first

My agents did not really seem to “get” Temporal at first, I had to do a lot of back and forth with them, because 1) in some cases it’s a matter of taste and requirements, and 2) they didn’t initially have a good internalized model for what Temporal was useful for.

My infrastructure agent built the city’s first Temporal pilot around a maintenance job that runs for forty-four seconds every two hours. We found that Temporal added little beyond cron and a lockfile for that job, and a Bash script still caught the failures it missed. The pilot showed me where not to use it.

I tested other boundaries and rejected many of them: the recurring-job scheduler, the formula engine, the dispatchers, and the watchdog. A systemd timer beat Temporal for the watchdog on every measure I took.

There was a very promising boundary though, the work from claim through agent session to acknowledged result. If the process died mid-flight, Beads still held the facts about the work, but the system lost what connected them, including what had happened, what was still waiting, and what should happen next. My new goal was to see whether Temporal would preserve that procedure so the city can recover it after a crash.

The boundary I chose

While trying to figure out where Temporal could help with my numerous problems, I wound up forced to answer ownership questions that were previously fuzzy. I first started with the procedure around the agents.

Put the unpredictable agent inside an Activity. Put the procedure around it in a Workflow.

The integration settled on three owners.

Beads owns the work state, including tasks, dependencies, claims, generations, artifacts, outcomes, and acknowledgement receipts. It remains the canonical record of mutable work facts. Temporal does not replace it. A user or agent can inspect the work without going through Temporal.

Temporal owns the procedure. It records which orchestration steps completed, which retry is active, what the system is waiting for, and what may safely happen next. That includes waits, retries, cancellation, signals, and acknowledgement progress.

The application owns the effects, including agent processes, tests, reviews, database calls, and external messages. An Activity is where Temporal hands control back to my code. Temporal schedules it, records its heartbeats, and decides how to recover if the Worker disappears. What the Activity touches, and whether touching it twice is safe, remains my responsibility.

To help illustrate these boundaries, you can think of Gas City as a workshop. Beads is the job board: it records the work, who owns it, what depends on what, and what happened. Coding agents are the workers. Their path through a job is whatever they decide is best given the task description and available information. Temporal records the procedure around them: which steps completed, which one is waiting, and which safe retry comes next after the inevitable mishap.

Those are three separate jobs. Beads holds the work. The agents do it their own way. Temporal remembers and keeps tabs on the running checklist.

Now we get into some of the distributed-systems weeds. Notably, stable identities, idempotency keys, and generation fences are not Temporal features. They remain application responsibilities, and I would need them under any architecture.

The two layers solve different problems. A fence decides whether a write is allowed: whether this agent still owns the claim, whether this result belongs to the current attempt, and whether an operation has already been applied. Temporal preserves the procedure around those writes. It remembers that a delivery is still owed, that this is retry three of five, or that the system is waiting for an acknowledgement a human may take a day to provide.

The fences prevent stale or duplicate effects. Temporal preserves the in-flight state my crashed processes kept taking with them.

So why not store the whole procedure in the database I already run? That is the standard controller answer: “a query, not a memory.” Reconstruct the current state from the store each time the controller runs. Almost everywhere, it is simpler and it is enough.

It fails when the store cannot tell the difference between two procedures that leave behind the same records.

One example was a twenty-hour stranding where two work items finished and pushed their branches, but the worker died before recording completion or the next step. The proof of the work was sitting on origin; the store held an open item with an empty next-step field, which is exactly what work that never started looks like. Both watchers keyed on that field: the probe that decides whether to launch more agents, and the sweep that hunts abandoned claims. Neither saw anything wrong.

Solving this entirely in the Beads work store would mean recording every intermediate step, recovering partial transitions after a crash, and keeping timers alive across process failures. I had already seen what a homegrown version looked like. One PR-state poller grew to 243 lines of Bash, used two duplicate-prevention mechanisms, and created 174 cache files to track eight open pull requests. It still had a fire-and-forget failure that could not be fixed without turning the poller into something much larger.

In theory, I could build a custom workflow layer. But then I would own another reliability system: its recovery rules, failure modes, monitoring, and debugging. My goal is to make the city easier to operate, not to give it another piece of infrastructure it must eventually repair.

More importantly, that layer would not give me the two Temporal capabilities that made adoption worthwhile. Temporal records a history that I can replay against new code, so a change that would break a procedure already in flight fails in testing rather than production. Its separation between Workflows and Activities also forces every nondeterministic operation out of Workflow history. My reconcile loop imposed no such boundary.

The roughly one hundred patrol jobs running 850 times an hour do not, by themselves, prove that the city needed Temporal. They show that the system had accumulated repairs one incident at a time, without anything responsible for carrying a procedure through failure.

Once those repairs had become an implicit workflow system, I preferred adopting a durable one over continuing to build it incident by incident.

The three failures I wanted Temporal to stop

The implementation focused on three concrete breakpoints. The first two run in shadow, blocked from touching production work; the third runs continuously within a scoped Outcome canary in my live installation.

1. A crash between claiming work and launching the agent

The failure. The system could record that work was ready and then die while trying to deliver the event that launched its execution. The naive fix is a retry: if the launch never happened, send the event again. That is safe only if the second send cannot start a second execution, and nothing in the old path guaranteed that.

What Temporal contributes. The ready transition and its delivery event are written together in one Beads transaction. A bridge reads the pending event and uses Temporal’s Signal-With-Start operation with a stable Workflow ID derived from the city, run, and work item. If a Workflow with that ID is already running, the event is signaled to it. Otherwise, Temporal starts it and delivers the signal as part of the same operation.

What stays the application’s job. There is still an unavoidable crash window: Temporal can accept the event, then the bridge can die before acknowledging the outbox record. The fix is not pretending that window disappeared. The event is delivered again, and the Workflow ignores an event ID it has already processed (so, deterministic idempotency? I guess we just call that idempotency?). Stable identity turns redelivery from a dangerous anomaly into normal, expected behavior. That guarantee is bounded by how long Temporal retains closed executions: past the retention window the ID can be reused, which is why the generation fence and the canonical work identity, not the Workflow ID, are what keep a late duplicate out.

How it’s tested. A test against a real Temporal dev server delivers the same ready event twice and requires exactly one scheduled Activity. The bounded canary exercised the path end to end, with a scripted stand-in answering the agent side of the adapter protocol.

2. A retry accidentally starts a second agent

The failure. Temporal Activities may execute more than once. A Worker can die after an external effect succeeds but before Temporal records its completion. That means “Temporal will retry it” is not, by itself, safe for starting coding agents: a naive retry could launch a second agent against the same work.

What Temporal contributes. Retry scheduling, heartbeats, and cancellation delivery, all recorded durably.

What stays the application’s job. The Activity claims one exact work generation and receives a fencing token. A trusted adapter uses that token to resolve the agent session: it either starts the session once or returns the session already bound to that claim. What keeps a retry from becoming a second agent is not the heartbeat. A Worker can die before the first heartbeat ever lands. The retry presents the same claim token, and the resolver finds the session already bound to that claim by its stable identity, so there is no competitor to launch. (More on what stands behind that resolver below.) The persisted heartbeat details are how the new attempt picks up progress from the last checkpoint, not how it avoids starting a second agent. Completion also goes through the fence: Beads accepts a receipt only when the work ID, generation, claim token, and the claiming Workflow identity still match. An agent handed a stale task can complete successfully and still have the stale output rejected, because it no longer owns the right to mutate the current work generation.

How it’s tested. The Worker-kill recording below, plus the test pair that kills the Worker before and after the first checkpoint.

3. A completed result is never acknowledged

The failure. This was the failure I got wrong first. The initial canary appeared to work: the Workflow completed, and the agent produced a valid result. Buuuuut, the Mayor never learned that the result existed.

I had treated mail and session nudges as if successful delivery meant successful receipt (nope). A notification can wake an agent, but it cannot prove that the agent reviewed and acknowledged a specific result.

What Temporal contributes. Each verified result now creates an OutcomeReady The durable record that a finished result is owed to someone. It stays open, and delivery repeats, until an acknowledgement matching that exact outcome closes it. outbox record and a dedicated Workflow. The Workflow sends a notification, waits durably for acknowledgement, and redelivers while the result remains pending.

What remains the application’s job. The Workflow closes only when it receives an acknowledgement bound to the exact store, work item, outcome, generation, and session fence. A queued notification or accepted signal proves only that delivery was attempted. The canonical acknowledgement receipt proves that the intended agent accepted the intended result.

How it’s tested. This path failed its first canary, was repaired, and now runs continuously within a scoped Outcome canary. That continuous behavior is not the same as promotion to enabled.

The implementation underneath the boundary

The procedure I made durable is one unit: take one ready work item, assign it to one agent session, and accept one generation-fenced receipt. Its predecessor is public code that maps one to one onto its replacement, so you can read the before and the after side by side.

Before Temporal, that procedure had no single owner. Most of it lived in CityRuntime.beadReconcileTick The old orchestrator's loop step: every few seconds it woke (one tick), scanned the store for ready work, claimed it, and launched agents. Everything it knew between steps lived in process memory. , with session recovery and completion handled elsewhere. A single controller tick could load state, release orphaned assignments, reconcile active sessions, dispatch notifications, and run a secondary wake-up check. A separate close path searched for work abandoned by dead sessions.

Each part was reasonable in isolation. Together they formed one procedure spread across multiple code paths and persistence boundaries. Every arrow between them was a place where the process could die after performing an action but before recording what had happened.

func (cr *CityRuntime) beadReconcileTick(
	ctx context.Context,
	result DesiredStateResult,
	sessionBeads *sessionBeadSnapshot,
	trace *sessionReconcilerTraceCycle,
	bootReconcile bool,
) {
	desiredState := result.State
	store := cr.cityBeadStore()

	released := releaseOrphanedPoolAssignmentsWhenSnapshotsComplete(
		store,
		cr.cfg,
		cr.cityPath,
		sessionBeads.OpenInfos(),
		result,
		rigStores,
	)

	reconcileSessionBeadsTracedWithNamedDemand(
		ctx,
		cr.cityPath,
		sessionBeads.OpenForReconcile(),
		sessionBeads,

	dispatchSessionBeads, err := loadSessionBeadSnapshot(sessStore.Store)
	if err == nil {
		_ = dispatchReadyWaitNudgesWithSnapshot(
			cr.cityPath,
			cr.cfg,
			sessionpkg.NewStore(sessStore),

	// A separate backstop wakes live sessions that never claimed their bead.
	if stalledPoolBeads, err := loadSessionBeads(sessStore.Store); err == nil {
		nudgeStalledPoolClaims(
			cr.sp,
			cr.cfg,
			sessStore,

After, BeadOrchestrationWorkflow The Temporal Workflow that now owns one work item's procedure: claim, launch, wait, receipt, in that order, recorded durably. owns the procedure. It deduplicates delivered events and schedules exactly one Activity per ready event, with a stable Activity identity and a bounded retry policy:

	if _, exists := r.seen[event.EventID]; exists {
		return nil
	}

	activityCtx := workflow.WithActivityOptions(r.activityCtx, workflow.ActivityOptions{
		ActivityID:          activityID,
		TaskQueue:           AgentTaskQueue,
		StartToCloseTimeout: 24 * time.Hour,
		HeartbeatTimeout:    r.input.HeartbeatTimeout,
		WaitForCancellation: true,
		RetryPolicy: &temporal.RetryPolicy{
			InitialInterval:    time.Second,
			BackoffCoefficient: 2,
			MaximumInterval:    time.Minute,
			MaximumAttempts:    5,
		},
	})
	future := workflow.ExecuteActivity(
		activityCtx,
		ExecuteBeadActivityName,
		ActivityInput{Event: event},
	)

And ExecuteBeadActivity The Temporal Activity that touches the real world for that item: it writes the fenced claim, starts the agent or finds the one already running, heartbeats, and reports the completion. owns the fenced claim and the start-or-attach decision. On a retry that has a checkpoint, it reattaches to the recorded session; with no checkpoint, it asks the resolver which session already belongs to this claim:

func (w *ActivityWorker) runAgent(
	ctx context.Context,
	event ReadyEvent,
	lease ClaimLease,
	resume *HeartbeatCheckpoint,
) (agentExecution, error) {
	request := AgentExecutionRequest{
		Event: event, ClaimToken: lease.Token, ResumeFrom: cloneCheckpoint(resume),
	}
	if resume != nil {
		request.SessionID = resume.SessionID
	} else {
		sessionID, err := w.Agent.ResolveSession(ctx, request)
		if err != nil {
			return agentExecution{}, fmt.Errorf("resolve agent session: %w", err)
		}
		if err := validateSegment("session id", sessionID); err != nil {
			return agentExecution{}, temporal.NewNonRetryableApplicationError(
				"agent resolved an invalid session identity", "InvalidAgentSession", err)
		}
		request.SessionID = sessionID
	}

The complete files, each pinned to a reviewed revision and annotated section by section, are in the annotated source reader, alongside the pre-Temporal controller and recovery paths.

The resolver: the part you have to build

The start-or-attach decision depends on one interface that Temporal cannot provide: ResolveSession The adapter that answers, on a cold attempt with no heartbeat checkpoint, which existing session already owns this claim. It is a call into a separate trusted executable, not a Temporal API. .

ResolveSession calls a separate trusted executable using a newline-delimited JSON protocol. The Activity sends the ready event, claim token, and generation. The resolver must return exactly one session identity.

The resolver runs only on a cold attempt. Once an Activity has recorded a session identity in its heartbeat checkpoint, retries reuse that identity and never call the resolver again. A cold attempt has no checkpoint, so the resolver must determine which existing session owns the claim from state that survived the crash.

This adapter is specific to your agent runtime because it connects the claim fence to the system that manages sessions. In Gas City, it will look up the tmux session associated with the claim, along with its worktree and process. Another runtime may store that information somewhere else.

This remains an open item in my integration. The client side is implemented and tested: it requires an absolute executable path, does not invoke a shell, sends payloads over standard input, and accepts exactly one terminal response. So far, only test fixtures have implemented the resolver contract for the harness and canary. That is why agent mutation remains in shadow mode.

The contract also leaves one policy decision to you: what should happen when the session record exists but its process is dead? No framework can answer that for your runtime. Decide before the first crash forces the choice.

What the tests hold down

The full test suite runs with one command:

cd services/temporal-maintenance && go test -race ./...

It protects five invariants:

1. Duplicate delivery of the same ready event schedules one Activity, not two.
2. A retry attaches to the existing agent session instead of launching a competing session.
3. Stale completions and acknowledgements fail closed.
4. Cancellation reaches the session that owns the claim without leaving the Workflow blocked.
5. Captured Workflow histories continue to replay, while a deliberately nondeterministic Workflow fails the replay gate.

The failing replay test matters more than the passing one. The planted nondeterminism gives the gate something it must reject, which verifies that the gate is exercising the code path it claims to protect. Replay fixtures are therefore never regenerated merely to make a change pass. If a captured history stops replaying, the change is breaking until someone can explain why it is safe.

The replay gate has not yet caught a real nondeterministic change. Its only failures have come from the planted control. The two real defects I have found in this area were caught during code review, including recovery signals that could have been stranded after a history rollover.

When a Workflow definition must change while histories are still open, workflow.GetVersion Temporal's SDK call for versioning a Workflow definition safely: it records which code path a given point in history took, so a definition can change while some histories are still open without breaking replay of the ones already in flight. carries the compatibility patch inside those histories. Both safeguards matter: review catches problems that require understanding the recovery path, while replay testing catches incompatible changes that make it through review.

The complete test matrix lives with the annotated source.

Tests define the invariants. The failure harness in the repository shows what happens when the process actually dies. It runs the real Workflow and Activity against a local Temporal server, then sends an operating-system kill signal to the orchestration Worker at two points:

• After the agent is running and a heartbeat checkpoint exists.
• Before the first heartbeat has been recorded.

The harness kills the orchestration Worker, not the coding agent. The agent continues working after its coordinator disappears. The test is whether the retried orchestration attaches to that same agent or launches a second one.

The process pane on the right answers that question with two counters it measures for itself, once a second. agent procs counts operating-system processes and reads 2 during each retry window, when the surviving agent and the retry’s new process run side by side. sessions counts distinct session identities that have written to the work item, and it stays at 1. Two processes over one session is the fix working. The before-state recording shows the failure in the same two counters: there, both read 2.

Both runs end with one agent session and one accepted receipt.

In the first run, the retry recovers the session identity from the heartbeat checkpoint. In the second, no checkpoint exists, so the retried Activity calls the resolver again. The resolver must return the session that already owns the claim rather than create another one.

The second run is the direct test of continuity across a Worker failure. The orchestration process can be replaced, but the agent session already doing the work should not be replaced with it. It is also the only run that tests the resolver. The first resumes from its checkpoint and never calls the resolver again. A resolver that creates a new session on every call would therefore leave the first run green and fail only the second.

The demo runs on one host against a local Temporal development server, with a fixture process standing in for the coding agent. It uses a real Temporal server, a real operating-system kill signal, and a generation-fenced claim. That is sufficient for bounded canaries while canonical mutation remains in shadow mode. An unrestricted rollout would also need testing across multiple hosts under production load.

Cross-host recovery has not been demonstrated. The agent Task Queue is shared, Workers have no persistent identity, and any Worker polling the queue may receive the retry. The host-local part is the resolver adapter. Every adapter tested so far has resolved sessions stored on the same host.

The resolver contract itself does not require host-local state. A Worker on another host could reattach if the production adapter resolved session ownership from the shared control plane. That still needs to be implemented and verified with a two-host kill test. Same-host Worker recovery is supported. Cross-host recovery remains unproven.

The same failure against the pre-Temporal orchestrator is recorded in the before-state run. That run ends with two agents working in the same worktree and a stale receipt overwriting the current one.

The canary failed, but that’s why we use canaries

The result-delivery path didn’t land cleanly. Its first full-integration canary hit a failed promotion gate and exposed four real defects:

• concurrent notifications contended on a shared mail store;
• correlation metadata named a Workflow that did not exist;
• an observer confused a completed formula step with its still-running source task;
• legitimate completion paths closed real work without producing an OutcomeReady The durable record that a finished result is owed to someone. It stays open, and delivery repeats, until an acknowledgement matching that exact outcome closes it. envelope at all.

The Mayor rolled the integration back to shadow, repaired the delivery path, and ran a bounded recovery that delivered and acknowledged the stranded results.

This was somewhat disappointing, but it’s also why I put in explicit canaries and promotion gates, so at least we learned something. The first implementation wasn’t smooth sailing, but Temporal made the procedure observable enough that failures became specific and recoverable.

What’s running now

A path through this integration is in one of three rollout states: shadow, built and exercised but blocked from touching production work; canary, a bounded live activation someone explicitly authorized; and enabled, promoted for unrestricted production use. Today, Beads mode is shadow and Outcome mode is canary, activated only for the scoped result-delivery path. Delivery and acknowledgement run continuously inside that canary, but Outcome mode has not passed the rollout gate to enabled. Temporal cannot claim production work or start a coding agent without a separate, explicitly bounded canary.

Here is where the fix stands today:

The scoped Outcome canary:

1. Work reaches an end state in Beads, and an OutcomeReady The durable record that a finished result is owed to someone. It stays open, and delivery repeats, until an acknowledgement matching that exact outcome closes it. outbox record is written against that work item. When Temporal closes the work itself, the close and the record are one transaction; for the completion paths Temporal doesn’t own yet, a reconciler notices the end state afterward and writes the record then.
2. A stable Workflow starts for that result, and a notification Activity tells the current Mayor that evidence is available.
3. The Workflow waits durably and redelivers while acknowledgement is missing.
4. The Mayor or a human verifies the evidence and records a disposition bound to the current generation and agent session.
5. Beads stores the exact acknowledgement receipt, which is what allows the Workflow to complete.

Results remain pending until the final receipt exists. Delivery therefore forms a reconciliation boundary, not a same-transaction guarantee.

An independent watchdog scans every store for completed or verified work that has no outcome, as well as for failures reading the store. It runs outside Temporal so it can detect a silent gap in the system it is checking.

Durable waiting introduces two failure modes that need explicit handling.

The first is Workflow history growth. The orchestration Workflow covers a single execution episode and then completes. The longer-lived delivery Workflow uses Continue-As-New after every hundred delivery attempts, carrying only the state needed to resume. Test what happens when an acknowledgement arrives at the rollover boundary. A mistake there can discard the acknowledgement without producing an obvious failure.

The second is a result that is never acknowledged. The delivery Workflow retries indefinitely. That is intentional: the outcome path exists so an owed result cannot quietly expire, while moving it to a dead-letter queue would permit exactly that.

The pressure valve should therefore sit outside Temporal. In my system, the watchdog scans every five minutes and reports unacknowledged outcomes to the Mayor. The result remains pending, but it also appears repeatedly in an operational report rather than persisting unnoticed.

The next step is an escalation policy that distinguishes a newly missed acknowledgement from one that has remained unresolved for several days. For now, that escalation decision is manual and belongs to the Mayor and to me.

What the evidence shows so far

The scoped Outcome canary is live for result delivery, but Outcome mode has not been promoted to enabled. The larger path that claims work and binds it to an agent session is still in shadow, so the old reconciler and its repair jobs remain in production.

Within that bounded activation, the path has delivered and acknowledged more than seventy outcomes, with no silent outcomes detected. That shows the integration works under the canary’s current load. It does not establish unrestricted production promotion. It does not yet show that Temporal has reduced stranded claims or duplicate sessions.

Those comparisons become valid only after the claim-to-session path takes production ownership. At that point, I will measure duplicate sessions, stranded claims, and time from agent completion to acknowledged receipt. The sample will need to grow substantially before any percentage is meaningful.

The current cost is measurable at about 260 MB of memory for the Temporal server, 54 MB for the Worker, and an 80 MB SQLite file. I have not yet measured the added per-claim latency, and the requirements and costs may need revisiting as we add more capabilities.

What Temporal did not solve, and what it cost

Temporal gives a procedure a durable owner. It does not make external effects happen exactly once.

A Worker can die after an external API call succeeds but before Temporal records the Activity completion. Temporal may then run the Activity again. Generation fences make repeated Beads writes safe, and start-or-attach semantics prevent retries from launching duplicate agent sessions. But GitHub, Slack, a payment provider, and an agent process do not acquire exactly-once behavior because their calls came from a Temporal Activity. If an Activity posts to Slack (I set my agents up to do that) or creates a GitHub issue, safe repetition still depends on that destination’s idempotency model. This is where “Temporal handles retries” becomes an engineering decision.

Temporal also does not determine whether the procedure is correct. It does not repair a corrupt Beads store, clean abandoned worktrees, manage disk space, choose approval policies, or make an unsafe external action safe. It does not replace acceptance criteria, deterministic tests, independent review, or human judgment. It remembers the procedure the application defined. The application still has to define a good one.

It preserves mistakes as faithfully as correct behavior. In one canary, the agent completed its work correctly, but an adapter placed the wrong store identity in the outcome envelope. Temporal retried the same invalid envelope every fifteen minutes, exactly as instructed, until I rolled the canary back. Durable execution made the failure persistent and observable rather than silent. It could not correct the adapter’s mistake.

The engineering costs:

• Workflow changes require replay discipline. Workflow code must remain deterministic, and previously recorded histories must continue to replay after the code changes. Every orchestration change therefore needs compatibility planning and replay tests.
• Activities require explicit retry safety. Stable identities, idempotency keys, claim tokens, generation fences, and start-or-attach behavior remain application responsibilities.
• Workflow history must stay small. Prompts, transcripts, and worktree contents cannot be stored directly in Workflow events. Histories contain compact identifiers, fences, hashes, and references to external artifacts.
• The system gains more infrastructure. Temporal adds a server, Workers, monitoring, and rollout machinery such as shadow mode, bounded canaries, promotion gates, and rollback paths.
• Operators must reason about two durable records. Beads is authoritative for work facts. Temporal Event History is authoritative for procedural progress. Debugging requires knowing which system answers which question.
• Retries amplify bad instructions too. Temporal will reliably repeat an incorrect decision until the application changes the decision or stops the execution.

Not every job justifies those costs. The maintenance pilot from earlier stayed cron plus a lock, because a crash there does not leave behind an in-flight procedure that the database cannot explain. That selectivity applies inside the work graph too: I am not yet translating every Gas City formula, node, or transition into a Temporal Workflow. Doing so would add machinery without adding a guarantee the system needs, though it does warrant more investigation.

What transfers, and what is specific to this system

The transferable part is the division of responsibility.

A Workflow owns the durable procedure. An Activity performs the nondeterministic work. Stable identity lets a retry continue the same logical operation instead of starting another one. External effects remain the application’s responsibility.

The rest follows from the kind of work my Activity starts. A Gas City agent may edit a Git worktree for an hour and continue running after the Worker that launched it has died. That requires generation and claim fencing, session reattachment, an exact acknowledgement from the Mayor, and an independent watchdog. A short, idempotent Activity would not need most of that. The extra machinery belongs to the workload, not to Temporal itself.

The useful question is not which framework should own the system. It is which guarantees must survive a crash.

NDI fits the parts where any acceptable route to the result is valid. That is often the right model for the coding work itself. Deterministic replay fits the parts where the procedure is part of the promise: ordering, retries, waits, acknowledgements, cancellation, and gates. The same multi-agent system can use both without forcing one model onto the other.

That was the boundary I needed to repair. Losing an in-flight procedure could leave the system unable to say whether an agent was still working, whether its result had been delivered, or whether another agent could safely take over. Reconstructing the answer from claims, processes, worktrees, and partial messages was both expensive and unreliable.

The city will still stall and catch fire. Coding agents will continue to make locally reasonable choices that the surrounding system did not anticipate. The goal is not to prevent every failure. It is to keep one failure from turning into a chain of repairs, each adding another brittle mechanism to infer what the previous one lost.

A missing coordinator should not erase the ownership of a session that is still working. A delayed acknowledgement should not make a completed result disappear. A restarted Worker should not create a second agent against the same task because it cannot remember the first.

The agent can choose its route through the work. Beads can preserve the work record and the identity attached to it. Temporal can preserve the promises around that work: who owns it, what remains outstanding, when a handoff is complete, and what must happen next.

The replaceable parts should be safe to replace. The continuity the system depends on should survive them.

Durable execution is not only about surviving process failure. It is about choosing which forms of continuity the system refuses to lose.

The evidence, and where to go deeper

• The paired before-state Worker-kill run is recorded end to end: the video, or the raw asciinema event log it renders from.
• The annotated source reader renders every before and after file at its pinned revision, section by section, with the complete test matrix.
• The demo’s evidence bundle is downloadable: the invariant report for both arms, the retry-gap samples from the decisive arm, the agent session events, the before-state gate report, the before-state store with the stale receipt on top, and the run provenance, which names the revision, SDK, server, and host, and hashes every published evidence file.
• For Gas City contributors: the natural upstream shape of this integration is a pack, so an installation adopts durable procedure only at the boundaries where its guarantees earn their cost.
• Engineering Reliable Coding Agents is the guidebook I’m drafting; its containment and recovery chapters take this boundary across more systems than this one.

Visual style inspired by Ian 小黑 illustrations, available under the MIT license.