Prose skills execute 56% of their own steps; compiled harnesses hold 86% across model generations

A prose agent skill gets 56% of its own mandated steps executed, and the artifacts still pass output checks. That number is from SIGIL, whose second version landed on cs.SE today, measured across 30 skills and two model generations. A skill file is described to the runtime but never encoded in it, so the model re-derives the control flow on every run and drops the verification steps somewhere in the middle. Compile the same prose into a typed harness and the number goes to 86%, with 2.3x as many full-procedure completions at 0.58x the tokens. The part worth arguing about is the variance: the compiled harness holds at 86% across both model generations while the prose version swings from 56% to 68%. Capability improvements move the prose number around; they do not move the guarantee.

Mechanism in code, cognition in the model. That split ran through most of what landed in the last day.

Metis applies it at the tool boundary. It is a multi-provider runtime that converts provider streams into typed events before any admitted call reaches an external effect, so permission decisions, interference classes, terminal results, and lifecycle transitions become inspectable objects instead of things you reconstruct from a transcript afterward. Across 30 matched real-I/O pairs, four-class mediation brought median elapsed time down from 25.958 ms under forced serialization to 14.146 ms, a mean paired difference of -12.295 ms with a 95% bootstrap interval of [-12.968, -11.694], faster in every pair. In the child-boundary ablation the full gate-plus-registry condition blocked the declared unauthorized effect and hid all five escape tools, and removing both protections reversed both observations. The paper is unusually careful about what it does not show: a ten-case fault matrix exposed duplicate-identifier and rollback limits, and the authors state outright that none of this establishes semantic safety, working rollback, or superiority over another runtime.

The ops-side version of the same argument showed up the same morning in Daniel Martin’s durable change-control gate: revalidate policy immediately before the outbound call, route approval with the actual payload rather than a summary of the plan, transmit once under an idempotency key derived from the action’s business identity, then verify the receipt instead of resending. His simulation of 10,000 outbound agent actions is synthetic and he says so, with a 9% policy-change rate, 21% non-approval, and one to three resume events per action. The row that earns attention is the preflight gate: it eliminated policy-bypassing and unapproved sends entirely, and still produced 1,514 duplicate sends, because resumed workflow events re-entered the same action. Only the durable record with a stable key took duplicates to zero. Approval correctness and durability are separate failure modes, and a gate that handles the first one reads as finished in a design review.

Retry amplification is that same lesson from the pre-agent world, with worse numbers than the folklore suggests. Across 200 open-source Python microservice projects, explicit retry logic was detected in 11.5%, and the authors’ audit of their own false negatives puts true prevalence near 41%. Of the projects where retries were found, 60.9% contain at least one configuration with no backoff, and exactly one of 113 production configurations randomizes its delay. In simulation at 100 trials per strategy, under correlated failure a naive standard retry policy drops the success rate from 55.4% to 41.5% relative to performing no retries at all. Their Adaptive Retry Budgeting holds success near the no-retry baseline while still recovering transient faults. Anybody wiring per-tool retry into an agent loop is assembling the same call-path pileup one layer up, now with a token bill attached to every amplified request.

Debugging those systems is where Repair or Resample? cuts deepest. It asks whether existing multi-agent repair methods causally fix failures or merely exploit LLM sampling randomness. SymTrace records the execution trajectory and establishes intervention anchors, then on replay reconstructs everything before the anchor from logs and regenerates only the downstream portion, which is what makes a multi-agent failure reproducible at all. On SymFail, 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence, unguided rerun reproduced the original failure 67.97% of the time and repaired 6.90%. A symptom-driven intervention reaches 20.15%, a 191.89% improvement over the prior best and still four failures in five left standing. Resampling has been passing as debugging across three mainstream frameworks.

The counterexample to all of this arrived through Simon Willison, citing Johann Rehberger’s attack on Claude Code’s auto mode, which Rehberger reports working about 80% of the time. The mechanism is a zip archive the agent downloads and unpacks, after which running code that does import base64 picks up an extracted local struct.py. The detail that matters for anyone building a classifier-shaped safety layer is what happened after that. In several runs Claude noticed the compromise and tried to terminate the malware process, and auto mode denied the cleanup command. A layer that permits creation of an effect and then blocks its remediation is worse than no layer, because the operator’s mental model still reads as protected. Rehberger’s recommendation is the boring one, and it is the same shape as everything above: container or VM, restricted network egress, no home directory or SSH keys or cloud credentials inside the agent runtime, monitoring on top.

On the retrieval side, LivingRAG adds a writable experience store to a graph RAG backbone so that verified reasoning from earlier queries survives inference instead of being thrown away. Stored graph signals steer retrieval toward entities and passages that paid off on related queries; stored summaries hand generation a reference reasoning pattern. The authors analyzed online QA streams first to confirm the reusable signal is actually present in shared entities, graph neighborhoods, and question templates, and report accuracy above strong RAG baselines with lower completion-token use when prior experience applies. Caching answers is old. Caching the retrieval path that produced them is the more interesting variant, and it inherits every staleness problem a cache has ever had.

The claim to try to break next is SIGIL’s model independence. If a compiled harness really holds step completion flat while the model underneath it changes, then the durable unit of agent engineering stops being the prompt and becomes the typed intermediate representation, and every skill library currently written as prose turns into a compilation target. Two model generations is a thin basis for that.