Frontier models score below 64.3% on 20 live-cluster incident repairs

Frontier models score below 64.3% on Incident-Arena, a human-built benchmark of twenty incident-response tasks from Friday’s arXiv listing. Each task deploys a production application to an ephemeral Kubernetes cluster, injects a fault somewhere between the config layer and the underlying images, and holds a sustained load profile while the agent works. Trials average 2.81 million tokens and 41 turns. The verifiers are functional, so a repair counts only if system-level metrics hold steady and the fix is made safely. Failures span the whole chain, from diagnosis and localization errors through incomplete repairs to unsafe regressions. The abstract gives no breakdown by model or by failure stage and twenty tasks is a small set, so 64.3% is best read as the high-water mark under a strict verifier.

Two cs.SE papers from the same listing measure coding agents on axes that functional tests skip. Code That Works, Environments That Don’t runs three agents on fifty tasks in four languages and compares three layers of dependencies: what the agent declares, what gets installed at runtime, and the necessary-and-sufficient set. Agreement between agents on the dependency set for an identical task falls as low as 7%, newer agents show no meaningful improvement, and the widest gap sits between the declared and runtime layers, which the authors attribute to environment priors learned in training. Functional tests detect none of it.

SWE-chat, revised on Friday from its April release, covers the other blind spot, which is what happens when the user is a working developer and the task is whatever they needed that day. The dataset now holds almost 18,000 sessions discovered in public repositories, with more than 229,000 user prompts and 2 million agent tool calls, and it attributes committed code to human or agent. Usage is bimodal: in 41% of sessions the agent authors virtually all committed code, and in 25% the human writes all of it. Only 59% of agent-produced code survives into a user commit, users push back through corrections, failure reports or interruptions in 50% of turns, and agent-written code introduces more security vulnerabilities than human-written code. Survival rate is a metric a team can compute from its own session logs and commit history, and I would track it before any leaderboard position.

On retrieval, Kapa published Company Knowledge Bench: 1,000 eval cases built from real production queries over documentation, tickets, chat, wikis and code, each scored against a boolean criterion naming the minimal set of preferred-source chunks that completely answers the query. Plain hybrid search scores 0.41, a reranker lifts it to 0.50 for a third of a second more, and query decomposition reaches 0.56. A GPT-6.1 Sol agent with grep as its only tool ties Kapa’s tuned fixed pipeline at 0.61, but it takes 17 seconds per query against 3.3 and returns about 40,800 tokens at 4% precision, so the answering model pays roughly four times as much to read the result. Kapa’s own agentic retriever leads at 0.65 in about five seconds. The post states its bias up front, since Kapa built all seven retrievers and the ingestion pipeline. The labels were also written by agents, tuned against 170 hand-labelled cases, and the post describes that agreement as “high enough” without giving the figure.

Three more papers from Friday propose controls. Fault-Tolerant Budget Conservation in Distributed Multi-Agent Delegation treats a resource limit as an authorization boundary and asks what it takes to hold one when replies are lost, effects complete after a timeout, messages repeat, or a DAG join aliases one lineage. The design will look familiar to anyone who has built a payments ledger: exclusive escrow credits, a reservation bound to lineage, epoch, receiver and idempotency key before dispatch, a signed permit the gateway verifies before first acceptance, and uncertain effects that stay charged until authenticated settlement or a fenced no-effect proof. The authors prove that each charge settles at most once and that a child can never spend more than it was given, then exercise the design with bounded TLA+ checking and crash-injected two-process SQLite experiments. Those checks catch a deliberately broken variant that refunds budget when a call times out, which is the obvious shortcut in a hand-rolled limiter and is wrong because the effect can still complete after the timeout.

VeriHarness starts from the observation that disagreement between repeated rollouts of one task often exposes a correct alternative, while consensus can conceal an error every rollout shares. It turns the generator’s own model into a verifier with a workspace, evidence tools and reusable verification skills; one component resolves competing claims against the environment, and another challenges the claims all rollouts agree on and searches for omitted requirements. Across five long-horizon workspace benchmarks, evidence-backed revision brings the gain over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 with Claude Opus 4.8, and the authors release roughly 26,000 rollouts that cost over $100,000 to produce. Deny Without Disabling takes on multi-agent permissions, where contributions that are each admissible can combine into a prohibited use and blanket blocking defeats the authorized work. Its evaluation makes blocking the prohibited use and completing the authorized one a joint success criterion, and reviewing combined artifacts takes what the paper calls the denied-commit rate from 86.0% to zero with no loss of authorized supply in controlled composition experiments.

What to watch: whether Incident-Arena’s full tables separate diagnosis failures from unsafe repairs, because those point to different harness work, and whether anyone outside Kapa reruns the grep-agent comparison on their own corpus. Nothing in this window measured agents against governed business definitions, so the semantic-governance thread sits out this issue.

Feed note: arXiv items come from the Friday, October 2 listing, the latest before the weekend gap; blog and community sources were current through the morning of October 4. The mirror’s status endpoint still reports its last sync as 2026-09-01, though ingest is live.