The agent you pick explains under 3% of whether the task succeeds

Across three open agent-trace benchmarks, the agent you pick accounts for less than 3% of the variance in whether a task succeeds. The agent-by-task interaction accounts for 7 to 23%. That is the headline result from Vasundra Srinivasan’s Deployment Decision Reliability, posted this week, and it is the sharpest statement yet of something practitioners have been circling for a year: the leaderboard column you are reading is mostly noise plus specialization, and almost none of it is capability.

Leaderboards rank specialization, and the benchmarks underneath them are worse than the rankings suggest

Srinivasan runs a four-facet Generalizability Theory variance decomposition over TheAgentCompany, τ²-bench, and AppWorld, fit three separate ways (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agree to three decimal places. The follow-on findings are harder to shrug off than the headline. Aggregate reliability collapses on the hardest task quartile: Eρ² on τ² action_checks drops from 0.752 to 0.000, meaning the tasks you actually care about are exactly where the measurement stops working. Training-cell reliability correlates negatively with held-out reliability at r = -0.90, so the evaluation designs that look most stable are the ones that replicate worst. On the MAST failure taxonomy, trace-level failure-mode profiles are idiosyncratic (MAE 0.261) while cell-level profiles generalize (MAE 0.056, r = 0.83), which is a concrete argument for reporting failure distributions per task family rather than per run. The paper packages this into a one-page reporting discipline with code and fit artifacts released open source.

That is the statistics side. The instrument side got audited too. SWE-Bench ProMax opens by citing an audit finding that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, either narrow enough to reject correct patches or broad enough to check requirements nobody stated, and that frontier models can reproduce gold patches verbatim from training data. Yuling Shi and colleagues respond with 170 expert-curated refactoring instances across seven languages, issue descriptions rewritten from scratch, test suites manually reviewed in both directions, tasks averaging 11.4 modified files and 261.6 lines. Best frontier result under two scaffolds: 41.2% resolve rate. Refactoring is a good choice of task here because behavior preservation across files is checkable without the ambiguity that sinks issue-resolution instances.

The methodological complaint gets its own paper from the JetBrains group. Don’t Claim Benchmark-Oriented Optimization Improves General Coding Capability argues that post-training reports, model cards, and launch posts routinely treat SWE-bench and LiveCodeBench numbers as evidence of broad coding ability, when what those numbers measure is task-specific performance on a narrow slice. The argument is not new; the value is having a citable statement of it aimed squarely at the model-card genre.

Grab published the practitioner version. Grab Bench is built around a failure class they name precisely: not hallucination, but subtle plausibility. A right-looking SQL query, a plausible tool call, a patch that satisfies the surface tests. Their row-level failure analysis is what drove the benchmark design, and building an internal eval on your own production shapes is the correct response to Srinivasan’s variance decomposition. If agent-by-task interaction dominates, your tasks are the only ones that predict your outcomes.

The harness became a measured variable

The Devil Is in the Interface is the controlled experiment the field needed. Xiangzhe Xu and coauthors hold the underlying information and actions roughly constant and vary only how those capabilities are organized and exposed, comparing six tool architectures across three actors and 11,700 trajectories on repository-level issue fixing. Against a bash-only baseline, structured low-level interfaces improve consistency across repeated attempts by up to 4.7×. Natural-language search widens exploration and raises access to relevant files by more than 11%. Python CodeAct-style interfaces hit similar task performance with 41.6% fewer steps and 56.3% lower token usage. The null result is the one to keep: lightweight text-based cognitive scaffolding, tools that let the agent record intermediate reasoning, barely moved actor behavior. Scratchpad tools are cheap to add and apparently cheap in effect.

If the harness carries that much of the outcome, it belongs in the training loop. HELIX from Tianyu Fan and Chao Huang proposes model-harness co-evolution: build harnesses against a fixed model, update the model from verified sibling trajectories, rebuild harnesses as capability shifts. The substrate decomposes agent systems into typed ports, reusable atoms, recipes, product shells, and runtime policies, keeping interventions individually traceable. One evolution round on code repair: a 65-candidate portfolio finds a fixed harness improving task coverage 4.0% over the Pi baseline, while the full portfolio exposes up to 58.0% more verified coverage through complementary sibling behavior. A 200-slot sibling slice yields 438 verified SFT, critic, filter, and preference records. The gap between 4.0% and 58.0% is the interesting number: most of the value is in the portfolio’s diversity, not in any single winner.

Marco Tulio Valente’s architecture study makes the plainer point that no systematic architectural description of coding agents exists, comparable to what we have for compilers or operating systems, and documents the main components using a research prototype. Unglamorous, and overdue.

Reliability guarantees borrowed from database systems

Agentic Transaction, from Zhaoyan Sun, Xiaoxiao Wang, and Guoliang Li at Tsinghua, reinterprets ACID for agent execution as four semantic guarantees. Semantic atomicity: an exploration-execution-validation cycle commits or retries as a unit, and failed steps never touch the workspace or memory. Semantic consistency: committed outcomes must satisfy preconditions and evidence obligations even though the trace is nondeterministic, enforced here by a confidence-divergence check that compares an executed decision’s grounding against the alternatives found during exploration. Semantic isolation: concurrent sub-agents get dependency-aware policies, versioned workspaces, and Git-like branch-and-merge for the collaborative case. Semantic durability: append-only workspace, provenance tracing, knowledge-graph memory that records only validated units.

On KramaBench their ACID-Agent scores 74.6% against Claude Code’s 64.0% on the same Qwen3.5-397B-A17B backbone, a 10.6-point gap, and beats Claude Code running the larger GLM-5.2 backbone. The ablation is the load-bearing part. Removing failed-step isolation, so intermediate failures flow into workspace and context memory, costs 11.7 points. And ACID-Agent beats three-run majority-voting Claude Code while spending fewer tokens, which rules out the “it just thinks longer” explanation. Confidence estimation runs on a local Qwen3-0.6B because API models do not expose token probabilities, a practical constraint worth noting for anyone trying to copy the design.

AWS shipped a working counter-position on orchestration shape. kiro-flock runs clusters of headless Kiro CLI agents on EC2 with nothing between them but an S3 bucket: no orchestrator, no message bus, no broker. Agents sit in a logical ring, read a bounded window of neighbors’ append-only logs, and append one JSON line per iteration describing what they did, the result, and their next intent. That line is the entire coordination protocol. Ivo Kammerath is specific about when this loses: known task tree, strict ordering, or a required verification gate all favor a supervisor, and the cited 2025 scaling study found architectures without centralized verification propagate more errors. The failure modes map to design choices rather than bolted-on safeguards. Groupthink comes from mesh visibility collapsing the cluster onto the first signal, so you open amorphous and switch to mesh only to align. Drift comes from persistent session history building behavioral momentum, so every iteration starts a fresh session and all state lives in the shared logs. Their largest run was 184 agents across 11 cooperating clusters building a programming language. Rings past the low hundreds are extrapolation, and the post says so.

When these systems fail, someone has to find the step that broke. TRAJDEBUG attacks critical error detection, locating the earliest step responsible for a failure, and frames the difficulty as an error-lifecycle problem: long trajectories make individual errors hard to isolate, and the first error is not always the one that determined the outcome.

Agent memory grows because deletion is exponentially expensive

Kushal Chakrabarti’s Why Does CLAUDE.md Keep Growing? has the best mechanism argument of the week. Appending an instruction is cheap. Deleting one, once its rationale is gone, costs O(2^|D|) in a prompt of |D| instructions, because you cannot know which subset of remaining instructions the deleted one was compensating for. He names the resulting divergence catastrophic remembering, the inverse of catastrophic forgetting.

The measurement covers 247,694 instruction lifetimes across 1,867 repositories. Agentic prompts more than triple over their lifetime, +226%, gaining +4.9 net instructions per commit, and the log-hazard of deletion falls 0.032 per commit, so old instructions are progressively safer from removal regardless of whether they still apply. The fix he tests is comments. Inverting IFEval into verifiable worlds with known-optimal prompts, comments encoding latent reasoning remove 99.3% of excess instructions, cutting excess from +211.3% to +1.4%. On WildIFEval, comments improve real-world instruction-following by up to 23.1%. His closing line is the right one: if English is the new code, why don’t we have comments yet?

On the storage side, MemoryLake on MemoryArena is a matched comparison of memory backends against Mem0, text-embedding-3-small vector RAG, and a long-context control, all sharing agent framework, model alias, task samples, and scoring code. The result worth carrying is the absolute level, not the ranking: every system scored zero on travel planning, web shopping produced a single bundle-level success out of 150, and the best equal-weight average across five domains was 20.5%. The authors state plainly that sample sizes are modest, confidence intervals overlap, and they ran no paired significance tests. Multi-session interdependent task completion is not close to solved by any backend on offer.

One refactor, 31 audit passes, 201 defects, $2,430

Joël Abenhaïm published a fully instrumented case study of a single large refactor: dismantling a lifetime invariant (a UI panel stays open for the duration of an AI request) across a 717,725-line TypeScript application, with no pre-existing test oracle for the target behavior and no human review of the generated code at any stage.

The protocol is the contribution. The agent writes a formal specification, then 14 refinement cycles re-audit that specification against the real source, producing about 85 corrections and expanding scope from 110 to 160 files as unnoticed dependencies surface. Cycle 14 returns nothing, so cycle 13’s spec is frozen. Implementation runs against the frozen spec; the agent initially refused a partial implementation on atomicity grounds and proposed a decomposition instead. Then 17 verification cycles audit the code against that same frozen document, producing 116 corrections. Stopping rule: two consecutive zero-finding passes, met at cycles 16 and 17. Across 31 audit passes, 201 defects were corrected before a human ran the program once. Final commits: 288 files, 34,770 insertions, 16,422 deletions. Three days, USD 2,430, no regression in the pre-existing suite, no bug observed across roughly thirty later sessions.

The limitations section is unusually honest and does most of the work of situating the claim: single case, no control condition, self-reported by the tool’s author, closed source, one frontier model in extended reasoning mode. What survives all of that is the structural idea, which is that a frozen written referent lets you audit intent before generation instead of auditing code after, and that repeated auditing against a fixed target converges without requiring any single pass to be reliable. The 1,500+ pages of raw session logs are published, in French, explicitly so a language model can check them against the paper’s claims.

Stephanie Jarmak’s monograph Engineering Reliable Coding Agents landed on arXiv this week and covers the same territory systematically: harness, execution state, retrieval, memory and state management, permissions, review interfaces, resource allocation, treated as the boundaries where reliability is actually determined rather than as infrastructure around the interesting part.

What to watch

Three threads worth tracking into next week. Whether anyone replicates Srinivasan’s variance decomposition on a closed enterprise benchmark, since the claim that per-family rankings invert while population-level diagnostics transfer is the kind of thing that either holds up everywhere or falls apart immediately. Whether the ACID framing produces a general-purpose implementation rather than a data agent, because semantic isolation across sub-agents sharing a repository is the hard case and the paper’s Git-like branch-and-merge sketch is only a sketch. And whether the specification-first protocol gets run by an independent operator on a public codebase, which would remove three of its seven stated limitations at once and turn a compelling single case into something you could plan around.