Agent identity explains under 3% of benchmark variance

Across three open agent-trace benchmarks, the agent itself explains less than 3% of the variance in outcomes. The agent-by-task interaction explains 7 to 23%. That is the central number in Deployment Decision Reliability, posted yesterday, which runs a four-facet Generalizability Theory variance decomposition over TheAgentCompany, τ²-bench, and AppWorld, fit three separate ways (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agree to three decimal places. The conclusion the author draws is blunt: leaderboards rank specialization, not capability. Which agent wins depends far more on which tasks you happened to put in the suite than on anything intrinsic to the agent.

Two follow-on findings make the picture worse for anyone using a public ranking as a procurement input. Aggregate reliability collapses exactly where you need it: on the hardest task quartile of τ² action_checks, Eρ² drops from 0.752 to 0.000. And training-cell reliability correlates negatively with held-out reliability, r = −0.90 on τ², so the evaluation designs that look most stable are the ones that replicate worst. What does transfer is population-level, not per-agent: the capability-gap ratio holds at 0.35–0.40 across enterprise benchmarks while per-family agent rankings invert. The paper packages this into a one-page reporting discipline (DDR) that converts the variance-component table into five decisions a buyer can defend, with code and fit artifacts released open source.

If the leaderboard number is mostly interaction, the interesting question becomes where in the trajectory the variance actually lives. SWE-RPG attacks that by adding validated ground-truth references for two intermediate stages, requirement clarification and implementation planning, alongside the usual executable patch check. It covers 163 tasks from 31 Python and Java repositories (113 bug fixes, 50 feature additions), evaluated with Claude Code, Codex, and OpenCode across six backends including Claude-Sonnet-5 and GPT-5.6-Terra. Average resolve rate: 31.5%. The intermediate ground truth localizes the damage — implicit requirement recovery accounts for 24.5% to 46.0% of runs, depending on the configuration. The agents are not primarily failing at writing the patch; they are failing to reconstruct what the issue actually asked for before they start writing.

The benchmark supply itself is getting audited. SWE-Bench ProMax opens by citing an audit finding that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, either too narrow (rejecting correct solutions) or too broad (checking unstated requirements), plus evidence that frontier models can reproduce gold patches verbatim from training data. Their answer is to move to refactoring, where behavior-preserving changes must land coherently across many files: 170 expert-curated instances from real commits in seven languages, issue descriptions rewritten from scratch, test suites manually reviewed, averaging 11.4 modified files and 261.6 lines per instance. Best frontier model under two scaffolds: 41.2%.

Two papers this window treat the loop and the harness as the object of study rather than the substrate. Microsoft’s LoopsBench frames each of its 112 tasks as a dependency DAG over separately testable development units with source-evidenced prerequisite edges, and its runtime releases tests along the ready frontier while keeping completed nodes as standing regression obligations. Opus-4.7 with Claude Code and outer continuation, the strongest configuration tested, resolves 25.00%; recorded plans recover only part of the true prerequisite DAG, and regression events stay visible across every loop profile evaluated. One Recipe, Many Harnesses holds a self-evolution recipe fixed across eight languages of Multi-SWE-Bench and three base models, routing every edit through a typed failure signal recorded as a falsifiable contract so each modification stays attributable afterward. The evolved harnesses share an abstract playbook but instantiate it with almost disjoint language-ecosystem machinery; the shared core distills into one universal harness while the ecosystem margin resists transfer and requires native re-evolution. Gains track recoverable execution defects — where defect mass is near zero, gain is near zero. A harness closes the gap between what a policy can do and what it does, which is a much narrower claim than most scaffold papers make.

Organizational structure shows the same magnitude of effect. MSEval grounds multi-agent coding in 10 real full-stack projects across 10 domains, scores against hierarchical requirements and deterministic rubrics, and runs 10 collaboration topologies through periodic sync intervals and native CI/CD, measuring functional success, latency, and prefix-cached token cost together. Across 100 runs, holding task and model constant and changing only the topology shifts scores by more than 30 points and doubles wall-clock time. Structured pipelines converge fastest at highest quality; heavy managerial oversight degrades performance.

Read together, four of these six papers are measuring something the leaderboard averages away: the interaction between an agent and the specific shape of the work. Worth watching whether anyone reports a DDR-style variance table alongside a headline resolve rate, and whether the ~60% flawed-test figure gets independently replicated against SWE-bench Verified.