SWE-bench Verified cannot separate any of its 29 adjacent top-thirty pairs

Two systems sit atop SWE-bench Verified with 396 of 500 instances each, and an exact paired McNemar test cannot separate a single one of the 29 adjacent pairs in the top thirty at alpha 0.05. That is the headline of Coding Agents Have Converged, an audit by Fengshuo Liu and colleagues of 254 SWE-bench submissions across four splits, done without running a model. The top ten share 285 successes and 51 failures, which leaves 164 instances doing all the work of telling them apart. Frontier solution sets nest inside each other with a median nesting of 0.935 against a score-implied baseline of 0.774, so the systems are mostly solving the same problems and failing the same ones. The part that should change how you read a leaderboard is the model-scaffold interaction: within a single model, scaffold choice moves the score by up to 29.8 points, more than three times the 8.8-point spread across the entire top thirty, and six of nine cell-mean interaction tests survive Holm correction. The larger Test split does better, separating 14 of 23 adjacent pairs, which is the authors’ argument for reporting comparison-set-specific resolution and an instance budget rather than a rank. They release the partition and a five-step audit protocol. The design is observational, so none of this identifies a causal scaffold effect, and non-rejection is not equivalence; the claim is narrower and more useful: the leaderboard gap you are looking at is probably noise.

If the benchmark cannot order agents, the next question is what to measure instead, and two papers today answer with behavior rather than resolution. AgentGuard (Wuyang Dai and Song Wang) starts from 642 failure traces collected from real coding-agent runs across 382 repository tasks, mines the recurring execution-failure patterns, and generalizes them into instruction-level constraints packaged as a guardrail skill that activates only the rules relevant to the current instruction. Learned from 461 traces over 282 tasks and evaluated on a disjoint 100, with Claude Code driving Claude Haiku 4.5, the guardrails cut the abnormal execution rate from 69.0% to 26.7% and lifted successful task completion from 21.7% to 35.0%. Abnormal here means the things a pass/fail score never sees: editing unrelated files, rewriting tests, issuing unsafe commands, proceeding past a failed validation. The completion lift is the interesting part, because it says the constraints are not merely a safety tax; an agent that stops rewriting tests to make them pass also finishes more tasks properly. The remaining 26.7% is the open number.

When Agent Metrics Measure Different Things (Stefan Creadore and Peyton Woakz) is the evals paper for people who run agents in production and report numbers upward. It is a retrospective, source-linked audit of one company’s pipeline, Praxa AI, and every finding is a construct mismatch. A 139-case offline routing report holds 112 passes and 27 failures yet reports zero gating failures, because the known gaps are exempted from the gate. Of 8,843 tool-attempt rows, 121 durations equal exactly 2,147,483,647 ms, the signed 32-bit maximum, tagged as abandoned clients and produced by database code that clamps lifecycle age; the pooled 99th-percentile latency is therefore 2,147,483,647 ms while the server-observed p99 for completed calls is 38,118 ms. A compaction pilot reports a 94.39% reduction in follow-up input tokens, which becomes 46.54% once the trigger call is counted alongside the follow-up. None of these are arithmetic errors; the authors reproduce every calculation and verify 91 timing statistics with a separate rational-arithmetic implementation. The gate policy, the lifecycle timing, and the request-level accounting are each measuring something real, just not the thing their labels imply.

Anthropic published a production account of what happens when the agents are on the other side of the CI wall. Agentic coding strains CI reports a 25x increase in CI jobs over six months against a 10x growth in test volume with roughly flat headcount, with Claude writing about 80% of the code. Their test-impact-analysis service had two parts, a listener that records every test result into history and a selector that picks which tests to run on a PR from past results and package relevance. The listener was a singleton, and once its lag passed 20 minutes the failure cascaded: bad merges broke downstream tests, flaky tests blocked merges, and new tests could not run until the backlog cleared. Three patches bought time, doubling cores in October 2025 (70 days), sharding by package in February 2026 (29 days), and daily restarts in March 2026 (under a day). The fix was a three-week rewrite by one engineer: stateless listener workers appending to journals, a separate consumer rolling up per-test history, and an in-memory store instead of process state. The post gives no cost figure or test-skip percentage, only the before-and-after shape of the queue, flat after cutover. The design rule they take from it is to size v0 for 25x load within two quarters.

Which tests to rerun is a special case of which evidence to restore, and Assurance Envelopes for Autonomous Coding Agents (Anjan Goswami) formalizes the general case. A change carries obligations, properties it must preserve; the repository carries evidence, tests, type checks, proofs, static analyses, traces, each with a cost to reload. The paper models evidence and the rules that combine it as a typed inference graph and asks for the least-cost subset whose forward-chaining closure reaches every obligation, validated by that closure rather than by trusting the optimizer. On small graphs derived from prior agent runs on Rust, IronBlocks, and Pong, the minimum envelope depends on the task, sometimes does not exist, and sometimes requires several pieces of evidence together, a structure that a baseline ignoring conjunctions necessarily fails to recover. A 249-instance synthetic benchmark puts median CP-SAT solve time under 20 ms at 500-evidence graphs, but graphs with many alternative derivations per target time out at far smaller sizes; structure, not size, is the hard part. Discovering the obligations in the first place is left open, which is exactly where a listener’s history of which tests break with which changes would plug in.

RepoAtlas (Yunxiang Zhang and colleagues) works the context side of the same problem. Full repository graphs are too dense to render usefully, and a one-shot local view goes stale as the agent explores, so RepoAtlas runs a select-project-refresh loop: pick a task-relevant region of the code graph under a fixed budget using the issue plus the agent’s current exploration state, project it into paired visual and textual views, and refresh when the exploration state has moved. Training-free, it adds 2.4 points of resolve rate on SWE-bench Verified over the strongest multimodal graph baseline while cutting input tokens 5.8% and model calls 7.8%, consistent across three models of different families. Read against the convergence audit above, 2.4 points is inside the range a paired test would not separate; the token and call reductions are the more defensible claim.

On the retrieval side, Ivan Silajev shows that TF-IDF and BM25 are exact KL divergences between two probability models, covering the BM25 variant with the plus-1 IDF correction that production systems actually use as well as the original without it. Neither scorer has had a standard probabilistic derivation placing it in a unified statistical framework, and lexical lanes remain the backbone of hybrid retrieval in most agent stacks, so an exact information-theoretic reading of what the score measures makes them comparable to other retrieval methods on paper rather than only in a benchmark table.

What to watch: whether SWE-bench leaderboards adopt paired-test reporting or an instance budget, and whether failure-trace-derived guardrails of the AgentGuard kind get evaluated on the behavioral axes the convergence audit says are now the only ones with room to move.

Feed note: the code-intel mirror reported a last-sync timestamp of 2026-09-01 at run time, although the items above carry publication dates through 2026-09-16.