The Scores Fall Apart When You Change the Machine

Replay the official reference patches for GSO, SWE-Perf, and SWE-fficiency across four types of Google Cloud machine and most of them stop being valid. An audit posted in the last day (arXiv 2607.01211) reran 740 performance-optimization tasks and found the reference patches still satisfy the benchmarks’ own validity rules in only 39 of 102 GSO tasks, 11 of 140 SWE-Perf tasks, and 411 of 498 SWE-fficiency tasks once the machine underneath changes. SWE-Perf is the fragile one: many of its reference patches produce close to zero runtime change, so the thing it claims to score barely moves. The rankings are just as soft. Across eight submissions shared by GSO and SWE-fficiency the official orderings disagree on 9 of 28 pairwise comparisons, and SWE-fficiency’s scoring rule hands its ten worst tasks a combined weight of 58 to 83 percent. When a leaderboard says one agent optimizes code better than another, that gap is often an artifact of which machine ran the timer and how the score was weighted.

That paper landed alongside two others circling the same doubt, that outcome scores hide most of what we actually want to know. RigorBench (arXiv 2606.22678) argues an agent that reaches a correct patch through reckless trial and error is less reliable than one that plans, verifies, and recovers, even when both pass the test. It scores harnesses on five process pillars (planning fidelity, verification coverage, recovery efficiency, abstention quality, atomic transition integrity) across 30 tasks with names like Doom Loop Gauntlet and Don’t Break the Build. Running the same harness with structured discipline turned on raised process-quality scores by 41 percent and, more usefully, lifted downstream outcome correctness by 17 percent. How the agent works turns out to predict whether the answer holds. MemSyco-Bench (arXiv 2607.01071) points the same lens at memory: existing memory benchmarks check whether an entry was stored, retrieved, and updated correctly, and stop there. It instead measures whether a retrieved memory should have swayed a decision at all, testing five behaviors including rejecting memory as factual evidence, respecting its applicable scope, and resolving conflicts between a stored preference and objective evidence. The failure mode it names is sycophancy: an agent over-aligning with what it remembers about the user at the cost of getting the answer right.

If the measurement story is about what we check, the other thread this week is about what we hand the agent to begin with. “Stop Hand-Holding Your Coding Agent” (arXiv 2607.00038) takes the mid-2026 slogan (stop prompting the agent, design the loop that prompts it) and gives it a definition. A loop specification is a bounded, reusable artifact with a trigger, a goal, a verification step, a stopping rule, and a memory, handed to a harness like Claude Code or Codex so it pursues a goal without step-by-step nudging. The author hand-codes a corpus of fifty real loops and finds the practice has matured exactly where it says it should: 70 percent of loops verify in the autonomous zone of a five-level verification ladder, 74 percent name their terminal states, while automated triggering and durable memory remain underdeveloped. The honest part is the limits section, which names comprehension debt and cognitive surrender as the price of loops you no longer read.

The configuration data backs this up. A study of 2,853 GitHub repositories (arXiv 2602.14690) catalogs eight ways developers configure agentic tools across Claude Code, Copilot, Cursor, Gemini, and Codex, and finds context files dominate, often as the sole mechanism, with AGENTS.md settling in as the interoperable standard across tools. The advanced mechanisms the loop paper leans on (skills, subagents) are rare, and the skills that do exist mostly hold static instructions rather than executable scripts. Claude Code users reach for the widest range. The gap between what the research frontier describes and what most repos actually commit is wide, and worth watching as the standard artifacts settle.

Two pieces show what disciplined structure looks like at scale. Cognition described Devin’s security scanner as Agentic MapReduce (thread): map relevant signals across the whole repo, fan out focused agents over bounded shards, reduce their findings into one report, then re-verify serious vulnerabilities in isolated sandboxes before marking them confirmed. The verify-before-confirm step is the same instinct RigorBench rewards, applied to a task where a report is only trustworthy if the whole codebase was considered. And a 12-week first-person case study (arXiv 2607.01087) records one expert engineer building a document-accessibility system with frontier agents: 420 thousand lines of production code, 1.16 million lines of tests and tooling, 88 field notes. Its theory of “governance conversion” describes how high-velocity agentic work keeps surfacing the same structural failures, and how judgment turns each failure into a durable control rather than a one-off fix. Governance here is discovered from failures that only become visible during agentic work, not derived from a checklist written in advance.

The common thread across all six: the interesting questions have moved off the model and onto the scaffold around it, the loop, the verification ladder, the memory policy, the shard-and-reduce topology. Worth watching whether the benchmark builders follow, and start scoring those.