288 runs say your AGENTS.md doesn't move the pass rate

Across 288 evaluated runs on two frontier agents, changing the context-file strategy moved correctness by nothing you could measure. Prakhar Khatri’s two-agent ablation, posted yesterday, ran 17 tasks mined from merged pull requests in three Python repos (pdm, firebase-admin-python, opshin) under three conditions: no context at all, the full AGENTS.md injected into the system prompt every turn, and a topic-organized wiki the agent retrieves on demand. Claude Code (sonnet-4-6) passed 53.3%, 55.6%, 55.6%. Codex CLI (gpt-5.5) passed 58.8%, 56.9%, 52.9%. Equivalence testing bounds every pairwise difference below 10pp for Claude and 15pp for Codex, and a failure triage explains why: the near-misses fail on implementation skill, not on repository facts a markdown file could have supplied. One task needed a subtle union-expansion optimization and got a correctness bug. One needed proactive auth-token refresh and got reactive retry. One needed a validator check wired to the right rule, and the agent knew the rule from the code but miswired it. A pre-registered probe closes the loop on the two convention-closest near-misses: the real, unmodified AGENTS.md never converts a failure into a pass on either agent, 18 runs out of 18 for Codex.

One effect did survive, and it is the one worth copying. opshin is the only repo in the sample whose context file warns that the full test suite runs longer than twenty minutes. Give Claude that warning and its blind full-suite pytest invocations drop from 3.67 per cell to 2.44 under always-on injection to 1.67 under selective retrieval, with wall clock falling from 2689s to roughly 2030s. The file did not make the agent better at the task; it stopped the agent from spending twenty minutes discovering an operational fact a teammate would have said out loud. That is the context that earns its tokens: something the agent provably cannot derive by reading the source. Style guides and architecture prose, in this sample, bought nothing on pass rate. Khatri is also candid about the measurement floor, which matters more than the headline: at 17 tasks and 3 repeats the minimum detectable effect sits above 30pp, and resolving a 10pp difference at 80% power needs 120 to 200 tasks. More repeats barely help, because task-level variance dominates. Most published AGENTS.md comparisons, including the ones this paper reconciles, are running well under that bar.

Which makes task supply the binding constraint, and that is what Change2Task attacks. The Microsoft-led team converts merged PRs into verified, executable tasks on current healthy revisions of the same repository instead of pinning each task to its historical commit, reconstructing the pre-change state through patch reversal, code mapping, or agent reconstruction. From 1,130 eligible source changes they get 79.6% verified construction success across bug fixes, feature additions, test generation, API migration, and security repair, recovering 29.2% more verified tasks than a PR-based baseline on a matched candidate set. Historical and reconstructed variants of the same task agree on agent outcomes up to 98.0% of the time, and reusing modern bases cuts measured pipeline expenditure by 10.8%. The environment reuse is where the savings come from: most of the cost in benchmark construction is standing up and storing one container per historical commit.

Completion tells you less than it appears to. AgentS4D built 328 risk-injected workspace tasks and ran all 20 combinations of four harnesses (Hermes, OpenClaw, Claude Code, Codex) against five backends (GPT-5.5, Gemini 3.1 Pro, DeepSeek-V4-Pro, MiniMax-M3, Qwen3.7-Plus), for 6,560 runs. 4,461 of them, 68.0%, tripped a prespecified unsafe signal, and 4,344, 66.22% of the total, were unsafe and finished the task. Two findings deserve to change how people build these evals. Safety varied by harness-LLM pairing rather than by model, so a per-model safety card describes a configuration you are probably not running. And the same target harm produced different behavior depending on which carrier delivered it, which means a suite that tests one form of a risk reports a clean bill for the forms it never tried.

Retrieval improvements are landing in the same place: cost, not pass rate. OwlPath puts an OWL2 ontology layer over CodeGraph, using SPARQL property-path queries to pull transitive callers, subclass chains, and interface implementations in one hop instead of letting the agent grep its way through them, plus a precomputed 3KB map of module trees and core APIs to aim the first query. On 18 SWE-bench Pro instances the strict-apply rate moves from 66.7% to 68.4%, which is noise at that sample size. The token and runtime numbers are not noise: 28.8% fewer tokens and 39.5% less wall clock. Offline, recall goes from 0.226 to 0.464 and hit rate from 59.7% to 88.1% over 67 instances, and on a 37-question structural benchmark recall climbs from 4.4% to 28.8%. Structure-aware retrieval is buying speed and budget on tasks the agent was going to solve anyway, and the honest read is that the remaining failures are the implementation-skill failures Khatri catalogued.

The tooling is starting to reflect all of this. Simon Willison and Prime Radiant released smevals yesterday, a small-suite eval runner whose stated axes are models, harnesses, and prompts, which is the correct unit of analysis if AgentS4D is right that the pairing carries the behavior. It runs from uvx smevals docs, and Willison notes the hardest part of the build was settling the vocabulary, which tracks with how inconsistently this field names its own primitives. On the orchestration side, qm, billed as a multiplayer agent harness for work, took 514 points and 108 comments on Hacker News within hours of posting, which says more about appetite for shared multi-agent workspaces than about the tool itself.

Watch whether anyone runs the 120-task version of the context-file ablation, and whether the answer changes when the context is task-specific rather than a general style guide. That is the experiment Khatri explicitly did not run, and it is the one every team writing an AGENTS.md is implicitly betting on.