An audit of 34 benchmark tools finds graders scoring what the tool call reported

A clinical agent benchmark ships a tool that tells the agent every write executed, under a documented no-write design its interface never discloses, and the grader takes that confirmation message as evidence. Its action success rate therefore records whether a request carried the expected payload, with no check that any record changed. That is the sharpest case in Do Agent Benchmarks Do What They Say?, posted today by Rohith Reddy Bellibatlu, Zichong Wang, and Wenbin Zhang, who treat each simulated tool’s advertised surface as an executable contract, check the implementation against it, and trace each score through task files and evaluator code to the verdicts that depend on state a defective tool should have written. Across 34 mutating tools in four benchmarks they confirm seven tool defects and one evaluator property at pinned commits. In tau2-bench no gold trajectory reaches either defect, but on 1,120 paths built to isolate the telecom one, the evaluator rewards a refuel of a suspended line and fails the repaired tool. On AgentDojo’s full 25-tool mutating surface, at least 5 tools diverge from what they advertise as the authors’ contracts read it. The paper is just as direct about its own checker: on injected defects it raised no false positive in 25 flags and missed most of the rest, with 29 of 33 scored misses covered by a contract clause that no probe exercised, and the static half alone flags 14 of the 17 confirmed sites, so the dynamic half confirms and traces more than it discovers. A defect under a score is present on every rerun, which means repeating the evaluation buys no protection against it.

Merged, Not Measured puts the same question to a signal practitioners trust more than any benchmark, the maintainer’s merge. Zhenyu Qi and colleagues filtered the 71,677 agent pull requests in AIDev v4 down to 1,262 performance issues fixed by six agents across 582 repositories, coded each issue and its tests, and re-executed 23 rejected and 30 merged fixes. Of the 30 merged, 18 met the authors’ delivery criterion, 3 fell short of the claimed speedup, 9 showed no significant gain or regressed, and 14 change behaviour on inputs no test covers. What predicts a merge is history: acceptance rises from 31-37% to 70% with the agent’s track record in that repository, and from 33% to 84% with the repository’s prior merge rate on its other agent PRs. The one content difference the coding detected is that merged fixes delete a larger share of the lines they change (0.26 against 0.15), a gap that holds within agent and within repository, while description, tests, and measurements show no detectable difference. Only 11% of fixes carry a performance test or benchmark, and 61% of rejections state no reason, though the unexplained rejections look defensible: only 6 of the 23 re-executed rejected claims held, in a three-run pilot on mostly agent-built workloads that the authors flag as such.

Pass rates hide a slower cost. Do Coding Agents Reuse Existing Code or Reinvent the Wheel? introduces RepoReuse, a multi-turn benchmark where requirements arrive turn by turn and the workspace accumulates, built by an automated pipeline of AST dependency graphs, guided evidence collection, and execution-verified task synthesis. Dongsheng Ma and colleagues score reuse rate, recall, and cross-turn structural redundancy beside the pass rate. Over 3,000 audited turns, agents progressively stop exploring relevant repository code, reuse their own earlier work less even when all of it sits in the workspace, and leave duplicated logic in 50.8% of task chains by turn 5, while pass rates barely move. Every duplicate is a future fix that has to be applied twice, and nothing in a functional-correctness score prices that in.

The multi-agent version of the gap shows up in AsynCodeBench, which represents each task as an explicit dependency graph with executable Dependency Checkers: 19 tasks from real repositories exposing 52 directed cross-agent dependencies. Two measures fall out of the tracking, the share of dependencies eventually satisfied (ADPR) and the step at which each first becomes satisfied (DRS). Across model families, scales, and generations, better coding scores do not reliably bring better collaboration, and task-level metrics diverge from the dependency-level ones. The trajectory analysis adds a detail orchestration designers can use: coordination tends to resolve in concentrated bursts the authors call hopping windows, many dependencies inside a short stretch of the run, so a progress monitor that expects steady dependency resolution can misread a healthy run as stalled.

Recovery benchmarks have their own ordinal blind spot. Same Winners, Different Success Rates takes aim at set agreement, the checkpoint-benchmark habit of asking whether independent runs pick the same best action. Dong Xu and colleagues prove that for equal-cost Bernoulli actions, success probabilities of (0.9, 0.8) and (0.2, 0.1) produce identical best-action-set distributions at every sample size, so no procedure that only looks at which action wins can tell a system that usually recovers from one that rarely does. When every action fails, they tie at zero and the runs agree with each other almost perfectly. On 864 frozen RecoveryBench episodes and two planning cohorts totalling 3,456 responses, agreement and held-out quality moved in opposite directions, and permuting checkpoint-to-action bindings changed 8 to 13% of cell-level conclusions. The proposed fix costs no new data: report agreement, the all-zero fraction, held-out success, and pooled success together.

A production team did this kind of evaluation audit on its own system. On Evaluating and Improving Conversational Agents in Production, from Kasra Hosseini and colleagues, describes the framework behind a large multi-agent shopping assistant and names three obstacles to offline evaluation: a logged conversation cannot be replayed against a modified system because one changed response changes every later turn, the unchanged system varies between runs as prices, inventory, and personalization signals move, and aggregate quality scores say that something changed without saying which behaviour. Their harness generates targeted assertions and a fixed cohort of customer scenarios for each reported behaviour, reproduces it locally with a simulator that writes new customer turns conditioned on the recorded messages, and stores repeated runs of the unchanged system as a baseline. An orchestrator then implements each hypothesis as an isolated modification and compares it to that baseline with paired percentile bootstrap intervals over scenario-level differences. Two of the reported findings are about the evaluation itself: a judge that lacked the evidence it needed, and a model setting that was configured but never applied.

The retrieval paper measures displacement inside the top-k. Financial Evidence Crowding studies passages that match a financial query’s topic while conflicting with its period, segment, metric scope, or table scope. With query, relevant evidence, ranker, candidate count, and budget held fixed, swapping in incompatible candidates cut Recall@10 by 0.147 against equally difficult compatible pools on a FinDER split of companies unseen in training. FinDeCrowd-RAG adds a learned score correction built from typed compatibility and local lexical competition, applied only when a query-level gate predicts a better order. Top-10 evidence inclusion rises from 0.757 to 0.902 on the controlled candidates and from 0.420 to 0.492 on a plain FinDER index, with top-100 coverage unchanged at 0.743 by design, and answer accuracy and citation recall improve on FinanceBench and FinQA with a fixed generator. It is the closest today’s pool comes to the governed-semantics question: the wrong fiscal period is a topical match, and relevance scoring alone cannot see the difference.

What to watch: whether the maintainers of tau2-bench, AgentDojo, and the clinical benchmark patch the confirmed tools or contest the contract readings, and whether any leaderboard starts publishing a tool-contract check beside its scores. For teams merging agent PRs, the cheaper experiment is local: require a benchmark in every performance PR and see how many of the claimed speedups survive three runs.

Source note: the code-intel mirror reported its last sync as 2026-09-01 (41,640 minutes stale), but items ingested through 2026-09-30 08:04 UTC were present and used here; the arXiv entries come from the 2026-09-28 through 2026-09-30 announcement batches. No item in the pool searched was a direct semantic-governance hit for this window.