Replay-corrected scoring cut one agent benchmark from 94.3% to 65.5% pass

Replay-based correction of a single reward-function defect dropped measured success on an LLM agent benchmark from 94.3% to 65.5%. Junjie Chen’s audit of evaluation harnesses (When Agent Evaluation Becomes Unreliable) treats the harness as software with its own bugs: five state-integrity invariants plus three known-answer controls, validated by injecting seven classes of oracle defect across two task domains and a synthetic one. Every injected class was caught and the repaired oracle produced no false positives. Pointed at a live benchmark, the audit found a reward function that mutates environment state before post-run scoring; across 4,320 episodes the count of database-pass labels fell from 4,194 to 2,883 once scoring was replayed against a clean terminal state. The second half of the paper is less flattering to anyone who thinks a score is a property of the model. A 3,240-episode ablation shows the agent’s policy documents, tool descriptions, and tool error strings all shift observed trajectory reliability, with tool descriptions the largest effect at minus 9.2 points, and the sign of the language effect flips in a second domain. The portability note deserves attention too: the method could not validate a second, independently built benchmark because the reference terminal state cannot be reconstructed from its recorded API-call log. If your harness does not log enough to rebuild end state, nobody can audit your numbers.

A companion failure shows up on the robustness side. Shahriyar, Iftee, Barton and Wright (What Counts as the Same Task?) rescore 14,336 stored agent runs under different rules for deciding whether a perturbed test case still tests the original task. Embedding-distance thresholds, the common choice, admitted 75 to 100% of edits that changed the task (swapping the payee on an invoice, for instance) while rejecting 24 to 45% of valid hostile-note injections, across four encoders. Under a MiniLM cosine threshold Mistral looked vulnerable on all 64 tasks; only 26 had a successful attack that was actually valid. Their fix is an executable task contract: which facts define the task, which edits are permitted, and what completion means, checked per test case before the agent runs. Read alongside the oracle audit, the pattern is the same. The gap between reported and real agent robustness is often a harness defect, not a model property.

The agentic-coding results today concern what agents leave behind and what they spend. Pan, Zhang, Wu, Zhang, Zhang and Zhang (SWE-Maintain) build 100 three-event maintenance episodes from 50 Python repositories with hidden tests certified across each sequence, then run each later task twice with the same agent: once from the developers’ preceding patches and once from the agent’s own earlier changes. Across five coding systems, two backbones and 1,996 paired events, starting from agent history cuts later-task resolution by 5.6 points on average, and by 14.0 when success also requires all earlier tests to still pass. The decomposition is the useful part. Two-thirds of agent-history starting states already fail earlier tests, and only 1.9% of events that begin there end with every accumulated test green. When the inherited code does pass earlier tests and the next task is not already solved, the disadvantage vanishes (+2.7 points, 95% CI from -0.5 to +6.1). The downstream cost of agent-written code is inherited breakage rather than a diffuse quality penalty, which argues for a hard regression gate on every agent change rather than a style review.

On spend, Kumar and Pandey (TurnCost) analyze 600 public coding-agent trajectories across two scaffolds and two models and find that the observation-compression literature is aimed at a secondary cost. Median tool observations run 118 to 286 tokens; the giant build logs used to motivate compression sit near the 99th percentile. With 98.6% of input tokens served from a prefix cache, flat input pricing overstates loop cost by about 7x. Turn count explains 84% of the variance in log trajectory cost; mean observation size explains 4%, and the most expensive trajectories have the most turns despite the smallest observations. Their closed-form break-even condition for delegating turns to a cheaper model predicts, and the data confirm, that delegation raises cost in half the evaluated settings. The authors are careful not to claim turns can be removed without hurting task success, which is the obvious next experiment.

Samway, D’souza, Darwiche Domingues, Fadaee, Yi-Chern and Dziri (Scaling Inference-Time Compute with Teams of Coding Agents) treat team design as a compute-allocation problem and vary one axis at a time: the stage a lead delegates (planning, implementation, verification), the model mix, and worker count. On ProgramBench, teams beat solo agents at three to seven times the cost, but a heterogeneous 17-agent team lands within 1.4 points of a strong homogeneous 5-agent team at 70% lower cost. Three findings drive that. Delegating only planning, with the lead implementing and verifying itself, matches full delegation at half the cost. Cheap models pay off as planners and not as implementers: a strong lead with cheap planners recovers 47.5% of the all-strong reward gain for 13.9% of the added cost, while cheap implementers reduce reward. And planners keep adding reward as the team grows, whereas implementers and verifiers plateau past four workers. That is a concrete allocation rule for anyone running lead-worker harnesses today: spend the extra budget on more and cheaper planners, keep the strong model on the diff.

For the semantic-governance charter, Luo, Wu and Xu (DSABench) remove the assumption nearly every text-to-SQL benchmark makes, that the target database is handed to you. Their 815 human-reviewed tasks, drawn from Spider, BIRD and LiveSQLBench, require the agent to find every database a request needs inside a catalog, work out how the sources fit together, and execute the workflow; 618 tasks need several databases and 105 admit more than one valid interpretation of the required sources. Executable contracts define source participation, cross-source dependencies and source-local execution, and a RouteEnv lets agents inspect public catalog evidence without leaking gold annotations. The numbers are low. Routing alone tops out at 60.55% across seven backends, and 34.66% once coordination requirements count. With execution required, median non-privileged success sits between 6.69% and 7.98% without clarification, 11.44% to 13.25% with one clarifying turn, and 16.99% in the best clarified configuration. On ambiguous tasks a single clarification lifts exact branch-source recovery by 39.8 to 43.2 points. Picking a plausible database and writing correct local SQL is not the hard part; carrying the right meaning across sources through the whole workflow is, and this is the first benchmark that scores that directly.

On retrieval, Ng S. T. Chong’s revised factorial study (Post-Generation Verification Dominates Retrieval Optimization, v3) crosses four RAG features (section expansion, agentic search, a completeness check, table-of-contents-guided retrieval) in a full 2^4 design: 16 configurations, 24 queries across eight interaction types, two models, 759 scored runs on five public documents of 78 to 492 pages. The completeness check is the strongest feature (d = +0.48, p < 0.001) and on its own scores 4.31 of 5, ahead of every configuration without it, including the three-feature stack at 4.11. Table-of-contents retrieval adds a significant d = +0.22 at zero extra LLM calls; agentic search is small, unstable and does not survive multiple-comparison correction; section expansion is neutral. The effects hold under two alternate judges, and the top configuration roughly doubles latency, so there is a real quality-latency frontier of six configurations. Feature utility is query-type dependent (the completeness effect reaches d = +0.83 on completeness-demanding queries), which is the methodological point: single-query-type evaluations mis-rank retrieval features.

What to watch: whether other benchmark maintainers adopt replayable terminal-state logging so their oracles can be audited at all, whether anyone shows turns can be cut from coding-agent loops without losing task success, and whether multi-database routing moves off single digits once agents are given a governed catalog rather than raw schema listings.

Feed note: the mirror reported its last sync as 2026-09-01, yet every item above was ingested on the morning of 2026-10-14; the staleness counter looks wrong, not the data.