Agent model rankings reverse when the harness changes, by up to 30 points on Terminal-Bench 4

On Terminal-Bench 4, Claude leads GPT by 7.94 points when both run inside OpenHands and trails it by 30.16 points inside PI. That reversal comes from Finding the Right Fit, posted today, which scores 66 configurations: four configurable harnesses (OpenHands, DeepSeek Harness, PI, and openJiuwen) crossed with five models on TUA-Bench, ALE-CLI, and Terminal-Bench 4, plus the native Codex-GPT and Claude Code-Claude pairings. For four of the five models the best harness changes from one benchmark to the next, a vendor’s own harness is not reliably the best home for its model, and spend does not track score: on Terminal-Bench 4, GPT scores higher under PI than under DeepSeek Harness at less than a quarter of the cost per task.

The matched trajectories supply a mechanism. Models start almost all of their own repairs, so the outcome depends on whether the harness hands a failure back in a form the model can act on. GPT does best in PI’s lean scaffold, while Kimi, which often issues malformed tool calls, gets its highest score on all three benchmarks in openJiuwen, by 5.61 to 11.11 points. The authors release the harness adapters, the evaluation code, and all 6,204 scored trajectories, so the grid can be extended by anyone whose model or harness is missing from it.

Agent Evaluation Reliability, from the batch a day earlier, measures what that reversal does to a leaderboard. The authors fit a Bayesian variance decomposition to 22 benchmarks from the Holistic Agent Leaderboard and the Harbor Index and separate two claims that usually share one table. Ranking fixed model-plus-scaffold systems is reliable, between 0.935 and 0.994. Ranking the underlying models, which is how most readers use the table, has reliability between 0.148 and 0.841, and scaffold effects vary substantially from one evaluation to the next. Adding tasks does not repair this: when uncertainty is dominated by limited scaffold coverage, infinitely many similarly constructed tasks raise model-ranking reliability by at most 0.097. Pooling diverse benchmarks does more, lifting projected cross-task reliability from 0.44 to 0.75 at the same task budget and cutting projected cost by up to 83%. A benchmark maintainer with budget for more tasks would learn more by running the existing ones under a second and third harness.

Code review by a second model depends on the same variable, what the reviewer is handed. Groundability, Not Scale Alone asks when a nominally weaker model can decide whether a coding agent’s patch solves its issue, using 411 execution-labeled traces from three agents and 101 controlled cases. Structured but unchecked evidence, tested on 154 GPT-5.4 traces, raises defect catch and over-rejection together. Official execution evidence, supplied as an upper bound, improves both rates for five of six reviewers on 122 held-out traces, and two reviewers classify every trace correctly; reviewer size is not a consistent predictor of quality. The deployable version is weaker. A frozen cascade built from patch-caused static errors and generated tests that first fail on the unpatched repository reaches catch of 0.76 and 0.80 on held-out GPT-5.4 and Gemini traces, with over-rejection at 0.66 and 0.67, and most false rejections occur when the checks leave a case unresolved and it falls to the reviewer. The bottleneck the authors name is producing checks as decisive as official tests for repositories that have none, and reviewer capability comes second to that.

Orchestration got a smaller result in a similar direction, with its limits stated by the authors. Can AI Scientists Coordinate at Runtime? introduces Runtime Agent Coordination, which selects agents from existing AI-scientist hosts during execution, assigns scoped work contracts, and adds artifact-grounded verification that informs later agents without blocking transitions. In a single-seed exploratory evaluation across Agent Laboratory, EvoScientist, and ARK on ResearchClawBench, with host models, tools, and permissions preserved, runtime selection yields the highest observed mean score for every host. Adding contracts and verification on top lowers those means. One seed supports a direction and nothing stronger, and the direction is that choosing who works next paid off under a fixed budget while the extra contract and verification machinery did not.

The data-agent side of the window produced two benchmarks that measure what text-to-SQL leaderboards skip. Argo-Bench simulates a New York City food delivery platform at true scale, 81 million orders in 2024 with fraud patterns and marketplace incentives, and exports it to an ERP warehouse of 235 tables and 7.5 billion rows modeled on the Oracle E-Business Suite schema. The simulator’s ground-truth state is withheld, so each of the 210 tasks requires reconstructing business facts from the warehouse before doing anything with them. Then the agent acts: it bans accounts it judges fraudulent, allocates courier incentive budgets, or issues back pay, and the grader scores each action by its consequences in the simulator. Every task ships with an executable reference solution that uses only the warehouse. The strongest of 14 frontier and open-weight models scores 95 or higher on 34.8% of tasks and averages 59.5 points. The authors built it this way because public text-to-SQL sets sit on datasets where a business event fits in a single table, and audits have found their answer keys frequently wrong.

BudgetSchemaBench isolates the step before that: how much schema goes into context. Over a pooled 80-database catalog it sweeps four schema-context budgets and three serializations while holding each retriever’s table ranking fixed, with relevance labels derived mechanically from gold SQL and a source-namespace check that rejects correct results obtained from the wrong database. Raising the budget from 2.5% to 50% of the catalog improves execution accuracy on 1,279 held-out questions by 18 points under lexical retrieval and 3 under dense retrieval, because the dense retriever already finds most required tables at the smallest budget. With the required tables guaranteed present, the three serializations differ by at most 2 points. The removal probe is the result that should worry anyone grading on execution accuracy: with the required tables deleted from context, 94.6% of the predictions that still come out correct name one of the missing tables exactly, consistent with the model reconstructing absent schema from parametric knowledge. A private warehouse gives a model nothing to recall, which suggests scores on public schemas overstate what transfers.

In retrieval, SkillSeek applies the standard two-stage recipe to agent skill selection. Open-source aggregations of SKILL.md directories have passed 230,000 skills, and the prevailing approach has the agent run an LLM-mediated loop that rewrites queries and refines candidates on every task. SkillSeek is a BGE-base bi-encoder feeding a small cross-encoder, exposed over MCP. On the 89-task SkillsBench under OpenHands, plain BM25 alone records a pass rate at or above the refined LLM loop in three of four settings, and the cross-encoder covers the difference in the fourth; a first-stage recall ceiling explains the pattern. Per-trial spend falls from $51.30 to $27.54, within fifty cents of the no-skill baseline.

What to watch: whether any public agent leaderboard starts reporting per-harness results, since the reliability paper locates the uncertainty there and the fit paper has 6,204 trajectories showing its size. Argo-Bench’s abstract reports agents working against the raw warehouse; the same 210 tasks with a governed semantic layer in front of the 235 tables would show how much of the distance from 59.5 comes from not knowing what the business means. BudgetSchemaBench’s table-removal probe is cheap enough to run against any text-to-SQL result that claims enterprise relevance. The runtime-coordination result needs more than one seed before anyone designs around it.

Source note: the code-intel mirror reported its last sync as 2026-09-01 (44,520 minutes stale), but items ingested through 2026-10-02 07:21 UTC were present and used here; the arXiv entries come from the 2026-09-30, 2026-10-01, and 2026-10-02 announcement batches. Only abstracts were available in the feed for these papers, so every figure above is an abstract-level figure.