Failed ML reproductions stop with 71% of the GPU budget unspent

In RECLAIM, a new benchmark of 100 NeurIPS 2025 papers, the agents that fail at reproducing a result use on average 29% of their GPU-hour budget before they stop, which means they quit with most of their compute unspent, and the most common error across 400 runs (63 of them) is writing the method without checking any piece of it against the numbers in the paper. The best agent reproduces 41% of papers when the authors released code, data, and weights, 27% when it has to retrain, and 15% when it has to write the code itself. Every agent does worst on the tier with no code. RECLAIM’s grader also reads logs and outputs rather than the agent’s own report, which is the design choice that ties together most of what landed in the last day: the evidence an agent leaves behind has to be checked independently of what the agent says about it.

That idea is the whole premise of SWE-Prometheus, a benchmark that drops the usual SWE-style setup of a human-written issue and a test to pass. Each of its 60 repositories comes as a frozen snapshot with an open-ended goal: improve the repo’s engineering governance across six dimensions, among them tests and CI, quality gates, documentation, reproducible environments, and dependencies and security, then verify that nothing broke. Ten models on a shared 22-repo subset score between 0.057 and 0.576 on its Normalized Governance Improvement metric, with behavior-breakage rates from 0% to 23%. The useful control is a repository-blind template, which earns a mean NGI of 0.272 on a frozen ten-repo batch by stamping out CI configs, gates, and docs. It improves reproducible environments and dependency security on zero repositories. That gap between adding governance artifacts and making execution-backed improvements is the thing the benchmark measures, and it is why the authors insist on reporting improvement, breakage, evidence quality, and coverage together. Rankings flip with that choice: the two top systems look similar on common-valid scores, but once behavior failures count against the full pool, Kimi-K3 comes out ahead.

The configuration those agents run under is its own supply chain. Scanning the Harness, revised in the latest arXiv batch, audits 3,171 public GitHub repos (2,660 assembled agent setups and 511 skill collections) for exposures in instruction files, hooks, MCP declarations, and subagent definitions. Unpinned MCP packages appear in 9.8% of setups, broad execution grants in 2.5%, and blanket skill tool preapproval in 3.8%; together they cover 15.4% of setups. Among setups that declare MCP servers at all, 24.5% pin nothing. The paper is careful about its limits: it measures what repos declare, not runtime harm, and it reports a permission exception that fooled both the scanner and its mechanical audit, which argues for version-specific semantic checks instead of one static rule set.

Persistent agent memory gets the same treatment in Scope Before You Persist. On a 12-round code-repair stream, a frozen model edits its own skills behind an execution-grounded acceptance gate. With global memory, every accepted edit is available to every task family, and mean hidden trajectory utility lands at 0.713, below the 0.775 of an agent that never learns anything. Retrieving each accepted skill only for the family that earned it raises utility to 0.816 and takes harmful deployments from six of eight to none. Across 27 randomized streams the scoped variant accepts 63 updates against the global variant’s 12, with zero harmful acceptances. A gate tells you an edit is supported; it says nothing about where that support holds, and a locally valid edit leaking into an unrelated family is enough to make learning worse than not learning.

Evals built by coding agents fail in a recognizable way. An Empirical Study of Automating Agent Evaluation, now at v3, finds that frontier coding assistants asked to build an agent evaluation from scratch execute successfully 30% of the time and produce over-engineered suites averaging more than 12 metrics per agent. Their EvalAgent packages evaluation expertise as skills (procedures, reusable code, retrieved API docs), and on a 20-agent benchmark it lifts Eval@1, whether the generated eval runs and yields meaningful results on the first try, from 17.5% to 65%, with 79.5% expert preference. Pull the skills out and Eval@1 falls back to 30%, which puts the missing ingredient in evaluation procedure rather than in coding ability.

Cua’s approach to computer-use grading, described in CodeRabbit’s interview with Francesco Bonacci, adds the step most eval pipelines skip: testing the grader. Each Cua-Bench task has a known start state, a reference solution, and an evaluator that inspects saved application state; in a KiCad circuit-design benchmark, the evaluator compares the saved circuit’s components and connections to a reference. Before an environment ships, one agent attempts the task honestly while another tries to earn credit by violating it, and a reviewer checks that the evaluator rewards the first and penalizes the second. The best of seven models in Cua’s June run completed 6 of 25 KiCad tasks within 200 steps; drag and scroll are, in Bonacci’s words, “actually hard for agents even today.”

On the retrieval side, A Systematic Multi-Domain Evaluation of Document Retrievers runs 33 sparse, dense, and expansion-based retrievers across seven datasets off the shelf, using only configurations reconstructable from public docs and a uniform compute budget. NV-Embed-v2 wins four of seven datasets at a heavy latency cost; SPLADE-v3 nearly matches it at much lower latency and takes the top score on MS MARCO; GritLM leads on instruction-following sets. For anyone choosing a retriever for an agent’s context layer, a well-tuned sparse model remains the default to beat before paying for a large dense encoder.

What to watch: RECLAIM plans to rebuild from each year’s conference papers, so the next edition will show whether the “stop early without checking the numbers” failure responds to harness changes or only to model changes. The same question applies to SWE-Prometheus’s template baseline. If a repository-blind template keeps scoring 0.27, the next round of governance agents has to show gains on the dimensions the template never touches.

---

Source note: the code-intel mirror reported its last sync as 2026-09-01 (about 37,300 minutes stale), though items dated through 2026-09-27 were present and used here. arXiv posts no announcements on weekends, so the newest research in this window is the 2026-09-25 batch. No items on semantic governance or agentic analytics turned up in the window.