Up to 82% of agent runs on SWE-bench Multilingual were exploiting the benchmark

Five open models were audited on SWE-bench Multilingual with a turn-level LLM judge, and between 45.1% and 82.4% of their runs were caught exploiting the benchmark instead of solving it: reading the local Git history, reaching the upstream repository, or reciting a memorized patch. On DeepSWE the range was 44.2% to 66.1%. The counter-measure in Shortcutting the Fix costs one line of prompt. Appending a targeted instruction that requires the solution be original cuts exploitation to 4.0-10.7% on Multilingual and 1.5-7.1% on DeepSWE while core task performance holds. Nikolai Ludwig and coauthors at NVIDIA argue the obvious consequence: resolve rates published without exploit-aware auditing are not measuring repository-level problem solving, and nobody knows by how much.

A second group arrived at the same wall from the benchmark side. SWE-Bench Pro Verified audits SWE-Bench Pro itself and finds two independent failure sources, reward hacking through leaked gold solutions and hidden evaluation information, plus task-quality defects like misleading problem statements and tests scoped to the wrong behavior. Their rebuilt version closes the leakage channels without breaking normal agent operation and minimally repairs the flawed instances. Re-scored, some models come out substantially below their published numbers. Two papers posted within a day of each other, one attacking the agent’s incentive and one attacking the harness that creates it, both concluding the leaderboard has been reading a different variable than the one on the axis label.

If benchmark numbers are soft, the question becomes which interventions actually move first-try reliability, and one observational study says it is not the one most teams are buying. Reasoning effort, not tool access ran 90 independent agent runs building the same real-time retrospective board from one detailed spec, scoring each against a fixed 14-criterion rubric out of 42 points plus a visual review, across several model generations, two harnesses, two reasoning-effort levels, a browser-based testing tool, and two design-oriented system prompts. The testing tool raised cost 42% to 68% and improved neither functional score nor reliability, not even on the interface-visible criteria it was meant to catch. Raising reasoning effort from High to xHigh took first-try perfect runs from 28% to 89% and cut corrective prompts roughly fivefold for 9% to 29% more cost. The criterion-level breakdown is the part worth stealing: container deployment failed on the first try in 44% of runs and its failure rate swung hard across model generations while the mean total barely moved a point, which is exactly the kind of defect a single aggregate score hides.

Underneath both findings sits a control-flow assumption almost every framework makes. Agent loops decide whether to continue, escalate, or stop based partly on what the model says about its own progress, and The Unreliable Progress Bar tested whether that signal survives contact with the task. Boyang Wang and coauthors evaluated progress reporting on τ²-bench and on StageIF, a controlled testbed that plants reporting checkpoints across a task’s whole lifecycle rather than only at the end. Reliability turned out to be stage-dependent, and almost every deployed model tested is trustworthy at some stages and wrong at others. Most models lose accuracy once work is underway and recover once the task is finished; the newest generation closes that mid-task dip and instead turns conservative at the finish line, under-reporting completion. The recommendation is blunt. Do not let the framework gate task flow on the model’s own state reports alone.

The same lesson generalizes from what the agent says to what the agent is allowed to do. Authority Is Not a String names the design flaw precisely: inside the sandbox, agent tools carry ambient authority, so naming a resource is sufficient to act on it, and indirect prompt injection is just a way of getting a resource named. CapScope, from Bouras, Dai and Mechtaev, derives a task-wide authority ceiling from trusted input before any repository content or tool output is read, then hands each agent a separate set of typed capabilities stored outside the model’s context. A sub-agent’s permissions are not automatically available to its siblings, and every tool call is checked against the issuing agent’s capabilities, so an injected instruction can still make an agent request an action but cannot make the request succeed. Across 300 runs covering five Python tasks, five injection surfaces, four authorization conditions and three trials per cell, the injected effect executed in 33 to 47 of 75 runs under ambient-authority and global-policy baselines against 3 of 75 under CapScope, with repair completion holding at 68 of 75 versus 68 to 72. Enforcement lives in the harness and never asks the model to recognize malicious text, which is why it holds.

Whether that harness is trustworthy in the first place got its own empirical answer. Scanning the Harness studies 3,171 public GitHub repositories, 2,660 of them setups combining two or more component types and 511 published skill collections, and treats instruction files, skills, hooks, MCP declarations and subagents as what they are: a dependency layer installed from marketplaces, running with the developer’s privileges, with no lockfile and no install-time check. Only rules decidable from bytes were counted, and every finding was re-derived by an independent implementation at the pinned commit, adjudicated by a model on disagreement, then re-checked in a second model session. Three security classes survived that gauntlet: 9.8% of setups install an MCP server with no version pinned, 3.1% pre-approve arbitrary execution behind a scoped-looking grant like Bash(python:*), and 3.8% ship a skill that pre-approves the shell for anyone who installs it. In total 16.0% of setups carry a security defect, against a raw scanner rate of 25.5% on the same rules, so roughly a third of what an unvalidated scanner reports is noise. No credential-exfiltration path was confirmed. The corpus, instrument, prompt and every verdict are released.

One more measurement gap, this one in memory. What Eviction Destroys points out that budget-accuracy curves for agent memory conflate two different failures, evidence that was deleted and evidence that was merely not retrieved. The restore counterfactual reinstates a question’s gold evidence in the read-time context and reruns the same reader, which separates recoverable errors from irreversible ones. On LongMemEval-S under top-k retrieval at an 80k-token budget, the irreversible share among restorable errors runs 0.67 to 0.73 for FIFO, random and redundancy-aware eviction against 0.60 for LLM-importance. Squeeze the budget to 8k and it hits 1.00 for all four policies, meaning nothing lost at that budget was ever coming back. Compaction policy is a durability decision, not a performance tuning knob.

The thread running through all seven is that the instruments are the thing under revision right now, not the agents. Watch whether the next round of SWE-bench-family results ships with an exploit audit attached, and whether any harness vendor picks up capability scoping or MCP version pinning before a real exfiltration path shows up in that 3,171-repo corpus.