A Third of SWE-Bench Pro's Grades Don't Survive a Second Look

Run every task thirty times, apply a statistical test, and only 6.11% of the “performant” implementations shipped inside EffiBench, Enamel, EvalPerf, and Mercury turn out to be significantly faster than the canonical solutions they exist to beat. That is the finding in Rethinking Code Performance Benchmarks for LLMs, posted July 8, which re-ran 1,538 tasks across those four benchmarks under repeated execution rather than the usual single timing pass. The obvious reading is that language models write code about as fast as the reference. The paper’s reading is sharper, and worse for anyone who has cited these numbers: the benchmarks cannot see the difference. A manual pass over 308 statistically non-significant tasks found 99 where the performant implementation genuinely changed nothing, and 209 where a real improvement existed but the shipped test suite never exercised the path that would expose it. The authors then built a three-agent loop, one agent generating performance-oriented tests, one diagnosing them, one repairing them under a functional-correctness constraint, and on the 1,345 tasks where the original tests detected nothing, tests written with DeepSeek-v3.1 surfaced a significant runtime difference in 24.01% of them, and GPT-4o in 25.43%.

The measurement instrument was the bottleneck, not the model. That claim shows up twice more in the last day, from groups that do not appear to be talking to each other.

PERFOPT-Bench, also July 8, takes performance optimization seriously as its own agentic task rather than a property of a patch. Each of its seven tasks hands the agent a correct but deliberately slow codebase and asks it to move a target metric, which means profiling a real execution, diagnosing a bottleneck that may sit across layers, editing without breaking behavior, and then proving the speedup reproduces instead of being a measurement artifact. Scoring runs hidden correctness tests, a verified-speedup measurement, and a trajectory-level audit. Across seven agent stacks the result is that no stack dominates: speedup is workload-dependent, and swapping the agent framework underneath a fixed LLM materially changes that model’s per-task speedup profile. The authors are explicit that raw speedup is unsafe as a benchmark score, because some of the largest gains they observed came from agents exploiting shortcuts specific to the benchmark harness. The trajectory audit exists precisely to catch that. One exploratory result is worth more attention than its “pilot” label suggests: when a stalled optimization session is restarted from an externalized summary of what the agent had already learned, additional headroom becomes reachable that the original session could not find.

That last observation is a memory finding wearing a performance-engineering costume, and it lands the same day as a paper that argues the point directly. Remember When It Matters, from Wu and colleagues on July 9, names the failure mode “behavioral state decay”: in long-horizon tasks the decision-relevant state, meaning task requirements, environment facts, prior attempts, diagnoses, and still-open subgoals, accumulates across an expanding trajectory until it is buried deep in the context window or pushed past it entirely, and stops influencing decisions at the moment it is needed. Their intervention is a separate memory agent running alongside an unmodified action agent. It maintains a structured memory bank from the recent trajectory and, on each step, decides whether to inject a memory-grounded reminder or say nothing. Pass@1 improves 8.3 points on Terminal-Bench 2.0 and 6.8 points on τ²-Bench, for both weak and strong action agents. The ablations carry the argument: selective intervention beats passive bank exposure, beats always-on injection, beats advisor-only guidance, and beats general retrieval. Deciding when to remember is the hard part, not storing or fetching. They also train Qwen3.5-27B on SETA with SFT and GRPO and get partial transfer to Terminal-Bench, which is an early gesture toward memory policies that are not a frontier-model API call.

Retrieval is where the same “what does the index actually see” problem is being argued in public rather than in a paper. A team building coding tools posted their case against static code graphs on July 9, releasing LatentGraph, which folds AI-inferred relationships and Git history into the usual static analysis of symbols, imports, and calls. Their claim is that a call graph faithfully encodes explicit relationships and encodes none of the architectural intent, runtime coupling, local conventions, or the history sitting in commits and pull requests, and that the resulting index gave them the highest retrieval precision among the knowledge-graph approaches they tested. That is a vendor’s benchmark on repositories the vendor chose, with no independent replication, so treat the precision number as a hypothesis rather than a result. The interesting admission is buried at the bottom of the post: they threw away multiple graph designs because some relationship types added noise, while others they nearly ignored turned out to matter. Nobody has published a principled account of which edge types earn their place in an agent’s context budget.

Against all of this sits DeepSWE, released July 8, which is the most direct attack yet on how coding-agent benchmarks are constructed. The SWE-bench recipe mines merged fixes from public GitHub repositories and grades each attempt with the tests that shipped alongside the merged fix. Both halves are broken. The fix and its discussion were almost certainly in pretraining, so a high score can reflect recall; and the shipped tests were written to confirm one particular fix, so they will fail a correct alternative implementation and pass an incomplete one. DeepSWE’s 113 tasks are written from scratch against 91 active open-source repositories in five languages and are deliberately never contributed upstream, keeping their reference solutions out of the corpus that training scrapes. Each is graded by a hand-written verifier that checks the requested functionality and accepts any implementation providing it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE’s verifiers 1.4% of the time, against 32.4% for SWE-Bench Pro’s inherited tests. Roughly a third of SWE-Bench Pro’s grades do not survive a second look. DeepSWE’s prompts run about half the length of SWE-Bench Pro’s while describing tasks whose reference solutions touch 5.5 times more code, and frontier agents that cluster tightly on public leaderboards spread across a visibly wider band. The benchmark, the verifiers, and the full evaluation trajectories are released together.

The practical version of all this arrived on DevOps.com, where Veera Ravindra Divi’s shadow-mode CI writeup on July 9 works through what it costs to gate agent changes on behavior instead of on a green build. The pipeline builds the promotion candidate’s actual container image, stands up a throwaway Docker network with mock tool APIs, a fixture database, and a trace collector, replays workflow fixtures chosen for pressure rather than fidelity (ambiguous requests, permission denials, malformed payloads, duplicate requests, schema migrations), and then scores the resulting traces against a baseline. The promotion gate in the sample workflow is four numbers: pass rate at or above 0.92, tool-error rate at or below 0.03, zero policy violations, p95 latency under 12 seconds. Two of the constraints are the interesting ones. Behavioral metrics get compared as distributions across repeated runs, so one bad retry spike does not fail a build, while a hard safety violation such as calling a forbidden tool fails immediately on a single occurrence. And the agent under test never gets production model credentials, with inference routed through a semantic cache replaying approved responses for known fixtures, a pattern he calls an LLM VCR.

Four independent groups spent the last day arguing that the grader, the index, and the reminder are where agent capability is actually being determined, and that all three are currently unmeasured. The DeepSWE verifiers are public. The interesting experiment is not another leaderboard run but pointing an independent judge at the graded trajectories of every benchmark a team currently trusts and reporting the disagreement rate. If it comes back near 32%, the score was never the thing being measured.