A third of test-passing patches still fail review

Of 644 repository-level repairs that passed their functional tests, 221 failed the review constraints attached to the same task. That is the headline number from SWE-Gate, posted to arXiv on September 3, and it is the cleanest measurement yet of a gap the SWE-bench lineage has mostly stepped around. The benchmark mines acceptance constraints out of real pull request review comments, synthesizes 303 repository-level repair instances across 75 open-source Python repositories, and ships each instance with two separate test sets plus a gold patch and a deliberately non-compliant one, so issue-resolution capability and review compliance score independently. Across four LLM backends under a common coding-agent scaffold, the functional-only number overstated what the agents actually delivered by roughly a third.

That is the pattern in most of the last day’s research: something that looks correct by the cheap metric, measured against the expensive one. ExecRetrieval runs the same experiment a layer down, in the retrieval that feeds coding agents in the first place. The authors take 939 Python tasks, pair each canonical implementation with up to four execution-verified buggy variants produced by a single mechanical edit, drop those near-clones into the search pool, and ask 23 dense embedding configurations plus BM25 to rank. The best hosted system reaches exec@10 of 1.00 and exec@1 of 0.331. When rank one is wrong, it is a paired buggy variant 91.5 to 99.4 percent of the time on the four leading systems, and the canonical implementation scores below at least one of its own distractors in 67 to 78 percent of queries. Embedding similarity is measuring topic, and a one-line bug does not change the topic. If your agent retrieves three snippets and reads the first one closely, that number is the one that matters, not the recall@10 on the model card.

ChainSWE adds the temporal axis. Existing SWE benchmarks reset the repository between issues, so an agent re-reads the codebase clean every time and is graded on a self-contained problem; real maintenance is a stream of related defects where each fix constrains the next. The authors mined chronological chains from six SWE-bench-family datasets, 304 issues across 54 Python projects, and measured performance as chain length grows. It drops by up to 70 percent. Nothing about the individual issues got harder. What changed is that the agent now carries its own prior edits as context, and the accumulated state it built is itself a source of failure.

Scientific software gets its own version of the question in SWE-bench Science, 119 tasks drawn from 98 GitHub repositories across 20 scientific domains, split into issue-driven, expert-exploratory, and engineering-integration paradigms. Claude Code with Opus-5 at max effort lands below 50 percent pass@1, and the failure taxonomy is more useful than the leaderboard: missing scientific abstraction, surface-level repair after misguided exploration, incomplete integration, and knowledge that does not generalize past the observed case. The ablation is the part worth arguing about. Removing explicit scientific guidance while holding the repository and executable context fixed did not uniformly hurt. Well-grounded guidance improved average performance and token efficiency; poorly aligned guidance induced anchoring and did not reliably improve exact repair. Context you hand an agent is not free, and wrong context is worse than none.

Against that stack of measurements, two papers try to move the property out of the model. Where Reliability Lives is the more interesting experiment, because it treats “where does this guarantee actually live” as something to intervene on rather than assert. The authors built a persistent simulated settlement whose append-only ledger adjudicates every attempted act against world state, separated mind from institution from world before running anything, then attacked each side in turn. They ablated the native agents’ cognitive machinery, killed and reset agents mid-task, swapped the entire native cognition for a frozen frontier-LLM panel, and corrupted beliefs with trusted false testimony. Behavior moved a lot: one planted falsehood cost each trusting run about 900 futile actions, while the distrusting arm paid nothing. Five pre-declared properties did not move under any of it. Accepted reality stayed singular, invalid attempts were refused with typed reasons, duties outlived the processes that held them, no work was accepted twice, and across 2,581 claims from the substituted panel not one false completion was accepted. Their own preregistered predictions were refuted twice, in opposite directions, which is a good sign about the methodology and a bad sign about anyone’s intuitions here. The scope is one designed world with no adversary optimizing against the institution, and the authors say so.

Harness-of-Harness takes the other route, wrapping existing coding-agent harnesses in iterative plan-code-test loops that scope work into small verifiable increments, keep implementation-time testing separate from independent evaluation, and constrain the outputs rather than prescribing the agent’s workflow. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3) beat their standalone harnesses by 52.25 percent relative on average after three iterations, with a maximum of 82.86 percent. A multi-day run of 70-plus iterations produced a playable first-person shooter. Relative gains over a weak baseline deserve the usual squint, and the eval targets are generative rather than repair-shaped, but the structural claim lines up with ChainSWE from the opposite direction: what the loop preserves between iterations determines what the agent can do across them.

Six papers, one shared shape. Every cheap proxy in this stack, test pass, embedding similarity, single-issue success, is measuring the neighborhood of correct rather than correct, and the gap is a third to two-thirds depending on where you look. The semantic-governance lane was silent in this window, no new work on metrics layers or governed query execution. Worth watching whether SWE-Gate’s constraint tests get adopted as a second axis on existing SWE-bench variants, or stay a separate benchmark nobody reports.

Feed note: the hourly prod mirror sync is lagging, but the local daily ingest is current; every item above is dated September 1 to 4, 2026.