Review constraints reject a third of agent patches that pass their tests

Of 644 agent-generated repairs that passed their functional tests, 221 failed the review constraints attached to the same task. That is the headline number from SWE-Gate, a repository-level benchmark of 303 repair instances across 75 open-source Python repositories, and the construction is what makes it interesting: the acceptance criteria are mined from the pull-request review comments that actually accompanied each change, then shipped as a second test suite alongside the functional one. Passing the first suite says the agent understood the defect. Passing the second says it understood how this particular codebase expects defects to be fixed, and roughly a third of the time it did not.

A week’s worth of new work converged on the same complaint from different directions. The resolve rate, the number every coding-agent announcement leads with, is measuring a narrower thing than the field has been reading into it.

Benchmark inputs stopped resembling the requests agents actually get

RealSWE put a number on the gap between a curated GitHub issue and what a person types into a chat box. Comparing real prompts from SWE-chat against problem statements in SWE-bench Verified and Pro, the authors find that requests carrying only a problem statement, alone or with thin additional context, are 88% of real prompts and 7% of benchmark problems; 87% of real prompts are casually written against 94% of benchmark problems that are formal. They then built 381 task families whose variants share a gold patch but differ in information composition and style. Realistic inputs cost 6.4 percentage points of resolution on average across seven models, and they reorder the rankings. The useful finding for anyone writing prompts is which fields carry the signal: stating the desired behavior and the motivation moves performance, while environment information and reproduction steps add tokens without measurable benefit.

DEPBENCH takes a different slice of realism, 203 real dependency-upgrade tasks across five package ecosystems, each carrying code-level breakage that the upgrade never announced. Signatures change, types shift, runtime semantics drift, and none of it appears in the changelog the agent would read. The best completed configuration solved 104 of 203, 51.2%, with the variance spread across harnesses, models, and ecosystems rather than concentrated in any one of them. Dependency maintenance is a large fraction of what real teams pay engineers to do, and a coin flip is the current state of it.

JetBrains published the most concrete argument that a single score hides too much. Their model-profiles evaluation ran Claude Opus 4.7 and Gemini 3.5 Flash inside Junie across 523 tasks and got 267 resolved (51.1%) against 254 (48.6%), with identical outcomes on 430 of them. The trajectories behind those near-identical scores were not close. Opus averaged 184 steps at USD 2.79 per run; Gemini averaged 271 steps at USD 1.24. Gemini was assessed with moderate or severe hallucination in 37.3% of runs against 130 runs for Opus, and with significant or severe patch redundancy in 15.3% against Opus at 6.5%. Opus, in exchange, ended 123 runs with no executable validation at all, including 68 runs that still resolved the task. Extending to four models on 522 shared tasks, GPT-5.5 led on resolve rate at 51.5% and never finished a run without an executable check, while Qwen 3.6 27B FP8 resolved 38.9% at three percent of GPT-5.5’s per-run cost. Two models four tenths of a point apart on a leaderboard were doing recognizably different jobs.

The harness moved more than the model did

Same Model, Different Harness is the cleanest demonstration this week that the scaffold belongs inside the unit of evaluation. Holding the model and the task fixed, the treatment condition mechanically shortens older tool results as the context fills and reacts to repeated or stalled work; the control keeps the full conversation in time order. On a tight-window SWE-bench Verified cohort of 169 tasks with a 20,480-token window and a fixed 480-second attempt endpoint, mean per-task fail-to-pass fraction went from 28% to 49%, and complete solutions from 43 to 72. The same frozen treatment, with no model-specific retuning, lifted both endpoints for three additional models of different design. Under a wide window the arms converge, which is the boundary of the result and also the practical one: context management is worth the most exactly where operators feel it.

HarnessDev pushes that observation one step further and asks whether models can build and evolve the infrastructure they run inside, treating harness construction as a measurable capability rather than an experimental constant. Given how much of the SWE-bench spread is now attributable to scaffolding, an agent that can improve its own scaffold is a different kind of artifact than one that can only fill a fixed one.

Between those two sits a cost nobody has been reporting. Handoff Debt interrupts a coding agent at deterministic points, freezes the repository, and hands the partial state to a successor under four different views: repository state only, raw trace, summary notes, structured notes. From 75 source tasks the protocol generates 181 handoff-point tasks and 724 takeover runs per successor model. Context-bearing handoffs cut median agent events by 20 to 59% and cumulative prompt tokens by 42 to 63% against a repository-only takeover. Solved-rate effects are smaller and depend on the model, but the efficiency gain is consistent across all three successors tested. If your pipeline reassigns work between agents, between sessions, or between an agent and a person, the predecessor’s notes are load-bearing infrastructure and currently nothing measures whether they are any good.

Twenty thousand real sessions describe a different failure distribution

The largest field study of the week annotated 20,574 coding-agent sessions from 1,639 repositories across IDE and CLI workflows, using developer pushback as the visible marker of a breakdown. How Coding Agents Fail Their Users reports that 90.50% of episodes impose effort and trust costs rather than irreversible damage, which sounds reassuring until you read the companion number: 91.49% of visible resolutions still require explicit user correction. The agent rarely destroys anything; it also rarely recovers on its own. Two categories are growing as a share of the total even as overall misalignment rates decline, and they are the two that most resist automated detection: constraint violations and inaccurate self-reporting. An agent that misstates what it did is exactly the failure a test suite cannot catch.

A companion study mined 3,553 SWE-chat sessions for a phenomenon requirements engineers named decades ago and coding-agent benchmarks define away by construction. Requirements After the First Edit tracks post-implementation requirement arrivals and links each one, where repository state can be replayed, to deletion or replacement of prior agent-authored lines. A requirement’s arrival is followed by roughly twice as much invalidation as matched non-requirement edits. That burden shows no detectable decline across a session, so agents are not learning the user’s real constraints as they go. A controlled experiment found that delaying disclosure relocates implementation to after the reveal, while warning the agent in advance produced no detected effect on overwriting. Curated benchmarks fix the requirements before the first line is written, which removes the dominant source of rework from every score they produce.

Orchestration acquired a cost column

GitHub shipped Project HydraFusion as a research preview in Copilot CLI, and the write-up is more useful than the announcement. For each request the runtime picks one of three execution patterns: single, cascade (an efficient model drafts, a quality gate decides whether to escalate), or critique (a read-only critic from a different model family reviews, the drafter revises once). Against Claude Opus 5 at matched medium reasoning, the tuned configuration reached +4.9 points on TerminalBench 2.1 at 67% lower estimated cost, came within 1.5 points on DeepSWE at 36% lower, and within 0.1 points on their internal CheckpointBench at 65% lower. The engineering constraints are the part worth stealing: cost accounted across every leg including retries and fallbacks, explicit timeout and cancellation per leg, review steps run in tool-less isolated contexts so a critic cannot touch the repository, and no patch applied at all when a workflow is cancelled or fails validation. Routing policies were selected by beam search against a frozen baseline rather than by hand-tuned thresholds.

ArcticSwarm attacks the opposite failure in multi-agent work: parallel agents that read each other’s partial findings converge on an early candidate before the alternatives have been tested. Its answer is gated isolation, letting selected search tasks keep their own prior while others publish to a shared bulletin board, plus structured review at three commitment boundaries. On the full BrowseComp-Plus set with open-weight Qwen 3.5-27B it reaches 82.6%, against 78.8% with gated isolation removed and 74.5% with structured review also disabled, over an aligned MiroFlow baseline at 70.6%. On live-web BrowseComp with GPT-5 it reaches 73.6% against a reported provider system at 54.9%. The ablation is the argument: most of the gain comes from restricting what agents can read from each other, not from adding more of them.

Evaluating any of this is now expensive enough to need its own research. PTA-IRT fits an item-response-theory model over execution trajectories rather than pass/fail matrices alone, using explored context, attempted edits, and solving paths as privileged information for choosing a calibration subset. Under low calibration budgets it beats prior IRT baselines on both score and ranking recovery across four SWE benchmarks. Subset-selection methods that only see outcomes are throwing away the signal that distinguishes a near-miss from a run that never found the file.

Retrieval that ranks near-clones above the correct implementation

ExecRetrieval is the sharpest result of the week and the most uncomfortable one for anyone shipping embedding-based code search. The benchmark plants execution-verified counterfactuals in the pool: 939 Python tasks, each with one canonical implementation and up to four buggy variants generated by a single targeted mutation, so a retriever’s ranking can be tested for functional discrimination instead of topical similarity. Across 23 dense embedding configurations plus BM25, evaluated with paired McNemar tests and query-level bootstrap intervals, the top hosted system reaches exec@10 of 1.00 and exec@1 of 0.331. When rank one is wrong, it is one of the paired buggy variants 91.5 to 99.4% of the time across the four leading systems, and the canonical implementation scores below at least one of its own distractors in 67 to 78% of queries. Embeddings find the right neighborhood reliably and cannot order it by correctness at all. Every agent that reads the top-1 result and moves on is inheriting that error rate.

The same grounding problem shows up on the structured-data side. text2ql puts a language-agnostic intermediate representation, QueryIR, between natural-language intent and the physical query, with pluggable renderers for SQL and GraphQL and a runtime confidence score in [0.15, 0.97] attached to every generated query. Its deterministic zero-LLM mode hits 100% execution accuracy at a 3.2 ms median latency with no API cost; the LLM-backed mode reaches 62 to 70% exact match and 84 to 91% execution accuracy on 50-query samples from Spider and BIRD, with full-set evaluation still pending. The ablation names the lever plainly: schema-aware prompting contributes +18.4 percentage points of exact-match gain over a schema-free baseline on both benchmarks. Separating semantic intent from rendered query is what makes both the confidence signal and the multi-target support possible, and it is the same architectural move the semantic-layer vendors have been arguing for.

Reflect-SQL works the retrieval end of that pipeline, targeting the three failures that stall enterprise text-to-SQL in practice: obscure and large schemas, ineffective retrieval of the relevant tables and columns when the schema is structured and the user’s question is vague, and generated SQL that is syntactically or logically wrong. Schema linking, not SQL syntax, is where these systems lose.

What to watch

The evaluation literature has now produced enough evidence that the harness, the prompt shape, the review constraints, and the handoff notes each move results by double-digit percentages. What it has not produced is a reporting convention that carries any of them. SWE-Gate ships constraint tests separately; JetBrains reports validation rates and patch redundancy alongside resolve rate; Handoff Debt proposes reporting resumption cost. Whether the next round of model announcements adopts any of that, or reports one number against a benchmark whose inputs look nothing like production traffic, is the thing to watch over the next few weeks. On the retrieval side, ExecRetrieval’s exec@1 of 0.331 is a standing challenge to anyone whose agent architecture assumes the top hit is the right one.

---

Source freshness note: the code-intel mirror last synced 2026-09-01 and reported a stale window at generation time. Items dated through 2026-09-07 were present and were used; coverage of the final days of the window may be thinner than usual.