Review agents catch 71.9% of verified bugs, and the misses need project context

The best code review agent in a new 1,800-task benchmark catches 71.9% of human-verified bugs within its top five comments. Five months ago the same harness with the previous model caught 47.7%. That is the headline from CRBugBench, the benchmark Chenyang Yang, Christian Bird and colleagues built by mining bugs from repository history and pairing each one with the pull request that introduced it: 1,000 review tasks across 98 open-source GitHub repositories, plus 800 tasks from proprietary repositories at one large company. The question in the title, whether a software factory should run lights-off, gets a measured answer. The top configuration, claude-opus-5 driven through copilot-cli, is good on local, self-contained changes and falls off sharply on complex diffs, on bugs that need non-local reasoning, and on the proprietary repositories where project context matters most. Spending more compute helped, but the cheapest lever was the prompt: nudging the agent to explore before commenting gained up to 5.8 points at roughly 10% higher cost. The guidance the authors derive is operational rather than aspirational: scale review effort to PR complexity, steer the agent with project priorities, feed it repository knowledge, route PRs to models by their characteristics, and keep monitoring quality because the number moves every time the model does.

SWE-Duel, from Wei Minn, David Lo and colleagues, attacks the same review problem from the evaluation side. Static benchmarks leak into training data and saturate, so the authors argue a benchmark should be self-scaling: tasks and oracles created at evaluation time, difficulty set by the participants. The mechanism is a mirrored Red-Blue duel. The Red agent, working in three context-isolated sessions on a real repository, proposes and implements a feature with tests, then plants a subtle bug in that feature along with sealed bug-triggering tests. The pull request is admitted only if it passes the feature tests and the repository’s existing suite while failing exactly the bug tests, and only if a fresh session of the same agent can repair the bug blind, which filters out unsolvable tasks. The Blue agent then reviews without being told a bug exists and wins only if its revision passes all three suites. A tournament of 11 model-harness participants on 15 repositories in five languages ran 1,500 turns. GPT-5.5 with Codex CLI ranked first as the strongest solver, repairing 98.3% of the defects it reviewed. Claude Opus 4.8 with mini-swe-agent ranked second as the strongest author, with a 27.3% Red win rate. Within one model family, tasks written by the newest version beat a fixed-reference solver 42.9% of the time versus 4.0% for older versions, early evidence that task difficulty does scale with authoring capability. Read the two papers together and the residue is the same: the bugs that survive are the ones requiring reasoning across the change, and SWE-Duel’s admission rule caps difficulty at what the author itself can repair, so the arena measures a ceiling rather than the open-ended non-local failures CRBugBench documents in industry code.

Two multi-agent papers today converge on the same discipline. A revised BenchAgent study from Yuhang Fu, Tao Lin and colleagues compares six multi-agent workflow configurations against a single-agent reference across ten reasoning, coding and tool-use benchmarks, all on GPT-4.1 with shared tool and scoring interfaces and identical step limits, 3,302 instances per configuration per run, three runs. EvoAgent reached 79.70% pooled accuracy against 76.65% for the single-agent core and beat it in all three runs; the other five configurations trailed the core by 1.71 to 8.76 points. The local wins are real and the global losses are too: LLM-Debate gained 9.67 points on HumanEval but lost 1.96 points pooled while spending 1.39 times the tokens, and ChatEval gained 22.25 points on IFEval yet trailed the core overall at 4.34 times the token consumption. Competence-Aware Orchestration, from Songmiao Wang, Ming Li and colleagues, turns that observation into a controller. It first models the backbone’s task-conditioned competence on the current query, then uses that signal to decide which roles to activate and what communication graph to run. Across six benchmarks the method averages 93.81%. For higher-competence backbones it raised average accuracy from 87.22 to 89.97 while cutting token cost by 36.26%; for lower-competence backbones it added 4.58 points over existing orchestration methods, and the trend held on backbones it had not seen. The practical reading: the default topology is one agent, and any additional role has to buy its place on a per-query competence estimate, not on a fixed org chart.

LongHarness, from Songtao Huang, Shiyang Feng and colleagues, is the retrieval item, and it is a harness paper rather than a model paper. Agents reasoning over corpora of up to 100 million tokens accumulate evidence and conclusions in one growing interaction history, which the authors say produces two coupled failures: unbounded exploration, where retrieval keeps expanding the history without closing a decision-relevant gap, and evidence-state drift, where earlier evidence or later corrections stop informing the current conclusion. The harness makes the loop explicit. Unresolved conclusions direct the next retrieval, new evidence revises the conclusions that sent for it, and a lifecycle layer keeps the evidence store current and traceable. On OfficeQA Pro with GLM-5.3-Flash, exact match went from 15.8% to 60.2% on V1 and from 4.4% to 61.1% on V2 against direct reasoning with the same backbone. Against the coding harnesses people actually reach for, it beat Claude Code by 12.0 points on V1 and Codex by 18.9 on V2; with Qwen3.8-Flash it reached 61.7% on V1, 8.3 points over Claude Code and 12.0 over Codex. Those comparison harnesses were built for repositories, not document collections, so the gap is partly a mismatch of tool to task, but the state-management argument stands on its own.

The semantic-grounding item today is MatQuery, from Chen Jason Zhang and Qi Chen, and it comes from materials science rather than a data warehouse. The problem shape is the one this track cares about: six public databases, each with its own interface, schema, units and calculation conventions, and a natural-language request that has to be turned into correct queries and correctly interpreted results. The prototype holds frozen, self-verifying local copies of all six (about 10,000 records each), a catalog of declared fields derived from them, a database-independent query representation with an adapter per database, and an eight-step pipeline in which the language model proposes and deterministic program steps check, execute and assemble. On an 800-question development set, direct generation with three demonstrations reaches 82.4% strict execution accuracy, 56.5% without, and a low-cost scorer selects the exact set of databases for 85.0% of complete requests with calibrated probabilities. The number that matters most for anyone building a semantic layer is a different one: band gaps differ by more than 0.5 eV in 31.6% of composition-matched cross-database clusters. The same named quantity means different things in different sources, so routing the query to the right database and compiling it correctly is only half the job; the other half is knowing which definition you are returning. The authors are candid about the limits: a probe-then-ask policy produced answers both reviewers rated valid in only 11 of 50 sessions, only 115 of 300 sampled record pairs could be compared by structure, and the full pipeline is in smoke testing and has not been evaluated at scale.

What to watch next: whether CRBugBench’s proprietary-repository gap closes when review agents are given the repository knowledge the authors recommend, and whether self-scaling arenas like SWE-Duel can admit tasks harder than their own authors can solve. On the orchestration side, the competence-gating result needs a replication on coding and tool-use workloads with real token budgets, where BenchAgent’s numbers suggest the single-agent baseline is still hard to beat.

Source note: the code-intel-copilot mirror reported a last sync of 2026-09-01, yet every item above carries a 2026-10-12 feed date, so the staleness indicator rather than the data appears to be wrong. Items are dated from their feed entries.