Agent PR revert rates split by vendor: Codex 6.1%, humans 11.5%, Devin 14.5%

Pull requests opened by OpenAI Codex were reverted 6.1% of the time. The matched human baseline was reverted 11.5% of the time, and Devin’s PRs 14.5%. Those three numbers come from Not All Agents Are Equal, posted to arXiv today by Obada Kraishan, which follows 37,623 provenance-labeled pull requests from Codex, Devin, GitHub Copilot, Cursor and Claude Code across 2,807 GitHub repositories between December 2024 and July 2025. The AIDev dataset is joined to 58,792 cached GitHub API responses so the study can see what happens after merge, not only at review time. Pooled across vendors, agent code was less likely than human code to carry a security smell (odds ratio 0.63), driven by fewer hardcoded credentials and fewer eval-style constructs. Review effort lands unevenly: Copilot PRs drew the most human reviews and change requests, while Claude Code PRs waited the longest for a first human look, a median of 12.6 hours. The finding that should change how you read vendor comparisons is that quality is vendor-specific rather than a property of “agent code” in general. Any adoption study that pools agents into one bucket is averaging over a revert odds-ratio range that runs from 0.50 to 1.31, and the pooled number tells you nothing about the tool you actually run. Pipeline code, statistical reports and figures ship with the paper.

A second study in the window accounts for some of the variance the first one cannot. A Few Pages of Markdown, a v2 from Yegor Denisov-Blanch, Bogdan Vasilescu, Sanmi Koyejo and colleagues posted 2026-09-15, introduces RAMP, the Repository AI Maturity Profile: a four-level cumulative scale read off the AI configuration a team commits to version control, running from behavioral rules and coding standards through named agent definitions to multi-agent orchestration. Across 441 repositories the levels behave as a proper cumulative scale, and independent human annotators reproduce the repository labels on 97% of a held-out sample. Adoption is forward-only and set-and-forget: 73.8% of the artifacts are committed once and never modified. Re-estimating an existing agent-adoption panel within each maturity stratum, agents add 28 to 38% more commits no matter the level, but quality diverges. Among agent-first repositories, the ones with no committed AI configuration show roughly twice the increase in cognitive complexity (+53% versus +27%) and 1.7 times the increase in static-analysis warnings. The authors are careful to call the result hypothesis-generating, since maturity is observational and correlated engineering discipline could explain part of the gap. Even with that caveat, a few pages of committed rules is the cheapest lever anyone has measured on the quality side of the velocity trade, and it costs an afternoon.

Two reliability papers in the same window arrive at the same design move from different directions: bind evidence to the exact state it was collected from, and refuse to act on anything else. Looping Is Not Reliability, a v2 from Xueping Gao, Jianwei Yang and Qiang Yang posted 2026-09-15, studies the gap between finding a correct patch and keeping it. In a sealed five-seed study over 30 HumanEval repairs producing 900 three-revision trajectories, forced revision drops current correctness from 0.820 after one revision to 0.673 after two, even as ever-correct rises to 0.847. The loop finds the answer and then revises it away. Two common-state studies over 2,430 branches from identical frozen programs isolate the mechanism: in a prespecified 14B replication, stale verifier traces harm 34 of 135 correct starts versus 4 of 135 with current traces, a 22.2-point increase with a task-cluster 95% CI of [8.9, 37.0] and exact Holm p = 0.0337. A prospective 540-rollout policy eliminates the correct-start harm but reduces wrong-start repair and fails its joint criterion, and repository experiments over 24 bugs and four coder stacks show floor effects without Holm-significant results. Out of that the authors separate admission, preservation, grounded certification, competence and liveness as distinct properties, derive an evidence-bound typed loop contract, and ship a reference implementation that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts. They state plainly that the implementation is a conformance artifact, not evidence of better repair.

Cognitive Admission Control, from Jun He and Deying Yu, applies the same discipline to agents that mutate external infrastructure. A policy maps a typed action and its modeled risk to assurance obligations that specify predicates, evidence classes, scope, freshness and witness-set constraints. A deterministic evaluator sorts each obligation into satisfied, violated or unresolved, and unresolved ones generate targeted evidence-acquisition requests rather than a block. Successful admission produces a certificate that binds the action, its witness manifest and dispatch-time guards. A TypeScript prototype runs 2,730 controlled local trials with independent effect observation and matched fault schedules. Across 390 CAC trials, 120 effects complete without modeled harm and no harmful effects occur, while a live-policy baseline reaches the same completion count but admits the constructed correlated-witness failure. A further 9,000 measurements exercise the full local dispatch path with persistent replay protection. The guarantees are policy-relative, and the authors say so: physical safety additionally needs sound evidence, an adequate environment model, and preservation of the relevant conditions through the effect. Read alongside the Gao paper, the shared claim is that a receipt tied to a state is worth more than a retry loop, and both groups declined to promise more than what they tested.

On the semantic-governance side, DualSQL, from Shijie Chen, Yu Su, Fatma Ozcan and colleagues at Amazon and Ohio State, posted 2026-09-16, trains the schema-linking agent and the SQL-generation agent as two roles on a single model backbone with a shared agentic scaffold, jointly optimized with multi-agent reinforcement learning. Three database-access tools ground the multi-step reasoning in live interaction with the database, a set of rollout guardrails keeps the multi-agent RL from collapsing, and a new correctness metric, robust execution match (REX), supplies the reward signal. Trained on only 3,755 examples, DualSQL-4B reaches 68.0% execution accuracy on the BIRD development set, matching earlier 7B systems, and DualSQL-8B reaches 71.1%, ahead of previous single-model state of the art at 32B parameters. The charter angle is schema linking, which is the step where business meaning either survives into the query or does not. Training the linker jointly with the generator means it learns to retrieve what the generator needs rather than what a separate loss says is relevant, and that is the right coupling for enterprise schemas where the relevant column is rarely the one with the obvious name. The open question is what REX rewards when the schema has governed definitions rather than BIRD’s raw tables; nobody has run that experiment yet.

The last item is an evaluation method for agents over data that changes under them. Skill-based Agentic Evaluation for Real-time Data Science Tasks, from Aniruddha Tamhane and colleagues, starts from a query like “what were last week’s audience sizes”, where the reference answer is different every day so a static gold label is stale by the time the judge reads it. Their answer is ground-truth-as-code: each expected answer is an executable reference function that recomputes the truth from live data at evaluation time. A factoid-level, format-agnostic judge then decomposes both the agent’s response and the computed truth into atomic claims and scores precision, recall and accuracy over them, whether the response arrived as prose, a list, a table or HTML. Validated through a human-LLM agreement study on a production machine-learning skill against a synthetic database that reproduces production schemas and entity relationships, the method improves Matthews Correlation Coefficient against expert annotators by 29% over a natural-language ground-truth baseline and cuts tokens per test case by 16%. The number to keep is the one about the baseline they nearly used: a self-directed judge with no explicit ground truth was anti-correlated with human judgment. A judge without a reference does not merely add noise, it points the wrong way.

What to watch: whether Kraishan’s released pipeline gets rerun on the post-July-2025 PR population, where agent volume is several times larger; whether the next adoption panel stratifies by RAMP level instead of reporting a pooled effect; and whether anyone trains a DualSQL-style joint linker against a governed semantic layer and reports wrong-join and wrong-grain rates rather than execution accuracy alone.

Feed note: the code-intel mirror reported its last sync as 2026-09-01, but items in the pool carry creation timestamps through 2026-09-17 08:05 UTC, so the sync marker is stale rather than the data. Items are dated from their arXiv or source publication dates.