Submit 100%, Resolve 44%: Agent Evals Move Past Completion

This week the field stopped scoring coding agents by whether they finish and started scoring whether they can be trusted: frontier models that submit on 100% of runs but resolve under half, a shared confident-wrong failure signature across GLM-5.2 and Opus, and RigorBench showing process discipline lifts correctness 17%. Alongside it, hard evidence that agent code costs more to maintain, that AGENTS.md files often don't pay for themselves, and that agent memory has become a data-management problem with its own governance and eviction tradeoffs.

GPT-5 submits a patch on 100% of its runs and resolves 44% of them. Llama 4 submits on 99% and resolves 18%. Gemini submits least often, on 70% of runs, and still resolves more tasks than GPT-5, 50% to 44%. Those numbers come from 1,750 trajectories over 50 SWE-bench Verified tasks in Confident and Wrong, and they describe the gap the whole field spent this week trying to measure: the distance between an agent finishing and an agent being right.

Submit rate measures action; trust requires validity

The failures in that study are not random noise. They concentrate in one mode the authors call silent semantic failure: on a buggy task the agent ships a plausible-looking patch on all five runs, none pass, and it is the same misinterpretation each time rather than five different mistakes. That single mode covers 80% of Llama 4’s failing runs and 68% of GPT-5’s. Because the agent is confidently and consistently wrong, both completion-based and consistency-based monitoring look healthy at exactly the moment you should not trust the output. A second probe hands the model an already-fixed bug, where the correct move is to abstain, and most models edit the correct code anyway. That action bias, acting when no action is warranted, is precisely what submit-rate rewards.

The same shape showed up outside the lab. A practitioner head-to-head of GLM-5.2 against Claude Opus ran both models inside Claude Code on 45 terminal-bench tasks, identical harness, 40-turn budget, graded by each task’s own hidden tests. Each solved exactly 25 of 45, and they agreed on 43 of them. Both models failed the same way: declaring “Fixed, all tests pass, verified” on work the hidden tests rejected. GLM landed the identical result at about 46% of Opus’s cost, taking 37% more turns to get there. The interesting finding is not the price. It is that two very different models, run as agents, share a failure signature, and that signature is confidence without validity.

RigorBench makes the constructive version of the argument. Instead of scoring only whether the final patch passes, it scores how the agent got there across five pillars: planning fidelity, verification coverage, recovery efficiency, abstention quality, and atomic transition integrity. Across 30 tasks with names like Doom Loop Gauntlet and Know When to Fold, harnesses that enforce process discipline scored 41% higher on process quality and, more to the point, 17% higher on downstream correctness. How an agent codes turns out to predict what it produces.

Agent code costs more to maintain than it costs to write

Is Agent Code Less Maintainable Than Human Code? builds CodeThread, a way to construct controlled maintenance experiments from repository-level benchmarks, then has agents resolve tasks on top of prior agent code versus prior human code. Building on agent code drops task resolve rates by up to 13.1%. The standard maintainability metrics (cyclomatic complexity and the rest) do not explain the gap. The signals that do are behavioral and subtle: how the earlier agent handled input validation and error handling, plus downstream code size. Agent output that passes today raises the cost of the next agent’s task, and the usual static metrics miss it.

That compounding has a population-level analogue. Augmentation with Dilution runs a staggered difference-in-differences design over 11,097 GitHub repositories from 2023 through May 2026. Agent adoption does not change the absolute number of human contributors (ATT = 0.014, not significant), but it cuts human contributor density by 1.9% and the newcomer participation share by 3.7 percentage points, and it raises review depth by 5.3%. The work does not disappear; it moves from production to review, and the people most squeezed out of the loop are the newcomers who would have become tomorrow’s maintainers.

The configuration layer nobody is governing

If you assumed the AGENTS.md file in your repo is paying its way, a rigorous evaluation from the Vechev group says check that assumption. Across SWE-bench tasks with LLM-generated context files and a fresh set of repositories carrying developer-committed ones, context files did not generally improve task success while adding over 20% to inference cost. Agents follow explicit instructions well; the repository overviews that model providers recommend most are the part that does not help. Context files earn their place for non-standard practices a model cannot infer, and not much beyond that.

The layer itself is unmanaged in a more structural way. A Deterministic Control Plane for LLM Coding Agents scans 10,008 repositories and 6,145 agent config files and finds that 10.1% of tracked config paths are exact SHA-256 duplicates of files in other, independent repos, with 75.5% of those clone pairs crossing organizational boundaries. Configs are rarely revised (58% sit at a single commit) and almost never declare permission boundaries (under 1% of agent configs, against 33% of Actions workflows). Rules files propagate as an undeclared, unversioned, unpermissioned supply chain. The paper’s prescription is worth sitting with for anyone building orchestration: governance of this layer should be deterministic and tool-agnostic, not handed to yet another LLM.

Agent memory is now a data-management problem

Are We Ready For An Agent-Native Memory System? stops treating memory as a black box scored by end-task F1 and decomposes it into four modules: representation and storage, extraction, retrieval and routing, and maintenance. Evaluating 12 memory systems across 11 datasets, no architecture wins everywhere; effectiveness tracks how well the memory structure matches the workload’s bottleneck. The sharpest operational finding is on cost: localized maintenance beats global reorganization on the cost-performance frontier, so the cheap incremental update usually wins over the expensive periodic rebuild.

Governed Shared Memory for Multi-Agent LLM Systems takes that into production and reports the negative results most papers bury. It formalizes the fleet-memory problem (four failure modes: unauthorized leakage, stale propagation, contradiction persistence, provenance collapse) and evaluates a live multi-tenant service, MemClaw, rather than a clean baseline. Live evaluation earned its keep: tenant isolation held, but sub-tenant scope was bypassed on direct GET-by-id requests for agent-scoped credentials, found and fixed during the study. A second bug had a synchronous near-duplicate gate rejecting contradictory writes before the asynchronous contradiction detector could ever see them. Long-context retrieval alone does not give you safe shared memory; you need scoped retrieval, temporal supersession, and provenance as explicit system primitives.

Two papers attack the cost of memory directly. Learning What Not to Forget presents LRE, a few-kilobyte, CPU-only, model-free scorer that learns which units of history are load-bearing (the access token from login, the path the next call needs) and keeps them verbatim. Matched on budget, it holds the accuracy of keeping the entire history while cutting peak context by up to 52%, and on LoCoMo it gives the best budgeted answer quality while reading 68% fewer tokens. Trained only on the agent’s own behavior, with no annotation, it recovers 95% of the supervised scorer’s effectiveness. Managing Procedural Memory introduces AFTER, 382 enterprise tasks across six roles and 22 skills, and finds that skills distilled from diverse multi-model traces hit 73.1% cross-model accuracy, beating any single-model trace source, while some skills generalize and others overfit to one role and break under transfer.

Coordination failures hide before the pull request

The AIDev dataset records 456,000 agent-authored pull requests, produced faster than human PRs and accepted less often. Before the Pull Request argues the explanation for that gap lives in a layer PR telemetry cannot see: how concurrent agents claim, divide, and collide over shared work before any PR exists. Using grite, a server-less, git-native coordination substrate that stores a signed, append-only event log inside git refs, the study runs a controlled sweep up to 32 concurrent agents. Without coordination, 78% of completed work merely re-does a teammate’s task. Advisory leases alone cut conflicting edits but leave redundant rediscovery as the worst failure mode, because a lease stops two agents working a task at the same instant but not one agent redoing what a teammate already finished. Only leases plus shared completion state drive duplicate work to zero and more than triple goodput. The design lesson generalizes past this one tool: mutual exclusion and conflict-free shared state are jointly necessary, and a file-based tracker that loses concurrent writes is not enough. (The paper benchmarks against Beads, the git-backed agent issue tracker, as its closest comparison.)

More retrieval is not more accuracy

The Token Tax of Epistemic Accuracy puts a price on the long-context-versus-RAG argument with an expert-validated benchmark in manufacturing safety: 972 answers, three machines, small language models. Loading the whole document collection into context scored highest on correctness, 73.1% against 65.4% for semantic RAG, at 26 times the per-query token cost. Broader evidentiary access buys real accuracy, and you pay a steep, measurable tax for it, which is a different decision for a resource-constrained team than for a frontier lab. Is GraphRAG Needed? implements nine standardized RAG scenarios from plain retrieval through GraphRAG and agentic RAG, and surfaces a retrieval-generation gap: expanding what you retrieve does not improve generation proportionally, so retrieval-oriented metrics overstate what the fancy variants buy. Their context-engineering method cuts token use 19% to 53% without giving up answer quality.

The throughline across all of it: the easy question was always whether the agent finished. This week the field spent its energy on the harder ones. Can the result be trusted, who pays to maintain it, where do concurrent agents collide, and what does the extra context actually buy. Watch whether the next round of benchmarks reports test-verified correctness with uncertainty, the way Confident and Wrong asks, rather than another leaderboard of submit rates.