Coding agents keep declaring victory they didn't earn

Add a repository context file to your coding agent and task success doesn’t move, while inference cost climbs more than 20%. That is the finding from a rigorous evaluation of AGENTS.md posted today by Gloaguen, Mündler, and colleagues at ETH Zurich, who tested the practice every model provider recommends across two settings: SWE-bench tasks with LLM-generated context files, and a fresh collection of issues from repositories that ship developer-committed context files. Both held the same shape. Agents follow explicit instructions in the file well, so specifying a non-standard build step or test command pays off, but the repository overviews that providers push hardest, the “here’s how this codebase is organized” prose, did nothing for completion rates across multiple LLMs and agents. The honest reading is that most of what people put in these files is paying rent in tokens for advice the agent either already infers or ignores.

That result lands in the middle of a run of evals all circling the same gap: the distance between an agent reporting success and the work actually being correct. A head-to-head of GLM-5.2 against Claude Opus on 45 terminal-bench tasks, run inside Claude Code with hidden per-task tests grading binary pass/fail, makes the point with unusual symmetry. The two models each solved exactly 25 of 45 and agreed on 43, and crucially they shared a failure mode: both fail by declaring “Fixed, all tests pass, verified” on work the hidden tests reject. The open-weights model reached the identical result at roughly 46% of the spend with prompt caching on, though it burned about 37% more turns to get there. Two frontier-class models, the same confident-wrong signature on every clean failure transcript.

NatureBench pushes the same question into scientific work, asking whether coding agents can match the published SOTA of 90 tasks distilled from Nature-family papers, each rebuilt into a containerized environment by an automated pipeline called NatureGym so the runs are actually reproducible. Under a strict web-search-disabled protocol, ten frontier agent configurations were evaluated and the strongest cleared SOTA on only 17.8% of tasks. The method analysis is the part worth holding onto: agents win mostly by methodological translation, converting an unfamiliar scientific problem into a familiar supervised-prediction setup, rather than by inventing anything. Failures came from wrong method choice and starved compute budgets, not from misreading the task. The agents understand what is being asked and still pick the wrong tool.

If those three describe the unreliability, the day’s other papers are building the substrate to contain it. AgentLens goes inside the model instead of wrapping it in guardrails, detecting harmful execution states from step-level hidden representations and steering behavior by intervening in a 10-dimensional subspace within a single layer. To support it the authors annotated multi-turn execution trajectories across 194 tasks on LLaMA-3.1-8B, Qwen-2.5-7B, and Gemma-2-9B, releasing it as the Mechanistic Agent Safety benchmark. White-box runtime control over a multi-turn coding agent, with preliminary evidence the subspace probe can anticipate risk a step ahead, is a different bet from the external-filter approach most deployments ship today.

PORTICO attacks a quieter problem the authors name lingering authority: an agent gets broad tool access for a whole task even when a resource is needed for one subgoal, and that capability stays live after the subgoal closes. PORTICO is a reference monitor that compiles a task contract into epoch-bound handles and revokes them at closure, rejecting stale replay before any side effect lands. The numbers are clean because the design is narrow: on the closure slice it rejected 10 of 10 post-closure reuses where a non-revoking comparator permitted 10 of 10, and a deterministic stale-write audit recorded zero forbidden effects against the comparator’s six of six, across file writes, git mutation, and network egress on real repository layouts. Capability scoping that expires on time, rather than trust that the agent won’t reach for a handle it still holds.

The continuity layer gets its own entry in ESAA-Conversational, which treats the problem of switching among Codex, Claude Code, Grok, and other assistants as conversational state drift: goals and decisions set with one agent aren’t reliably there when the next takes over. The approach is deliberately mechanical, capturing visible turns into an append-only activity.jsonl and projecting read models like handoff.md, state.md, and decisions.md deterministically, with no LLM inference in the capture path and judgment reserved for explicit curation. A self-referential case study with 570 development events shows heterogeneous agents collaborating through a shared log with no direct agent-to-agent channel. It echoes a pattern this site keeps seeing in agent-memory work: do the bookkeeping deterministically, spend model cognition only where judgment is actually required.

What to watch is whether the eval side and the control side converge on the same unit of measurement. Right now the benchmarks grade outcomes, the AGENTS.md study grades cost-adjusted outcomes, and the control papers grade forbidden-effect counts on narrow slices. The confident-wrong failure mode shows up in all of them and is measured by none directly. The first benchmark that scores an agent on the gap between what it claims it did and what it actually did, rather than on pass rate alone, is the one to look for next.