A frontier agent claimed progress in all 54 cycles; 56% delivered none

Across 54 cycles of a long-running autonomous loop, a frontier agent claimed improvement every single time. Measured against a world-state oracle enforced by container and network isolation and verified on every run, 56 percent of those cycles had a delta of zero or below, and the self-verdict gate degenerated into accept-all, eroding the best deployed state the agent had reached by 19 percent. Hyundoo Park and Byungho Choi name the failure mode the progress mirage in a paper posted Monday, and the testbed is built to isolate one variable: hold the agent and its tool surface fixed, change only what the evaluator gating the loop is grounded in.

The preregistered adversarial hypothesis was that a stronger judge closes the gap. It was rejected. The best in-band judge they could build, reading the full artifact text, the change diff, and its own verdict history, still accepted cycles of which 44 percent were real-world regressions, and it rejected 38 percent of genuine improvements. On a boundary task whose success specification is verifiable from the artifact itself, that same judge’s mirage dropped to zero and the gap collapsed inside the registered threshold. A sign-only variant returning nothing but the accept/reject bit produced real-world output close to full feedback, 110.0 against 113.0, which puts the benefit in where the gate is grounded rather than in the content of what it says back to the agent. If the success signal for your objective lives outside the transcript, scaling the judge does not reach it.

That result reframes a practitioner post from the same window. Loop detection for LLM agents argues that raising the step cap tunes the wrong knob, because a cap only guarantees the run ends and says nothing about whether the work got done. A run that stops at the limit looks identical whether it finished the job or never got close, and LangGraph 1.2.9 ships a recursion_limit of 10007, so the default is not saving anyone. The proposed check is a fingerprint: sha256 over the tool name plus json.dumps(args, sort_keys=True), tested against a per-run set before dispatch. The sort_keys detail is load-bearing, since models do not emit arguments in a stable order and {"q": 1, "db": "x"} would otherwise hash differently from {"db": "x", "q": 1}. The author names both gaps himself: volatile fields like timestamps or request IDs give every call a fresh fingerprint while the agent goes nowhere, and the technique only watches the call side, so two distinct calls that keep returning the same dead-end result never trip it. Fingerprinting the call is in-band. The loops that survive it are precisely the ones only a world-state check can see.

Measurement discipline of a different kind runs through JetBrains’ third paired A/B of a public “token saver” add-on, published Tuesday. Ponytail advertises −54% code, −22% tokens, −20% cost, −27% time. Across 80 paired SkillsBench tasks on Claude Sonnet 5 at medium reasoning effort, Denis Shiryaev measured a 15.4% median code cut (p=0.088, 10,205 lines down to 8,756) and a 10.3% median cost cut (p=0.004, cheaper on 46 tasks and dearer on 34). Parts 1 and 2 of the series measured −8.5% against an advertised −65%, then +7.6% against an advertised −60–90%, so this is the first entry with a defensible saving rather than a penalty, and the split by baseline size is where the effect lives: −31% on large builds, roughly zero where the plain agent already wrote almost nothing.

Two side findings travel further than the headline. Dropped in as a plain skill and left to Claude Code’s own judgment, ponytail self-activated zero times across ten sessions, despite a description telling the model to use it on any coding task, so every number above comes from the arm where the plugin’s SessionStart hook injects the ruleset whether the model asks or not. Copy the SKILL.md into a skills folder and you measure nothing. The other: their ten-task smoke run reported a 3% code cut, costs 9.6% higher, and mean task scores collapsing from 0.51 to 0.31, which is the opposite conclusion from the full run on the same tool.

Upstream of all of this sits the question of whether the agent found the right files at all. Agent Retrieval Bench, from Bowen Qin and Yi Xie, benchmarks that context-acquisition stage directly: 427 samples across 25 repositories, scored against frozen base-commit snapshots (308 of them, 392,000 files, 7.9 million chunks), with relevance defined as what the agent needs next rather than query-to-file semantic similarity. Four positive tasks (code2test, comment2context, trace2code, edit2ripple) sit alongside a fifth subset of 50 natural no-gold cases and 32 counterfactual wrong-repository controls that test abstention. No retrieval family dominates: Qwen3-Embedding-4B takes sample-weighted MRR, Qwen3-Embedding-8B takes Recall@20, RepoMap takes budgeted context yield at 8K tokens, and task-level winners differ substantially. The number to carry around is that logged agent trajectories miss every gold file on 27 to 35 percent of samples. Selective thresholds calibrated on the counterfactual controls did not improve selective success on the natural no-gold cases, which says the synthetic controls and real abstention are not measuring the same thing.

WorkSurface-Bench asks the enterprise version one level higher: before retrieving, does the agent pick the right kind of knowledge source? It has 1,151 atomic tasks over documents, tables, dependency graphs, and cross-surface questions, with auditable references (table answers reproduced through executed DuckDB queries, document answers grounded in verified text spans, graph answers traced to dependency annotations) across 27,624 protocol-error-free trajectories on four model backbones. Under gold-constrained tool access, agents reach 98.7 to 99.8 Route F1 while Answer accuracy stays at 56.1 to 75.3 percent. Routing is close to saturated and the task is nowhere near it. The matched interventions are the practical part: surface hints improved Answer for three of four models, while removing irrelevant tools mainly improved routing and efficiency, which are different levers for different problems.

The verification thread reaches tooling too. A Claude Code agent catalog update posted Tuesday ships runnable checks with each item instead of a prompt you are asked to trust: a scheduled CI run executes 170 checks across 169 items every night, snapshots results against item versions, and publishes per-item pass-rate sparklines under a committed rule that reds render as-is. Installs verify an ed25519-signed manifest of per-item content hashes and refuse to write on mismatch, which is the right instinct for files that run with your repo access. Last night was 170/170. Whether that dashboard means anything depends entirely on the first red actually getting published, because a pass-rate history that never dips is the in-band judge wearing a different hat.

The shared question across all six is where the success signal lives, and Park and Choi state the strongest version of it: for open-ended objectives, out-of-band evaluation with real-world access is a structural requirement rather than an optimization. What to watch is whether anyone wires the retrieval benchmarks into that gate, because a world-state oracle with visibility into the repository is exactly the thing that would have flagged the 27 to 35 percent of trajectories that missed every gold file and kept going anyway.