The verification loop was worth 1.5 of 11 points; the scaffolding carried the rest

Leni’s production agent beats its own frontier base model by 11.0 percentage points on SpreadsheetBench Verified: 91.25% against 80.25%, n=400, p<0.001. The team then took the architecture apart to find where those eleven points actually live, and the verification loop the whole system is built around contributed 1.5 of them.

That decomposition, published this week, is the most useful thing I read all week, and it reframes several other results that landed alongside it.

The reliability uplift is mostly scaffolding, and the verifier only helps when it isn’t the author

Where Does Agent Reliability Come From? evaluates one unmodified production configuration across three benchmarks chosen to stress different failure modes: SpreadsheetBench Verified for silent computation error, BullshitBench v2 for premise confabulation, GAIA validation for cascade error over long tool chains. The gains are real across all three, +11.0 points, +7 to +10 points (98% vs 91%, n=100), and roughly +15 points on GAIA (75.2% pass@1 at n=165, 83.0% best-of-k). The decomposition is what matters. Most of the uplift comes from scaffolding, routing, and task-specialized models. The execute-observe-compare-correct loop contributes +1.5 points in isolation.

Two details keep that from being a dismissal of verification. First, the +1.5 is concentrated at the top of the score distribution, where it converts tasks that would otherwise fail outright. Second, the specialist-swap ablation: replace the small post-trained verifier with the frontier model that generated the answer, and most of the rescues disappear. The loop’s value depends on who observes it. An agent checking its own work recovers the same blind spot it just walked into. The instrumented confusion matrix says the same thing quantitatively, a catch rate around 0.20 with a fix rate of 0.75 and no false-alarm regressions, plus zero over-rejections on 100 valid-premise expert questions. A verifier that catches one failure in five and fixes three of four it catches is not a safety net. It is a small, cheap, well-placed second opinion.

Proof-or-Stop attacks the same problem from the lifecycle side. Its premise is that states like reviewed, tested, DONE, and ready-to-merge are claims until evidence backs them, and the method permits a transition only when fresh, tracked-source-state-bound, mechanically verifiable evidence satisfies the gate. The binding to tracked source state is the load-bearing part. An agent that ran the tests, then edited three more files, then declared DONE has produced stale evidence, and most orchestration layers today accept it.

The infrastructure side of this arrived as a product. AWS made the Lambda Durable Execution SDK for .NET generally available on July 23, extending Lambda’s event-driven model with steps that checkpoint progress automatically and can pause for up to a year waiting on an external event. The listed use cases put AI agent orchestration next to payment pipelines and human-in-the-loop approvals, which is the right company for it. The SDK ships callback integration for human-and-agent-in-the-loop workflows, durable invocation for function chaining, and a local testing emulator on NuGet. Durable execution stopped being a Temporal-shaped specialty and became a runtime primitive you get from your cloud provider in the language you already use.

CodeRescue asks a narrower budget question: after a coding agent fails, when do you spend more cheap compute and when do you escalate? Existing cost-aware systems treat this as a cascade, cheap model first then escalate the hard cases. The paper’s finding is that in executable environments, failure produces actionable feedback, so cheap recovery and escalation succeed on complementary sets of tasks. They train a supervised router over execution rollouts and wrap it in a Conformal Risk Control layer that picks a deployment-time cost penalty without retraining. In the GPT-5.4-nano to GPT-5.4 setting, one calibrated point on the frontier exceeds the always-escalate solve rate at 35% of its mean recovery cost. The code is at github.com/Qijia-He/agent-budget-control.

Underneath all of this sits a class of bug nobody has good tooling for. Understanding Agent-Reactive Bugs at the Model-Harness Boundary studies issue reports from LLM agents spanning CLIs like Codex and frameworks like LangChain, where harness code parses model output, drives the loop, and manages context. The resulting bugs cannot be diagnosed by inspecting the harness or the model in isolation, because both jointly shape execution. Anyone who has watched a parser choke on a slightly reformatted tool call, then watched the retry logic paper over it until the trajectory silently diverged, knows the shape.

The benchmarks built this week are aimed at what agents actually get wrong

Tencent WorkBuddy Bench takes a real position on contamination. Every task is reverse-engineered from an actual commit, pull request, or business scenario, then rewritten as a short colloquial role-played request, so the prompt cannot be recovered by web-searching the underlying thread. The whole thing is released openly, task directories, environment images, harness, tests, reference solutions, with contamination resistance resting on construction plus dataset versioning rather than secrecy. Four subsets cover repository engineering, front-end, office and business workflows, and red-team/blue-team security, each with its own verification style, run under one protocol on two harnesses (CodeBuddy Code and Claude Code). They decline to publish a suite-wide average because the scoring instruments differ across subsets. That refusal is the most principled decision in the paper, and it is the one most benchmark authors skip.

IssueTrojanBench is the week’s uncomfortable number: 66.5% of malicious issue requests penetrate every guardrail, agent-level and LLM-level, across Cursor, Claude Code, and Codex Desktop running GPT-5.3 Codex, GPT-5.4, and Sonnet 4.6. The benchmark builds attacks from four categories embedded as instructions in issues, across six delivery vectors including PDFs and issue comments, plus perturbations. The finding that should change how people build: rejection comes almost entirely from the model, not the agent framework. Agent-level defenses added little. GPT models were broadly vulnerable; Sonnet 4.6 blocked more selectively on high-impact actions. If your threat model assumes the harness is filtering anything, this is the paper that says it isn’t.

Test Coverage Analysis of Agentic Pull Requests mines 4,882 agent-generated PRs from the AIDev dataset, 532 Java and 4,350 Python, across five coding agents. Agents include test changes in 49.6% of PRs that touch code under test files. Existing tests cover 61.5% of changed executable lines in Java and 27.0% in Python, and 64.8% of Python PRs have no changed line executed by any existing test at all. Agent-written tests improve on that in a minority of cases, 35.9% of Java and 22.5% of Python Code+Tests PRs. Error handling is the consistent hole, with miss rates of 86.0% in Java and 81.0% in Python for try and catch constructs. Merging agent PRs on green CI, with those numbers, means merging on a signal that never touched the changed lines.

Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation runs seven agentic strategies for Terraform on IaC-Eval v2, 186 AWS tasks with Rego v1 intent policies, separating failures into three verifier stages (validate, plan, opa eval) with McNemar tests and Wilson intervals across all pairwise comparisons. Active retrieval through ReAct agents with MCP or ChromaDB-backed RAG takes Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001), mostly by cutting VALIDATE_FAIL from 144 tasks to 66. Iterative refinement against verifier feedback reaches 62.9% for Qwen 7B and 84.4% for GPT-4o, and the convergence is binary: a task either resolves on the first retry or exhausts the budget. GEPA reflective instruction optimization adds 7.5 points over the Active RAG baseline (p=0.026) using 80 verifier-guided rollouts and no weight updates. The diagnostic that ties it together: 79% of post-refinement policy failures resolve when the policy text is visible to the agent (p=0.016). Most of what looks like reasoning failure is a missing document.

Agentic Code Review in the Terminal builds AgenticCR-Verified, 362 human-verified repository-level review tasks filtered down from 68K discussions, and runs Claude Code, Gemini CLI, Rovo Dev, and CodeRabbit against static LLM reviewers inside the Harbor harness at temperature 0. Requiring both localization within five lines and human alignment, the top scores are 5.81% for Rovo Dev and 5.60% for Claude Code against 1.72% to 2.61% for static reviewers, with the gap driven by localization (over 20% for the agentic reviewers versus 7% to 9%) while generating fewer comments. The trajectory analysis is the useful half. Planning takes under 10% of effort across every agent, and it is the only phase significantly associated with review success (p=0.012 for Rovo Dev, p=0.038 for Gemini CLI). Unsuccessful reviews spend more on validation, re-reading diffs they should have understood the first time. Cost does not track quality: Rovo Dev spends $0.75 on validation and $0.44 on exploration per task, Claude Code covers comparable exploration steps for $0.10, and their overall scores are within a quarter point of each other.

Context is the substrate, and retrieval design decides when it matters

How Agent Skills Fail under Long Contexts holds a code-audit task and 24 artifact checks fixed and varies only the surrounding context. Codex with gpt-5.4-mini passes 8 of 10 runs in a clean 10,991-character context and 3 of 10 in a 299,140-character context, whether that long context is relevant or irrelevant. The 50-point difference stays trend-level under a two-sided Fisher test (p=0.0698), and the author is careful to say a second task passes every condition, so there is no universal length threshold here. The detail worth carrying: requirement coverage stays above 92% in both long conditions. The skill’s instructions are still nominally being followed. A handful of omissions invalidates the artifact anyway. And a detailed external checklist passes 10 of 10 where a generic self-check passes 5 of 10 (p=0.0325), which lines up exactly with the Leni result about who does the observing.

Is Progressive Disclosure All You Need for Long-Context Agents? is the first controlled study of a pattern practitioners adopted on vibes. Comparing raw-document navigation, several Agent Skills pack designs, and a classical hybrid retriever across three harnesses and three model families on InfiniteBench, the gain from progressive disclosure on a single book depends almost entirely on the harness: large when the agent navigates raw documents badly, near zero when a strong harness already splits and retrieves on its own. Scale to many books and raw navigation collapses while one-level disclosure degrades slowly and wins. A second routing level never helps and sometimes breaks accuracy. The authors’ framing is the keeper: progressive disclosure buys context, not intelligence.

SWE-Pruner Pro finds that the coder model already encodes which lines of tool output matter, in its internal representations, while reading them. Instead of bolting on a separate classifier the way SWE-Pruner did, a small head reads those representations and emits a keep-or-prune label per line, with a length-aware embedding keyed to each tool output’s line count. Across two open-weight backbones and four multi-turn benchmarks it saves up to 39% of prompt and completion tokens with bounded overhead, and on MiMo-V2-Flash it raises SWE-Bench Verified resolve rate by 3.8% and long-context Oolong accuracy by 2.2 points. Pruning that improves the score is pruning that was removing noise, not signal.

Cost per verified task, not cost per token

A Databricks benchmark on a multi-million-line, multi-language codebase found that token price predicts almost nothing about the cost of finishing an engineering task. Same model, different harness, more than twice the cost. One harness sent roughly 3x less context per turn and reached similar quality. The post’s proposed metric is cost per verified task: model plus tool plus context plus retry cost, divided by a task that passes its checks. The open question it raises is the one worth arguing about, whether you hold the harness constant when comparing agents or treat the harness as part of what you’re evaluating. The Leni decomposition and the terminal-review cost spread both say the second.

TRIM names the artifact that accumulates when nobody measures this. CodeSlop is the residual, functionally unnecessary edits left in a final patch: speculative changes, abandoned hypotheses, temporary scaffolding the agent never cleaned up on its way to green. The insight is that you minimize it indirectly by minimizing the trajectory rather than the diff, which cuts CodeSlop by 17.9% to 32.9% across agentic scaffolds with negligible performance regression, at roughly half the validation cost of Delta Debugging. Every merged patch that carries its own search history makes the next agent’s context worse.

What to watch

The Leni decomposition and the Agent Skills checklist result point at the same design constraint from opposite directions: the observer has to be a different process than the actor, and it has to hold an explicit external artifact rather than a remembered instruction. Nobody has published the cost curve for that yet. What does a separate small verifier cost per task at production volume, and at what catch rate does it stop paying for itself? Leni’s 0.20 catch and 0.75 fix is one data point from one architecture on three benchmarks.

The second thing to watch is whether anyone reports cost per verified task in a comparison this quarter. IssueTrojanBench says the harness is not filtering attacks. The Databricks numbers say the harness is setting the price. The terminal-review study says the harness is deciding whether planning happens at all. Benchmarks that collapse model and harness into one score keep answering a question nobody is asking.