Verification has a price, and agents are overpaying

Stop a coding agent from running tests entirely, and on Claude Code or Codex with a current frontier model its SWE-bench resolve rate falls by 1.25 percentage points. Not statistically significant. That number anchors To Run or Not to Run, an ISSTA 2026 study that pulled 7,745 agent traces off the SWE-bench leaderboard, then ran 3,000 fresh repair attempts across 200 instances under four execution regimes. Agents lean on execution hard, 8.8 test runs per task on average, with some models firing 19 on a single instance, and the study’s point is that the payoff is concentrated rather than uniform: a few instances genuinely need the agent to run something, most don’t, and the agent can’t tell which is which, so it pays the token and wall-clock bill everywhere. Late-stage executions resolve more often than early ones, which fits the obvious read that running tests earns its keep once you have a candidate patch worth checking, not as a reflex after every edit. The authors’ framing is the useful part: treat execution as a metered resource with an explicit cost-benefit curve, not a default capability.

That same reframing showed up in three other places over the last day, each pricing a different piece of the verify-remember-review loop.

CircleCI is rebuilding CI/CD around it. In a conversation with DevOps.com, CTO Rob Zuber argues that the inner loop is where quality work now has to live: when code generates in seconds, holding validation, tests, and review until a pull request turns the outer loop into a backlog of preventable rework, and the pipeline becomes the bottleneck. His prescription is to push deterministic guardrails right up against the moment code is generated, because agentic review needs checks it can trust before it can act on them. The metric shift is the tell. Zuber says the teams ahead are layering token consumption and pipeline efficiency on top of DORA, watching whether AI is compounding throughput or just compounding cost.

The cheapest reliable signal sometimes turns out to be another agent. The Red Queen Gödel Machine attacks a blind spot in self-improving agents: their search assumes a fixed verifier, but a benchmark that stops moving as the agent gets stronger stops measuring anything. RQGM folds the evaluator into the improvement loop, holding the criterion fixed within an epoch and updating it at epoch boundaries so the per-epoch guarantees still hold. On verifiable coding tasks, adding a complementary agent-as-judge code-review signal beats the prior SOTA on test pass rate while spending 1.35x to 1.72x fewer tokens. The sharper result is in paper reviewing, where the strongest baseline reviewer over-accepts AI-generated papers at up to 1.91x the human rate; an adversarial objective discovers reviewers that hold AI and human work to the same bar. Evaluator drift is a real failure mode, and it is now getting optimized against directly.

Memory has the same problem at a smaller scale, and a cheaper fix. Temporal Validity in Retrieval Memory shows that plain RAG has no model of time: when a function is renamed or an API restructured, the stale value and the current one sit at near-identical embedding distance, so cosine similarity separates a contradicted fact from a duplicated one at AUROC 0.59, barely above chance. MemStrata stores facts like RAG but retires a superseded value through a deterministic (subject, relation, object) supersession rule in a bi-temporal ledger, no similarity threshold and no LLM call. Across six benchmarks on a local 7B model it ties RAG on static knowledge and hits 0.95 to 1.00 on evolving knowledge where RAG lands at 0.20 to 0.47, driving the stale-fact-error rate from 15-40% to roughly zero at 2.1s retrieval latency against 16-18s for LLM-reranking baselines. A whole failure class closes with a ledger and a rule rather than a bigger model.

The last piece prices trust itself. Knowledge-Based Pull Requests starts from the observation that code is now cheap while understanding intent, negotiating scope, and owning long-term responsibility stay expensive. KPR treats an external contributor’s code, tests, and cleaned agent trace as knowledge sources rather than the default merge candidate: agents distill them into a human-confirmed package rendered as design memos, risk checklists, and test plans, then a project-owned trusted agent regenerates the implementation inside the receiving repo under its own conventions, tests, and security policy. That splits two decisions ordinary pull requests collapse, whether the knowledge should enter the project and whether one specific diff should merge. The pilot is small, seven merged public PRs stress-tested under description ablation, diff ablation, and synthetic poisoned patches, so this is a workflow proposal with an evaluation agenda attached, not a result yet.

What ties these together is a move from “run it / store it / review it because you can” to “what does this check actually buy, and is there a deterministic version that costs less.” Worth watching whether the next round of agent harnesses ship execution budgets and supersession ledgers as first-class controls, or whether teams keep paying the flat rate and reading it off their token bills.