Measuring How Agents Work, Not Just Whether They Finished

A single World of Code snapshot holds 850,157 Claude Code commits. The bot-account lookup that most adoption studies lean on recovers 28,154 of them, 3.3%, a 30x relative-recall gap. That number, from a validated census of 180M+ Git repositories posted in the last day by Arsham Khosravani and Audris Mockus, is the kind of measurement the field keeps skipping past, and it reframes most of what we think we know about how widely coding agents are actually deployed.

The method is the point: configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup, each hand-validated across 495 labels with per-cell precision and Wilson intervals. No single signal catches more than a fraction. Commit-attributed agents generate over 320,000 commits per month across snapshots from December 2024 to April 2026, Claude Code leading at 886,122 commits over 17,295 projects and dominating the silent, config-file-only adoption nobody counts. The sharper finding is channel disjointness. Compared against the AIDev pull-request census, the two populations barely overlap: a PR census misses 79% of commit-detected Claude Code adopters and essentially all Codex adopters. PR-deployed cloud agents surface as feature work; commit-deployed in-editor agents surface as maintenance. The observed work profile follows deployment and detection mode, not the tool. If your read on agent adoption comes from one channel, you are looking at a biased slice and calling it the field.

Process discipline becomes a measurable axis

For two years the only question benchmarks asked was whether the patch passed. RigorBench, from Meher Bhaskar Madiraju and Meher Sai Preetam Madiraju, argues that outcome correctness hides the difference between an agent that planned, verified, and recovered and one that thrashed its way to a green test by accident. It scores harnesses on five pillars (Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, Atomic Transition Integrity) across 30 tasks in categories with names that tell you what they probe: Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, Don’t Break the Build. Run in a with/without design against baseline assistants, structured process discipline lifted process-quality scores by 41% and, the result that matters, raised downstream outcome correctness by 17%. How an agent codes turns out to predict what it produces, and now there is a number for it.

That same belief, that the orchestration layer should reason about its own uncertainty rather than follow fixed rules, drives Bayesian control for coding agents from Theodore Papamarkou and colleagues. They formulate tool use as cost-sensitive sequential hypothesis testing: a controller maintains a belief over candidate correctness and decides, step by step, whether to gather more evidence, refine, verify, or stop. Across six generators and nine benchmarks the approach pays off exactly where intuition says it should, when verification is expensive and critics are informative but imperfect. The belief state also doubles as an interpretable correctness score that beats token-probability and raw tool-success signals for uncertainty quantification. It is the same lesson RigorBench delivers from the eval side, reached from control theory: the value is in deciding when to stop, not in always running every check.

Steering beats scaling, at least for critics

Steer, Don’t Solve from Shubham Gandhi and co-authors at CMU freezes the agent and trains a small critic to supply the strategy-level signal that end-to-end training plateaus on. The departure from prior work is timing: most code critics score a finished trajectory, this one gives intra-trajectory feedback while the agent still has moves left. A critic trained on CWM-32B trajectories transfers to two unseen agents for +3.0 to +3.8 points on SWE-bench Verified, and adding target-agent trajectories pushes gains to +4.4 to +5.2 on Qwen agents, at 30-92x lower cost than a strong teacher critic. On Qwen3-Next-80B-A3B the critic-guided system is both more accurate (25.2% vs 20.8%) and cheaper ($0.04 vs $0.11 per instance) than the agent alone, because steering also shortens trajectories. A small model placed at the right point in the loop outperforms more of the same model run longer.

Memory gets a systems audit, not another architecture

The agent-memory literature has spent months shipping architectures and grading them on end-to-end F1. Are We Ready For An Agent-Native Memory System? from Wei Zhou and colleagues treats memory as a data-management problem instead, decomposing it into four modules (representation and storage, extraction, retrieval and routing, maintenance) and evaluating 12 systems plus two baselines across 11 datasets. No architecture dominates; effectiveness tracks how well the memory structure matches the workload’s actual bottleneck. The ablations quantify each module’s effect on representation fidelity, retrieval precision, update correctness, and long-horizon stability separately, and the cost-performance read is concrete: localized maintenance beats global reorganization on cost. Benchmarking the black box as a black box was hiding which part was doing the work.

Metis from Zijie Dai and a CUHK-led team supplies the controlled comparison that question implies, isolating text memory against code memory over an identical set of experiences. The two forms trade off in construction cost, execution efficiency, and transferability, and neither alone suffices, so Metis runs a hierarchical dual representation: textual experience organized into execution plans, environment facts, and pitfalls, with recurring plans selectively crystallized into validated callable tools only when repeated reuse justifies the generation cost. On AppWorld it improves task accuracy by up to 20.6% over ReAct while cutting execution cost by up to 22.8%. The design choice that text-versus-code should be derived from the experience rather than fixed at design time is the part worth stealing.

What ties the day together is a shift in what gets measured. Adoption counted by the right channels, not the convenient one. Agents graded on how they work, not just whether they finished. Memory profiled module by module instead of scored as a monolith. Watch whether the next round of harnesses ships a RigorScore or a Bayesian stop rule as a default, or whether process discipline stays an academic axis nobody wires into the loop.