We measure the model; the harness decides the run

Across roughly 400,000 Claude Code sessions logged between October 2025 and April 2026, Anthropic found that domain experts triggered about 2.4x more agent actions per instruction than novices, 12 versus 5, and came away with verified success on 28-33% of sessions against 15% for novices. When a run went sideways, novices abandoned 19% of attempts; experienced users abandoned 5-7%. The division of labor held steady across every tier: people made about 70% of the planning decisions, Claude handled about 80% of the execution. The headline most people will take from Anthropic’s study is that coding background is fading as a predictor, since every occupation landed within 7 points of software engineers on code-producing sessions. The more durable finding is the opposite of “anyone can do this now”: returns to expertise persist and compound, they just relocate from typing code to directing the system that types it.

Which raises the question the rest of today’s reading circles: what system are we actually measuring? A position paper from a team including Maria Gorinova and Amy Heineike argues our benchmarks answer that wrong. A coding agent in practice is not a model, it’s a composite of model, harness, context, environment, and feedback signal, and any one of those can move a benchmark score by the same margin that separates adjacent model generations. Current benchmarks collapse all of it into one end-to-end number, usually graded against a single reference solution that penalizes equally valid alternatives, with no component-level signal to tell you which part to fix. You can watch a score climb and have no idea whether the model got better or someone tuned the retrieval step.

If the agent is a system rather than a model, you can fingerprint it. Hamidah Oderinwale’s “Agent trajectories as programs” compares ten agents on SWE-Bench traces and shows they’re identifiable by procedural habits alone: a probe over those signatures attributes an unseen trajectory to the right agent at 85.7% accuracy, controlling for task leakage. Behavior clusters by lineage, models from the same release period look alike, and a distilled student sits at a Jensen-Shannon divergence of 0.25 from its teacher, roughly half the distance between unrelated pairs. The accompanying ProcGrep library reads traces top-down to audit how an agent works, not just whether it passed. Pair this with the position paper and a research direction comes into focus: as success rates saturate, the discriminating signal lives in the procedure, and that signal feeds task-aware routing, monitoring, and per-component cost analysis.

The same “measure the wrong thing” pattern shows up in memory. Dongxu Yang’s ForgetEval study makes the case that production failures are predominantly forgetting failures, not recall failures, yet nearly every memory benchmark scores recall. Comparing thirteen system configurations on a 385-case adversarial surface, the result turns on where the LLM sits relative to the control plane, the part that supersedes, releases, and purges stored facts. Deterministic primitives handle lexical and temporal cases but collapse on canonicalization, 5% on identifier obfuscation and 0% cross-lingual. An inscribe-time LLM recovers canonicalization fully but scores 0% on intent-aware deletion like prefix collisions and compound facts. A mutation-time hook recovers intent-aware deletion at 78-85% and lifts nearly every category at once, 91.7-93.2% overall, for $0.17 per 385-case run at 2.3 seconds per case while leaving the recall path untouched. Placement, not model strength, decides which failures you survive.

Two more pieces fill in the harness around the model. The configuration files that steer these agents turn out to rot like any other code: a catalog of AGENTS.md and CLAUDE.md smells from Marco Tulio Valente’s group mined 100 popular repositories and found Lint Leakage in 62% of files, Context Bloat in 42%, and Skill Leakage in 35%, with Context Bloat, Skill Leakage, and Conflicting Instructions frequently co-occurring. The instructions you write to constrain the agent are themselves an unmaintained surface. And on control flow, Junjia Qi and colleagues’ “LLM-as-Code Agentic Programming” argues that token explosion, control-flow hallucination, and unreliable stopping aren’t bugs to prompt away but consequences of handing deterministic looping, branching, and sequencing to a probabilistic system. Their inversion puts the program in charge of control flow and invokes the model only where a step needs reasoning or generation, building each call’s context from the execution call tree as a DAG so context length tracks call depth instead of accumulating over every step. In a computer-use case study it steadied long visual operation sequences that otherwise drift.

The connective tissue across all six: the model is the part of the agent we’ve gotten best at measuring, and it may be the part that matters least to whether a run succeeds. Worth watching is whether anyone ships a benchmark that scores harness components separately, because that’s the artifact that would let the iteration loop close.