The harness is where the leverage went

Caveman, a token-compression skill for Claude Code, advertises a 65% cut in output tokens. JetBrains ran it through a paired A/B on SkillsBench, 82 tasks, forced on in every reply, and measured 8.5%. The gap is the whole story of the last day or two: the advertised number came from chat-style prose, but an agent’s token stream is code, diffs, tool calls, and exact error strings, all of which the skill deliberately leaves byte-exact. Compress the narration between tool calls and there is not much narration to compress. Their first 10-task run showed a 30% saving; it dissolved as the sample grew, which is the more durable lesson (never trust a k=1 eval). Quality was untouched: 8 tasks better, 10 worse, 64 tied, sign-test p=0.82. Safe, honest about style, oversold on savings.

That is a small study, but it lands on the same nerve as everything else worth reading today: the leverage in agentic coding has moved off the model and onto the harness, the process, the context plumbing, and we are only starting to measure that layer honestly.

Lilian Weng made the frame explicit. Her new post, Harness Engineering for Self-Improvement, argues recursive self-improvement will run through the harness rather than direct weight self-modification, and recaps 35 papers to back it. The line worth keeping: “Even when many harness improvements get eventually internalized into core model, the need to specify goals and context will not disappear.” She is a Thinky cofounder now, so the post doubles as a hint about where that lab is aiming, but the claim stands on its own. The product world moved the same way this week, with Anthropic pushing Claude Cowork to mobile as a background teammate and Google’s Gemini API adding managed background execution and remote MCP servers. The harness is the surface everyone is building on.

If the harness is the object of study, resolve rate is a poor instrument for studying it. TraceProbe makes that case with data. Coding agents are ranked almost entirely on whether the final patch passes the target tests, yet two agents can reach the same pass through very different processes, and a single pass/fail label says nothing about why a run failed or why an accepted run burned extra steps and tokens. TraceProbe normalizes 2,500 SWE-Bench Verified trajectories into a nine-type action taxonomy with deterministic effect labels, then names single-run anti-patterns (search loops, verification skips) and aligns pairs of runs to locate where they diverge. Two findings sharpen the critique: file-level choice is too coarse to separate success from failure, while function selection and completion behavior localize it; and even resolved runs differ in how fast they reach relevant code and how much failed work they carry. The trajectory holds the diagnostic signal the score throws away.

The trajectory holds more than diagnostics. Latent Programming Horizons in Coding Agents, from André Silva, Han Tu, and Martin Monperrus, probes the residual streams of the model inside a coding agent and finds they linearly encode properties of the evolving program: a logistic-regression probe on hidden states decodes whether the current code parses, passes its tests, reduces failing tests, and introduces regressions, reaching AUC up to 0.83 for correctness across two models and two benchmarks. The stranger result is that the representations run ahead of the agent’s own edits. Probes trained to predict the outcome of edits not yet written to disk beat chance up to roughly 25 steps in advance. They call it the agent’s latent programming horizon, and the probes transfer across benchmarks without retraining. The model appears to “know” where a run is headed well before it commits the change, which is an odd and useful handle for anyone trying to intervene mid-trajectory instead of grading the corpse.

Two papers went at the cost side of the same problem, from opposite directions. CoACT compresses the observations that pile up in an agent’s context as it reads files and runs tools. The idea is next-action preservation: a compressed observation is acceptable only if it induces the same next action as the raw one, which gives a concrete signal for whether the compression kept the information the run actually needed. A teacher model generates compressed candidates, an action-preservation reward filters out any that would change the next action, a length reward picks the compact survivors, and a lightweight compressor learns from that. On SWE-bench Verified across three agentic models, CoACT cut total token consumption by 33% while holding task-solving close to the uncompressed agent. NapMem works the memory side: instead of exposing long-term memory through a passive retrieval interface that hands the model pre-selected evidence, it organizes history into a linked multi-granularity pyramid (raw conversations, typed records, topic tracks, user profiles) and lets a reinforcement-trained agent choose which granularity to inspect before answering. On PersonaMem-v2, LongMemEval, and LoCoMo it stays competitive on memory-heavy tasks while preserving general reasoning and tool use. Retrieval becomes an action the agent takes, not a pipeline it sits downstream of.

The connective tissue across all six: we are learning to instrument the layer between the model and the task. TraceProbe reads the trajectory after the fact, latent-horizon probes read it from inside, CoACT and NapMem manage what the agent carries and recalls, and the JetBrains study is a reminder that claims about this layer are cheap to make and expensive to verify. Worth watching next is whether the process-level diagnostics start showing up in the leaderboards, or whether resolve rate holds its monopoly for another quarter while everyone privately knows it hides more than it reports.