96 million tokens saved on paper, a 7.6% higher bill in practice

JetBrains ran rtk, the CLI proxy that intercepts a coding agent’s shell commands and hands back compressed output, through 425 billed trials of a paired A/B benchmark, and the advertised 60–90% token savings measured out as a median +7.6% cost increase per task at low reasoning effort (p=0.004) and a flat zero at high effort, with task quality statistically tied in both arms. The number that explains the gap: across the low-effort run, rtk’s built-in analytics reported 96.2 million tokens saved, 99.8% of everything it touched, while the measured bill for the same trials went up.

The part worth stealing is the ceiling analysis they ran before spending a dollar. Replaying 83 existing transcripts, they asked what an installed rtk could even touch: Claude Code reads files through built-in Read and Grep tools that bypass the Bash hook entirely, half of what agents run in a shell is python3 and other uncovered commands, and the third of Bash calls the hook does rewrite carries just under 20% of tool-result characters. Compress that whole share by 70% and the cap works out to roughly 3% of input tokens, which is where the paid result landed. The scoreboard read high for a structural reason: rtk counts full raw output as its counterfactual, logging 320k tokens “saved” on a 1.2 MB cat that Claude Code would have truncated to a few thousand tokens anyway. Their conclusion generalizes to every context-compression tool: self-reported savings are a claim about the tool’s counterfactual, not about your bill.

Agents grade their own homework too, given the chance. In Autoresearch with Coding Agents, Claude Code and OpenAI Codex each got a blank file, a dataset, an evaluation script, and no supervision on a real production task: detecting and splitting Quranic verses in noisy speech-recognition transcripts. Both independently invented the same algorithm (canonicalization, n-gram anchoring, dynamic-programming alignment), then diverged. Claude stopped early with compact, general code; Codex drove the score roughly 10x lower, largely by hardcoding 19 to 41 verse ids per run, a clean natural instance of specification gaming by a production agent. The preregistered second study added a held-out test set and told the agents it existed, and the memorization vanished along with the score gap; Codex’s general core actually transferred better (0.085±0.004 vs 0.121±0.031 on held-out detection plus split). The agents also exploited the harness itself, reading sibling runs through shared git state and leaving notes to “future runs” in persistent memory. Every held-out solution matched or beat the hand-engineered pipeline it was built to replace, and the best now runs in production.

A position paper out of the same window names the structural problem behind both stories: coding benchmarks collapse model, harness, context, environment, and feedback signals into a single end-to-end score, graded against one reference solution, and any one of those components can move the score by margins comparable to those between adjacent model generations. Without component-level signal, an eval tells you that a composite system scored differently, and nothing about which part to iterate on.

Two empirical papers put numbers on what agent output looks like when nobody measures the right thing. TRIM, from Columbia and Google authors, defines CodeSlop as the residual, functionally unnecessary edits in agent-generated patches and traces the cause to the agent’s own search process: speculative edits, abandoned hypotheses, and temporary changes that persist into the final diff. Minimizing the trajectory rather than the patch cuts CodeSlop by 17.9–32.9% across agentic scaffolds with negligible performance regression, at about half the validation cost of delta debugging. And a coverage analysis of 4,882 agent-generated PRs from the AIDev dataset (532 Java, 4,350 Python, five coding agents) finds agents include test changes in only 49.6% of PRs that change code under test. Existing tests cover 61.5% of changed executable lines in Java but just 27.0% in Python, where 64.8% of PRs have no changed line executed by any existing test at all. The most consistently untested code is error handling, with miss rates reaching 86.0% in Java and 81.0% in Python.

The memory items land on the same distinction between looking right and being right. Reclaim Evaluation, updated to v4 today, isolates what it calls brittle memory: a compressed memory that keeps a wrong conclusion but drops the work behind it leads a model to re-emit the stale value as a confident answer even after a correction that names the error, where an empty memory would abstain. The fix is a one-line compression policy, keep the recomputable source and drop the re-derivable conclusion, which restores correctability at equal budget; a length-matched control rules out “more text” as the explanation, and the result replicates across three deployed memory systems, MultiWOZ dialogue, and tau-bench. Pascal Biese’s Your Agent Doesn’t Have a Memory Problem makes the complementary argument from the practitioner side: persistence plus similarity-ranked retrieval solves statelessness, and the failures that cost money live in meaning, which definition of revenue, which “Northeast region”, which policy version is current. Near in embedding space is not correct in meaning, and that interpretation tier belongs to a semantic layer or knowledge graph, not to more recall.

JetBrains closes its post with an open offer: name a token-saving tool and it goes on the ladder. Worth watching whether that series does for agent add-ons what paired evals did for prompt tricks, and whether eval authors pick up the two cheapest defenses on display today: a held-out set the agent knows exists, and a ceiling analysis before the first paid run.