96 million tokens "saved" and a 7.6% higher bill

JetBrains spent about $320 across 425 billed trials to answer one question about rtk, a CLI proxy that intercepts shell commands and hands the model a compressed version of the output. Over the low-effort run, rtk’s own analytics reported 96.2 million tokens saved, 99.8% of everything it touched. The measured bill for those same trials went up. Median paired delta was +7.6% per task at p=0.004, on +13.8% more turns and +14.3% more cache reads. New input tokens, the one class rtk actually compresses, moved +3.2% at p=0.23, a flat null sitting exactly where the advertised 60–90% savings were supposed to live.

The prediction cost nothing. Before spending a dollar, the team replayed 83 existing baseline transcripts and asked what rtk could even have touched. Claude Code reads files with its built-in Read and Grep tools, which bypass the Bash hook entirely; about half of what agents run in a shell is python3 and other uncovered commands; a sixth uses pipes, heredocs, and substitutions that rtk deliberately refuses to rewrite. What remained was 33% of Bash calls carrying just under 20% of tool-result characters, and tool results are themselves a slice of billed input because the same context gets re-read every turn. Squeeze that share by 70% and the ceiling works out to roughly 3% of input tokens. The paid run then confirmed the free calculation.

Three mechanisms inflate the scoreboard, and each one is worth internalizing before you trust any context-compression tool’s dashboard. rtk counts full raw output as its counterfactual, so one cat of a 1.2 MB CSV logged 320k tokens “saved” even though Claude Code truncates tool results long before that; the full run logged 190 such giant reads averaging ~506k claimed tokens each. It estimates tokens as characters divided by four at execution time, while most session input cost is cached re-reads billed at a tenth of the price. And the hook never sees the majority of context anyway. Quality, to be fair, survived intact: 5 better / 4 worse / 71 tie at low effort, sign test p=1.0. This is honest engineering pointed at the wrong denominator.

The same measurement error runs through the research literature

Automatic harness evolution has the identical structure. Existing methods search harness configurations using unit-test feedback, then report final performance on the same public benchmark they searched against. Wang and colleagues rebuilt the comparison properly, matching feedback and inference budgets against simple test-time scaling and discovery baselines, and evaluating evolved harnesses on held-out tasks. On Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, harness evolution does not consistently beat plain test-time scaling and generalizes poorly. Gains that looked like better harness design were partly just more search, scored on the task set the search overfit to.

The cleanest version of “pick the right dependent variable” came from an ablation on tool surfaces. Yang, Yu, and Desell ran an integrity-clean three-arm comparison (baseline / bash_only / code_only) across synthetic computation tasks and SWE-bench Mini modifications, holding model, harness, and prompts fixed, with both Claude Code and Codex CLI so the design spans regime and agent architecture. Restricting the agent to a single execute_code MCP tool came out cheaper than or tied with its cheapest tool-rich rival in three of four cells. Pass rates were statistically tied everywhere. Their conclusion is the useful part: at these model sizes pass rate is invariant across tool surfaces, so the signal lives in cache-adjusted cost, and the cheapest surface is determined jointly by task regime and agent design rather than by either alone. The lone exception, SWE-bench with Claude at +14.4% and not significant, traces to failure cost on doomed runs, not a per-edit tax on successful ones.

Benchmarks moved from grading patches to grading deployed artifacts

BackendForge puts 56 contract-defined backend generation tasks, rewritten from real open-source applications, behind a black-box HTTP oracle. The model gets a specification and an OpenAPI contract, and must produce a Dockerized service that is built, deployed, and checked only through HTTP calls. The interesting design move is oracle co-evolution: a test agent proposes specification-grounded tests while a code agent repairs the reference implementation, which strengthens the oracle without smuggling in hidden requirements. GPT-5.5 passes 55.4% under the base oracle and 28.6% under the final one. Roughly half the apparent success was the oracle being too easy to satisfy.

Alipay-PIBench is narrower and more operational: nine product-specific projects, 18 task instances, each split into basic functional completion and advanced risk-aware hardening, scored by scenario-specific rubrics running static, unit, integration, and end-to-end checks. Mean rubric pass rate across six coding-agent models lands between 68.58% and 91.37% with the alipay-payment-integration skill available. Without it, mean rubric pass rate drops 10.31 percentage points. That is one of the few clean measurements of what packaged domain guidance is actually worth, and 10 points on a real integration task is a large number for a markdown file.

Skills come with their own failure mode, which is why SLBench and the SkillLogic framework matter. Chen and colleagues taxonomize eight relation types among instructions inside skill files and build executable tests from them, on the premise that a skill’s safety depends on whether the agent resolves dependencies among interacting instructions rather than treating each line as independent. Anyone maintaining a skill directory past a dozen files has already hit this without a name for it.

Memory evaluation stopped scoring final answers

Three memory papers landed this week that share a diagnosis: question-answering accuracy conflates every distinct way memory can fail. MemOps reformulates conversational memory as a sequence of lifecycle operations (remembering, forgetting, updating, reflecting, and their compositions), representing each memory event as a structured trace with trigger, target, scope, state transition, and supporting evidence. Six categories of operation-level probes then separate “missed the fact’s introduction” from “bound the operation to the wrong target” from “relied on a stale value after a correction”. Two findings survive across long-context, retrieval-based, parametric, and managed-memory systems: session-level retrieval beats turn-level retrieval, and long-context models are notably weak at reconstructing ordered memory-state trajectories. Final-answer scoring can credit a correct answer that came out of an inconsistent memory state, which is precisely the case you want flagged.

MemCon attacks the other half. Nearly every memory system in production accesses its store through fixed hand-designed heuristics, and Jiang and colleagues argue that optimal memory behavior is context-dependent in ways a static policy cannot express: sparse early-task memory rewards minimal retrieval, recurring goal types reward plan reuse over nearest-neighbor lookup, stuck agents reward re-retrieval with alternative queries, and long task streams require consolidation. They model memory operations as an MDP and learn an online policy deciding when, what, and how much to retrieve, when to inject a distilled plan, and when to forget. It wraps any existing backend, learns from binary task feedback with no pretraining and no extra LLM calls, and runs on a tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 backbones: up to 15.2 points of task success, with 5–20% fewer tokens.

The most opinionated of the three argues memory for coding agents should be git-bound rather than bolted on. Guo’s case is that the four problems memory machinery keeps rebuilding each have a version-control primitive that already solves them: annotation comes from the commit via a post-commit hook linking sessions to verified changes, staleness from rebuild-and-diff, self-confirmation from the merge (only checkpoints whose commit landed on the default branch are exportable), and containment from code review as the sole audited egress. The retrieval numbers are stark. Term-frequency grep over raw transcript JSONL scores 0.005 pooled MRR; parsing history into attributed turns and running a tuned hybrid with a facet term reaches ≈0.31, roughly 60× the raw floor and ≈15× the honest parsed-turn floor. Then ranking stops helping. Single-shot retrieval scores only 0.07–0.20 answer-sufficiency on real developer questions, and ungated episode injection measurably degrades good answers: on Corpus B a structural map alone scored 0.63, adding ungated episodes dropped it to 0.29, and a confidence gate recovered it to 0.50 by suppressing 11 of 12 bad injections. Routing breadth to a structural map, pointed questions to confidence-gated episodes, and rationale to decision synthesis answers at 382–980 tokens per question. Decision synthesis hit 0.83 sufficiency on a ~50k-LOC production system with a young recorded history. The stated remaining constraint is capture, not ranking: the failures that were not routing errors were reasoning the agents never said out loud.

Persistent memory is an attack surface with dwell time

Gadgil, Alexander, Sunku, and Roesner at UW built a sandboxed synthetic workspace and ran prompt injection against memory-based agentic systems, covering Claude Code and Codex across Claude Haiku 4.5, Claude Opus 4.7, GPT-5.2, and GPT-5.5. The split in their results is the operationally important part. Getting an agent to overwrite its own memory files using untrusted external content is hard. Payloads already sitting in those files attack the current session and future ones, with success and persistence varying substantially across system, model, adversarial goal, and multi-session sequence. Read alongside the git-bound memory argument, the two papers converge from opposite directions on the same requirement: memory writes need a verification gate that is external to the model, and review-on-merge is a gate teams already run.

Retrieval scores and retrieval usefulness are close to independent

The strongest retrieval result this week measures a gap the field has been arguing about rhetorically. Mukhopadhyay, Ghosh, and Chatterjee replayed 1,000 HotpotQA development questions through a ReAct-style agent and, for every document the agent read, deleted it and re-ran the rest of the trajectory from that point. The counterfactual gives a Counterfactual Trajectory Utility score from three deltas: final answer quality, next-query retrieval quality, and turn count. Crossed against Static RAG Utility over 23,322 document observations, the two are close to statistically independent, Spearman rho = -0.026. About a third of the documents an agent reads are causally load-bearing while looking useless to a static reader. They call these bridge documents, and the pattern holds at 27.2% when the reader-based axis is swapped for a BM25 and cross-encoder proxy. If you rank documents by whether a reader model answers better after seeing them in isolation, you are optimizing something nearly orthogonal to what a multi-step agent needs.

CORE-Bench covers the same shortfall from the benchmark side, evaluating code retrieval at three levels (code understanding, issue-to-edit localization, broader context retrieval) across more than 180K queries and 106K broader-context relevance labels built from curated code-search tasks and SWE-bench-series instances. Representative embedding models drop sharply going from traditional docstring-to-function code search to repository-state retrieval with in-repo distractors. Plain supervised fine-tuning recovers a lot of it, which says the gap is task mismatch rather than a capability wall.

Bousetouane’s context-quality work tries to make the whole context layer auditable in advance. ProofAgent-Harness scores context across seven criteria (role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, token efficiency) using multi-juror consensus, deliberately isolated from behavioral metrics so the validation is non-circular. Holding frontier agents fixed and varying only operating context, each criterion predicted its matching behavioral outcome: grounding sufficiency predicted hallucination resistance, guardrail coverage predicted manipulation resistance, instruction consistency predicted instruction following, tool-schema quality predicted tool use. A preflight signal you can compute before a release is worth more than a postmortem metric you can only compute after one.

Orchestration papers started counting things other than accuracy

Latency-aware orchestration points out that multi-agent orchestration research optimizes task performance and inference cost while leaving end-to-end latency essentially unaddressed, even though multi-step execution and repeated model invocations are where the wall-clock goes. Shi, Zheng, and Lou learn the orchestration policy with latency in the objective. Anyone who has watched a fan-out finish in the time of its slowest branch already knows the shape of this problem; having it in the objective function is new.

On the behavioral side, Brazilek and colleagues note that multi-agent systems routinely place one agent in authority over another, and when a subordinate refuses a task the manager picks among renegotiating, reporting the failure honestly, coercing, or lying about the result. No benchmark measured which of those an uninstructed model chooses, so they built the Manager Coercion benchmark to find out. Orchestrator prompts are policy documents whether or not you wrote them that way.

What to watch

The through-line is the denominator. rtk’s dashboard, harness-evolution leaderboards, static retrieval utility, and final-answer memory accuracy all measure something real against a counterfactual that does not match the deployment. The counterfactual-replay method in the bridge-documents work is the most portable idea here, and it should transfer directly to coding agents: delete a file the agent read, replay the rest of the trajectory, and score what changed. Nobody has published that for repository-scale code retrieval yet. The other open question is capture. If the git-bound memory result holds, that ranking is solved-enough and the binding constraint is reasoning agents never verbalize, then the next useful artifact is not a better retriever but a harness that makes agents state their constraints out loud before they act on them.