A 60-90% token-saving claim measured out at 31.6%

Five codebase-retrieval tools, 48 Django questions drawn from SWE-bench, 261 runs, a fresh index per tool and one no-tools baseline. The best arm cut the agent’s output tokens by 31.6% against the bare agent, CodeGraph came second at 24.4%, and the rest landed between 6% and 15%. The advertised range for this category is 60–90%. Nobody reached it, including the author’s own tool, and the disclosure sits at the top of the post: they work on repowise, and the harness, the preregistration files, and the invalidated runs are all public.

The quality column carries the harder result. A blind judge scored every tool in the field, repowise included, between 0.04 and 0.25 points below the bare agent on a ten-point scale. The same benchmark moves 0.69 points when it is rerun unchanged. Every measured quality difference in the study sits underneath the evaluator’s own noise, which is the correct way to report a null and an unusual thing to publish about your own product.

Then the finding that reframes the whole exercise. The author ran the same questions, the same MCP servers, and the same indexes under Claude Code with Sonnet 5 and again with Opus, and most of the field was barely invoked. code-review-graph was never called once across 15 questions. Graphify was called three times, Serena four. Under Codex with gpt-5.6-sol, every tool was called on every question. Nothing differed on the tool side; Claude Code loads MCP schemas on demand, so the agent has to go looking before it can call anything and frequently never does, while Codex mounts them up front. What that arm measured was the harness, not the tools, which is why those tables were pulled out of the main comparison.

Two near-misses in the write-up are worth more than most published methodology sections. The first is a cost table showing one tool 43% cheaper than baseline in a run where Claude Code never invoked it, an artifact of prompt-cache warming: whichever arm ran first paid full price and later arms reused the cache, which is why the final tables report output tokens instead of dollars. The second is the framing that this whole category tends to publish. Loading one commit’s context through repowise takes 393 tokens against 13,984 for reading the changed files, a 35.6x compression on a single payload. Measured across a complete session, where agents re-read, backtrack, and re-plan, it nets out to 31.6% on Codex and 15.9% on Claude Code.

HarnessOpt-Bench landed on Thursday and makes the harness the object of measurement directly. An optimizer, meaning a frontier LLM paired with a coding harness, receives a target agent’s seed harness, graded evaluation feedback, and a fixed target-evaluation budget; it edits the prompts, tools, control flow, memory, and orchestration code, then nominates one final candidate that is scored by its normalized gain over the seed on a held-out test partition it never touches during search. A trusted execution environment enforces that boundary, meters the target agent’s resource use, and preserves every candidate version for audit. Across 5 frontier models, 4 downstream tasks, and 111 scored runs, the optimizer models separated more than the coding harnesses they acted through, native harnesses were not consistently better than the shared one, and gains varied widely by task and by seed regime. Harness optimization is now a scored capability with a protocol attached rather than a craft skill people trade in Discord.

Predicting Task Difficulty Without Rollouts attacks the cost side of all this. Estimating how hard a task is before you run it, across 17 agentic benchmarks spanning coding, math, ML, web navigation, and function calling, matters because rollouts in stateful environments are the expensive part of building either a benchmark or a curriculum. Two results carry beyond the immediate use. AUC can look healthy while the underlying difficulty estimates are poor, so the standard metric hides the failure. And the residual between predicted and observed difficulty works as a diagnostic on the environment itself: a task the agent finds much easier than its description implies is a contamination candidate, and one it finds much harder is a candidate for being infeasible as written. Token-level entropy turns out to be a useful predictive signal.

SkillTV-Bench moves the same skepticism onto the judges. It is 681 cases built from real agent trajectories across 50 tasks in eleven domains, aimed at verification of complete executions rather than final-response scoring, with the task-time skills exposed alongside inspectable artifacts and environments. The proposal that goes with it, SkillTV-Evolve, externalizes verification knowledge as a reusable JudgeSkill telling an agent judge what evidence to inspect and which failures are task-critical, then refines it in an automated loop over misjudged cases drawn from a disjoint development pool. The refined skill raised the same agent judge’s accuracy by 14.8 percentage points. Used for offline rollout selection, it took selected-trajectory success from 22.9% with one rollout to 45.5% with ten, which is the number to care about if you are picking a best-of-N trajectory in production and currently trusting a single judge pass to do it.

AssertMate applies the aggregate-disagreeing-perspectives pattern to unit test assertions on Defects4J. Static analysis and type-aware heuristics identify the assertion target, three separate agents predict the expected value by different routes (code generation, retrieval-augmented generation, and chain-of-thought), and an LLM judge selects among them. It beats prior approaches such as ChatAssert on compilation success, pass rate, and bug detection, improves mutation coverage and kill counts when composed with EvoSuite, and the ablation shows all three components contributing. The stated motivation is that one-shot prompting plus oversampling is vulnerable to model randomness, which is the same instability the repowise judge measured at 0.69 points and the same instability SkillTV-Bench is trying to price into verification.

Outside coding, Pascal Biese’s LLM Watch leads with an audit that holds the model family constant and varies only the access conditions: 401 stratified prompts from BBQ and SafetyBench, 4,812 responses, chat UI against API, search on against off, three runs each. The chat UI was less accurate than the API with search disabled. Turning web search on cut accuracy by up to 8 percentage points and in one case flipped which modality won on SafetyBench. Repeated runs of the same prompt disagreed on up to 21% of cases, the two modalities grounded the same answers in different citations, and abstention behavior shifted across both conditions. A single accuracy number from a single-condition API sweep is not describing the system anyone actually talks to.

The one number nobody has yet is the repowise author’s planned rerun of Claude Code with hooks that force tool adoption. That measurement separates “the retrieval tool does not help” from “the harness never asked,” and every result above hinges on the same distinction. Until benchmarks report the harness, the indexing cost, and the cache state as first-class variables next to the headline delta, the correct reading of most agent numbers is that you are looking at a property of the scaffolding with a model’s name on it.