Same pass rate, half the cost: coding-agent evals turn to cost per task

Matei Zaharia posted results yesterday from benchmarking coding agents on Databricks’ own internal engineering tasks: real work their engineers had done, with curated test suites, in Scala, Go, Rust, Java, TypeScript, Protobuf, and Jsonnet (thread). Three findings carry beyond their walls. Top-tier models have converged, open source included; Zaharia calls GLM 5.2 a major step forward for open-source coding agents. The Pi harness matched the stock setups’ success rate with both Claude Opus and GPT 5.5 at half the cost, mostly by sending the model smaller inputs. And cheaper per token did not mean cheaper per task: Sonnet 5 undercuts Opus on token price but burned more tokens on these tasks and produced lower-quality output, costing more end to end. Once pass rates tie at the top, the informative number is dollars per completed task, and the harness owns a large share of it.

A new ablation makes the same argument with controls (arXiv 2607.10569, posted Sunday). Yang, Yu, and Desell crossed three tool surfaces (full toolkit, bash-only, and a single execute_code MCP tool) with two agents, Claude Code and Codex CLI, on synthetic computation tasks and SWE-bench Mini, holding model, harness, and prompts fixed. Pass rates were statistically tied in every cell; cost was not. The lone execute_code tool came in cheaper than or tied with the cheapest tool-rich arm in three of four cells, and the one exception, SWE-bench on Claude Code at +14.4% and not significant, localized to failure cost on doomed runs rather than a per-edit tax on successful ones. Their conclusion meets the Databricks thread from the opposite direction: the cheapest tool surface depends jointly on task regime and agent design, and the signal lives in cache-adjusted cost, not pass rate.

What merge acceptance hides, this morning’s arXiv batch starts to measure. Xia and Miller tracked agentic and human contributions across 182 repositories after merge (arXiv 2607.09902), classifying the intent of every subsequent modification. Overall maintenance rates look similar between the two populations, but agentic contributions need significantly more corrective maintenance and introduce more security weaknesses and dependency vulnerabilities. The predictor worth keeping: each 10-percentage-point increase in a project’s no-review rate associates with roughly 6% more agentic maintenance burden. Merge acceptance was always a proxy, and this is the first longitudinal measurement of what it proxies badly.

The day’s retrieval result prices context the same way. Caumartin, Chen, and Costa study file-level bug localization as a representation problem (arXiv 2607.11046): across Long Code Arena and SWE-bench Verified, LLM-generated role-aware summaries of files beat file-path retrieval by up to 40% Hit@5 while carrying a representation footprint 10.4 to 20.9x smaller than raw source code. Fusing complementary representations and reranking with an LLM lifted an Agentless pipeline to 94% Hit@6 on file localization, 4.7% over baseline. Representation choice, priced in footprint tokens, is a first-class design decision in an agent’s retrieval stack, not plumbing.

Memory frameworks got the same independent-audit treatment. Wolff and Bennati built a reproducible testbed (arXiv 2601.07978, revised today) that measures accuracy alongside CPU time, peak RAM, disk I/O, and network for mem0, Graphiti, and cognee against RAG and full-context baselines on LoCoMo. Two clusters emerge: mem0, RAG, and full-context at 77 to 81% accuracy, Graphiti and cognee at 55 to 56%, with the gap driven by retrieval incompleteness rather than reasoning failure. The plain RAG baseline matched the top cluster at 8.4x lower total cost of ownership than mem0, and full-context forwarding underperformed mem0 despite shipping the entire conversation, so compression precision matters more than context volume.

Terence Tao spent Saturday’s post on the practitioner’s version of all this (blog). He ported roughly two dozen Java 1.0 applets from 1999 to modern JavaScript in a matter of hours, had the agent surface two bugs in his original code along the way (“a net wash as far as code quality was concerned”), and finally built the special-relativity spacetime-diagram tool he had shelved in 1999 as too much work. A Fields medalist’s revealed preference: agents clear the activation energy on projects that were never intellectually hard, just expensive in hours.

Four of today’s sources measured cost in four different units: dollars per task at Databricks, cache-adjusted cost in the ablation, representation footprint in the localization paper, total cost of ownership in the memory testbed. Pass rates and Hit@k have standard definitions; the denominator that now decides engineering choices has none. Watch who fixes that first, because their unit becomes the one everyone else reports against.