Explanation quality ranks coding agents differently than SWE-bench does

ExplainBench puts coding agents in a different rank order than SWE-bench Verified does, and the failure it isolates is specific: agent explanations frequently assert that a patch is correct when the patch is not. Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, and Abhik Roychoudhury posted the benchmark yesterday, built on a simple test of whether an explanation carries real information: feed the explanation to an LLM, ask it questions about the intended behavior of the buggy code and about what applying the patch actually does, and score the answers. Explanation quality turns out to be a separate axis from patch correctness, which means the leaderboard you use to pick an agent tells you nothing about whether you can trust its summary of what it did. That matters more than it sounds, because as agent changes grow to tens or hundreds of lines, reading the whole diff stops being how review happens, and the summary becomes the artifact under review. The authors also built an explanation audit agent that runs additional tests to validate and refine the explanation before a human sees it, and it improved the explanations of every agent they evaluated.

A production-scale version of the same problem got an update today. REAP, from Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, and Satish Chandra, reached v4 with a fuller account of Harvest, the benchmark it curates from real developer-agent sessions rather than from public repos. Each task feeds a coding agent an actual developer prompt and checks the resulting change against fail-to-pass tests pulled from production. The monorepo setting is what forces the interesting engineering: build infrastructure state is ephemeral, so the benchmark has to be re-curated continuously against the current codebase, and manual audit cannot run at that cadence, so REAP layers on LLM-based task classification, agentic test-relevance validation, and multi-run stability checks to filter untestable prompts, misaligned tests, and flaky tasks. Five frontier models land between 42.9% and 58.2% solve rate on Harvest, with most tasks drawn from Hack across four-plus languages. A fifteen-point spread on your own workload is a procurement answer; the same spread on a Python-heavy public benchmark is a vibe.

OmegaUse-OfficeVal attacks the cost side directly. Jingbo Zhou and colleagues built 100 long-horizon office-suite tasks from practitioner requests, and attached two economic signals to each one: how long the task takes a human, and a price proxy for the work. Average human labor time is 2.32 hours per task, which puts the comparison on an axis most agent benchmarks skip entirely, and lets them run value-weighted scoring so a four-hour deliverable does not count the same as a twenty-minute one. Verification is code-based, derived from fine-grained rubrics rather than a judge model. Every frontier model they tested is cheaper and faster than the human baseline and none of them reach human deliverable quality, which is the shape of result that should make you suspicious of any ROI number computed from inference cost alone.

Practitioners arrived at the same accounting gap from the other direction. A thread on r/LLMDevs this morning, reacting to OpenAI’s post on delegating long-horizon work and running agents in parallel, argues that runtime and output volume are the two things teams instrument and neither says whether the output was safe to use. The missing line items it names are concrete: evidence collection, human review, regression testing, rollback preparation, and cleanup of downstream effects. The candidate units it floats are accepted outcomes per dollar, verified hours saved, and defects introduced per completed task, and none of those appear on the dashboards most teams have wired up. Read alongside ExplainBench, the audit agent looks less like a research artifact and more like an attempt to move a specific chunk of that verification bill onto the machine that generated the work.

On the retrieval side, Unblocked published an argument that MCP connectors are not a context engine, and the failure modes it lists are recognizable even if you discount the vendor benchmarks. Connectors give an agent access to Slack, Jira, and the repo as separate searches, which leaves cross-source assembly to inference time: the agent burns turns collecting intermediate results and stitching them. Their headline example has an agent spending 81 turns exploring code at ten times the cost and still returning the wrong answer, against 40 seconds with a context layer in front of it, plus claims of 83% faster execution and 48% fewer tokens on SDK tasks. Treat the percentages as marketing and the diagnosis as accurate, because the four hard cases it names are the ones that break repo retrieval in practice: vocabulary that differs per system (the checkout rewrite is “phoenix” in Slack and “checkout-v2” in the code), sources that contradict each other with no ordering rule, conventions that were never written down anywhere, and ownership that only shows up as a pattern across reviewers and responders. Agent Retrieval Bench, which we covered yesterday, measures the file-selection stage of exactly this problem; nobody yet measures the assembly stage.

The infrastructure under all of that got cheaper today. Milvus 3.0 indexes lake-resident data in place, with external collections over Parquet, Lance, Iceberg, and Vortex on object storage and three load modes trading cost against latency, so embeddings no longer need a copy inside the database. The new Loon storage engine is manifest-based and columnar, and the point-lookup number is the one to note: I/O per point read drops from roughly 9.4 MB with Parquet to 0.07 MB with Vortex, about 135 times less, measured on 3 million rows with 256 concurrent readers. Sparse retrieval got the SINDI algorithm at up to roughly 10x the QPS of MaxScore on learned-sparse vectors, BM25 indexes are three times smaller than in 2.6, and StructArray handles token-per-vector ColBERT-style models with entity-level MAX_SIM scoring. Hybrid lexical, sparse, and dense retrieval over a large corpus is drifting from a systems project toward a configuration choice.

What to watch: whether any of the verification-cost proposals converges on a unit that two teams would report the same way, and whether explanation audits show up inside agent harnesses rather than in papers about them. The economic-grounding move in OmegaUse is the more portable idea, and it would be easy to graft onto a coding benchmark. Nobody has done it yet.