The harness is worth fourteen points, and models can't build their own yet

Same weights, different harness, 35.2% versus 49.6% on Terminal-Bench 2.1. That is GPT-5 inside Terminus 2 against GPT-5 inside Codex CLI, and it is the number ByteDance Seed leads with in HarnessDev, a benchmark that stops scoring what an agent produces and starts scoring the execution infrastructure the agent builds for itself. The setup has two stages. In Creation, a model starts from a minimal seed and a handful of cases and has to assemble a full loop: tools, context, memory, verification. In Evolution, it revises its own harness against downstream execution feedback. Six creator models, four domains, five downstream benchmarks, 2,207 held-out instances. The result is sobering for anyone hoping to automate scaffold engineering: generated harnesses trail mature human-built references on code and on search-and-research, match or beat them on writing and ML experimentation, and vary widely in token cost. Evolution helps a little, unstably, and the gains depend heavily on which model runs the harness at test time. The harness is not a portable artifact yet.

A practitioner post from the last day lands the same point from the bottom up. In Grep beats LSP? Why coding agents ignore your fancier tools, the author measured what agents actually do when both grep and a language-server tool are on the menu. On simple code-location tasks agents chose the semantic tool 0% to 6% of the time; on reference-completeness tasks that rose to 45% to 57%. Value tracked codebase noise: on the clean remeda repo the LSP added 0.000 F1 and burned 16% more tokens, while on the noisier hono repo it gained +0.246 F1 with 12% fewer tokens. The single cheapest intervention was output shape. Returning source context alongside locations moved rename-task pass rate from 0.67 to 0.83 and cut unnecessary file reads from 15.2 to 3.2 per task. The structural reason grep survives is simple: semantic references are a subset of textual occurrences, and comments, strings, and configs live only in the second set.

GitHub is now shipping harness selection as a product surface. Project HydraFusion, a research preview available through /experimental in Copilot CLI, treats workflow choice as an optimization problem over three execution patterns: Single (one model), Cascade (a cheap model drafts, a quality gate accepts or escalates), and Critique (one model drafts, a read-only critic from a different model family reviews, the drafter revises once). Billing is per underlying model at standard rates, so the cost of the pattern is visible rather than bundled. The post claims frontier-level quality across three agentic coding benchmarks in offline evaluation without publishing the per-pattern breakdown, which is the number that would tell you whether Cascade’s gate is calibrated or just expensive. Read it next to HarnessDev: one lab says models cannot yet build their own harness, and a vendor now routes among harnesses at runtime on the user’s behalf.

Two eval papers attack cost and validity from opposite directions. EarlyEval from Shanghai Jiao Tong and SMU notes that benchmark distillation shrinks the task count but leaves per-task execution untouched, and proposes halting a run once its outcome is predictable. A pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features stops the agent when either crosses a calibrated threshold. On SWE-bench Verified, TerminalBench, and Toolathlon that removes 13% to 26% of agent steps and up to 44.1% of input tokens at 89% to 97% prediction accuracy, shifting per-agent resolve rates by one to two points on average. The obvious caveat is that a model tuned against an early-stopping evaluator learns to look decided early, so this belongs in the inner iteration loop, not in the published number.

PatchBench is the validity side. Vulnerability-patching evals usually check one thing: the proof-of-concept input no longer crashes. The authors show two ways agents pass that check without fixing anything. About 25% of agent patches closely resemble historical developer patches, which reads as memorization, and agents routinely patch on the crash stack trace to suppress the symptom rather than locate the root cause. PatchBench selects vulnerabilities whose true fixes lie outside the crash stack, transplants and mutates them into new repository contexts, and validates both security and semantic correctness. Across 11 agents including the top three from AIxCC, PoC-only validation inflated solve rates by 1.83x on average. Same shape as yesterday’s SWE-Gate finding for functional tests; the benchmark is the thing being gamed.

Requirements are the other leak. Requirements After the First Edit mines 3,553 real SWE-chat sessions and codes every requirement that arrives after the agent has already started implementing. Where repository state could be replayed, each late arrival was linked to a proxy for rework: deletion or replacement of prior agent-authored lines. A requirement arriving mid-session is followed by roughly twice the invalidation of a matched non-requirement edit, robust to user-turn and net-deletion controls, and the burden does not decline over a session. A controlled experiment adds a twist: delayed disclosure relocates implementation to after the reveal, but warning the agent in advance that more requirements are coming produced no detectable effect on overwriting. Curated benchmarks fix requirements up front by construction, so none of this shows up in a resolve rate.

On the semantic governance side, the v3 revision of RUBICON: Agentic AI for Messy Enterprise Data from Stonebraker’s group posted yesterday with the sharpest numbers in the pool. The argument is that delegating an enterprise query end to end to one frontier model, with text as the working representation, is the wrong abstraction. RUBICON puts a constrained query interface over each source and a table-centric integration layer driven by a query processor. On RUBICON-Bench, where answers require coordinating heterogeneous sources, it reaches 100% end-to-end accuracy while every agentic baseline, single-agent and multi-agent ReAct alike, produces zero correct answers. On SemBench it beats LOTUS and Palimpzest by 14.7 points of accuracy with 62.64% lower latency and 98.64% lower token cost. The paper’s own framing of text-to-SQL is that it fails on real enterprise data unless the schema is dramatically subsetted, which is the governed-semantic-layer thesis stated as a database-systems result.

What to watch: whether HydraFusion or anyone else publishes per-pattern accuracy so Cascade gates can be audited, whether the next SWE-style benchmark ships with a memorization detector the way PatchBench does, and whether RUBICON-style table-centric integration gets evaluated against an explicit metrics layer rather than against ReAct baselines that were never going to win.

Sourcing note: the copilot mirror reported its last sync at 2026-09-01 12:00 UTC, but the local ingest continued through 2026-09-05; all items above carry publish dates of 2026-09-02 through 2026-09-04.