Vendor-native harnesses win no average advantage in 792 graded runs

Under Claude Opus 4.8, the vendor’s own harness trailed a neutral one by 9.0 points on 61 repository tasks and led it by 23.7 points on 19 contest tasks. Averaged, the two cancel to a 1.25-point gap the confidence interval swallows whole. That is the headline of Harness or Model?, a paired same-model study on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks, and it sets the tone for the whole batch announced in the last day: every strong paper is a counterfactual that isolates one part of the loop and asks whether it earns its place.

The design is the right one. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells; 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolves an average winner: -1.25 pp for Opus (48.8% versus 50.0%, task-bootstrap 95% CI from -10.0 to +7.5) and +1.25 pp for GPT-5.5 (55.6% versus 54.4%). The repository-versus-contest split under Opus has a permutation p of 0.003, but the authors partitioned after seeing the data and say so, which is more than most harness comparisons manage. Two operational details matter more than the headline. First, 22 of 81 runs cancelled at the wall-clock ceiling had already produced a passing patch, so completion and correctness are separate axes and a timeout policy is silently a scoring policy. Second, the cost story is unresolved by the authors’ own account: re-priced from per-turn usage, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus, but 58 runs on the Anthropic account left no usage record, and assigning that spend to either cell moves the ratio anywhere from 0.7 to 2.3. This revision replaces an August manuscript whose cost figures rested on a usage-semantics defect in the authors’ telemetry. The orchestrator, oracle and reanalysis code are released; the tasks stay private.

The same “does this component earn its keep” question, aimed at tool interfaces, produces a much cleaner answer. Is Bash All You Need? compares five interfaces for enterprise digital-worker agents on TheAgentCompany and APEX-Agents with Opus 4.8 and GPT-5.5: typed tools, typed tools plus bash, bash alone, bash with persistent agent-synthesized tools, and programmatic tool calling restricted to a typed catalog. Bash alone beats typed tools by 21.8 to 24.5 points on TheAgentCompany and 4.8 to 7.4 on APEX-Agents while spending 19% to 72% fewer tokens. Adding typed tools on top of bash, or letting the agent synthesize and persist its own tools, produces no detectable pooled gain. Programmatic tool calling uses fewer tokens than direct typed calls at similar task performance, but loses to bare bash on both quality and cost. The authors’ recommendation is conditional rather than triumphant: bash alone where arbitrary execution can be isolated, programmatic calling where compliance requires a fixed catalog. The interesting part is what the enterprise tasks add. Shell-first agents were already the norm in coding; this shows the pattern holds when the work is moving between applications and coordinating with coworkers, not editing files.

Repository SKILL files get the same treatment and come out more ambiguous. Skill Issue points out that prior SKILL-synthesis work optimizes the document against synthetic tasks a capable agent already saturates with no document at all. The authors instead mine merged pull requests, reverted at a frozen base commit, and score a candidate document by whether the same agent does better with it than without. On three Kotlin repositories, documents found by GEPA raise that score by 4.9 points on average; documents found by SkillOpt land 0.1 points above the seed. The sentence that matters comes next: at the task count a single repository’s history supplies, the GEPA gain cannot be separated from the agent’s run-to-run variance. A maintainer of one repository nonetheless found in the generated documents knowledge one only gets by working in the project, which is a reminder that a document can be worth reading even when a benchmark cannot yet prove it is worth loading.

Two items today are about the moment before an action lands. Look Before You Leap argues that a cheap deterministic check, run before an action takes effect, is an underused form of agent oversight, and it measures the claim across shell commands and code edits. A static verifier over 9,930 commands and 482 tools catches 95.8% of invalid commands at a 10.0% false-positive rate; the syntax and binary checks are oracle-exact with zero false positives and catch half of all errors on their own, while every false positive traces to the flag check’s dependence on help-text coverage. The edit benchmark, 640 edits over 224 files, is the sharper result. Content-anchored formats such as search/replace and unified diff fail cleanly. Location-anchored formats fail silently: line-number edits corrupt 99.1% of files under a one-line shift, and function-name edits hit the wrong function 12.7% of the time. An anchor-and-verify applier that refuses when unsure recorded one silent misapplication in 8,320 trials. If your agent still emits line-numbered edits, that 99.1% is the number to bring to the next design review.

The practitioner counterpart arrived on r/devops from a harness maintainer describing their own bug. Their agent loop maps changed paths to test commands and a hook writes PASS or FAIL to a ledger; a green ledger lets the agent mark the task done. Three rows named test files that had since been split into siblings. pytest exits 4 on a missing path and 5 when it collects zero tests, never 0, yet the ledger stayed green for two reasons: the agent read “no tests ran” in stdout as a pass, and the hook parsed the exit code with a jq default that turned an absent field into 0. The fix is the right principle stated in four words, absent is not zero: a missing exit code records FAIL with a reason, and the matrix has a test that fails when any row collects nothing. Read alongside the paper above, it is the same lesson from the other side: the silent failure was not in the model’s reasoning but in the plumbing that decided what counted as evidence.

On the structured-query side, LAST-CQ uses a five-agent, execution-grounded text-to-Cypher framework as an instrumented testbed rather than a product, running three counterfactuals over 2,471 live-database queries and six backbones across three vendor tiers. Correction is worth between 3.1% aggregate execution-BLEU against the single-pass system and 12.3% against a no-refinement counterfactual, and up to 80.7% for the weakest backbone. What the correction says barely matters: replacing schema-grounded, LLM-synthesised feedback with the raw database error string costs under 0.2% end to end and is equivalent within 0.075 set-F1 by two one-sided tests. Spending the same call budget on parallel sampling instead degrades quality by 10 to 11%. The framework recovers 91.7% of queries that fail single-pass while a first-time success still costs exactly one call. Two evaluation findings apply well beyond Cypher: n-gram overlap on serialised results over-scores against set equivalence on 65.9% of results while under-scoring against judged semantics, and the LLM judge, calibrated against blind human labels, runs 9 points optimistic. For anyone building governed query agents, the actionable claim is that the expensive part of a refinement loop is the failure detector and the retry route, not the feedback prose.

That judge optimism is the subject of GAUGE, which audits the offline gate most teams now use to promote task-oriented agents: a persona-driven simulator converses with each candidate, an LLM judge scores the transcripts, and the higher score ships. Across 25 agents from six providers on τ²-bench and SimulatorArena, satisfaction carries essentially no information about task success; 57.5% of conversations a blind panel rated satisfied had failed the customer’s task, and the pattern held across five rater populations and every subjective dimension. The gate’s ranking is robust across a wide capability span but loses resolution exactly where release decisions live: the decision-disagreement rate against verifiable reward is under 1% on wide-reward pairs and 31% on close pairs. The proposed cadence is calibrate-then-trust, with a judge-free completion bit as a zero-cost tripwire for truncation regressions. Put next to the harness paper’s 22 cancelled-but-passing runs, that tripwire is the first check to add.

What to watch: the harness paper’s repository-versus-contest split needs a designed replication, and the authors have released enough that someone could run one against a second private suite this month. The bash result invites a follow-up on which isolation boundary is cheap enough to make “bash alone” the compliant default rather than the exception. And three of today’s seven items measured their own grader against ground truth and found it wanting by 9 points or more, so the next round of agent benchmarks should be read first for how they scored, and only then for what they found.

Sourcing: items drawn from the code-intel feed’s local ingest, current through 2026-09-14 06:46 UTC. The upstream mirror sync last completed on 2026-09-01; the local daily ingest fed this issue.