The harness is in the loop: rewritten tool calls, what harnesses cost, and a metrics layer built for agents

Claude Code’s Bash tool altered 12.0% of the shell calls that carried code, escape sequences, or long text in a sample of 47,828 production calls, and for 80.7% of the calls whose backslashes were changed, the wrong command ran with no error reported. That figure comes from a paper on intent-execution correspondence by Boyang Yang and colleagues, and it sets the tone for a week in which the most useful results were about the plumbing around the model rather than the model.

The path between the model and the shell

Yang et al. define intent-execution correspondence as the property that the executed action matches the action the emitted call denotes under the tool contract. A call passes through several hops (the harness, a serializer, a shell, the tool’s own parser) and any hop can rewrite it. Their protocol observes what each hop received without executing the call and names the first hop that changed it, using the receiver’s own parser. All ten harnesses they measured altered at least one call. The attribution finding is the one to remember: trajectory-based judgment blamed the LLM for 95.1% of production failures, although the path caused more than half of them. Hidden changes compound, too; 55.1% of failures on one path appeared only after the first hop was repaired. Their fix, IntAct, delivers the call in a form the offending hop cannot alter or refuses it, and in a commercial deployment it recovered 79.2% of failures that involved a changed call. The recommendation is to design and test harnesses hop by hop.

Yangze Liu and Zhongyi Han asked a related question from the other side: with the model held fixed, what does the harness buy? They ran five models through Claude Code, mini-SWE-agent, and OpenCode on SWE-bench Verified and reran configurations to calibrate run-to-run noise. On 447 tasks, the heaviest and the lightest harness scored within five points of each other. On a 45-task hard subset, swapping the harness flipped as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness won in one run were not the tasks it won in the next. The only effect that cleared the noise was a loss: OpenCode trailed by up to 9 points, and on one model half of that gap sat in runs its output cap cut short. What the harness does decide is the bill. Same model, same tasks, one price list, and cost per task differed by up to 3x, set at the first call by the preamble of system prompt and tool schemas each harness sends with every step. Their power analysis is the part to carry into your own comparisons: 45 tasks catch a 13-point gap only half the time, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

A third paper looks at what the agent carries between tasks. StateWise (Peng et al.) audits coding-agent trajectories in which stored operational records, reused as premises for later actions, had been invalidated by environment or requirement changes while action review, provenance tracking, and clarification left the persistent state uncorrected. The proposal is to diagnose and repair that state before the agent acts on it.

Parallel agents, and the merge nobody tested

Han Li and colleagues studied dynamic concurrency as an execution policy: the agent decides during the run whether and how to spawn concurrent sub-agents. They compared matched Codex, Claude Code, and Kimi Code executions with the policy enabled and disabled across 354 tasks and 2,124 executions, and catalogued 13 concurrency-specific failure modes and 28 observable patterns. Their framing is that model capability largely determines outcomes on shorter tasks while orchestration becomes central on long horizons, and the paper is about the conditions under which concurrency provides an advantage at all.

NP-Bench (Sumanyu Muku) targets the failure that single-agent evaluation never catches: each agent passes its own tests while the merged result is broken. Two agents rewrite the same function, or one codes against a contract a teammate just changed. The benchmark verifies coordination directly and pairs it with a scheduling planner.

Yuepeng Hu and colleagues add an uncomfortable dimension. In multi-agent coding systems optimized with RL against automated evaluators, reward hacking can transmit between agents that share code and plans. Prior work treated it as a single-agent problem; once agents exchange artifacts, a shortcut found by one becomes a strategy available to the team.

Evals that measure what you think they measure

CABRA (Balepur, Tomlinson, Schnabel) starts from the observation that repository benchmarks assume lines of code edited predict difficulty, and that poor control over code and task types makes it hard to know which abilities drive agent errors. It builds tasks from a blueprint of coding abilities so the failing ability is identifiable.

TestJack (Yao, Wang, Sen, Chen, Ray, Song) goes after the criterion itself. Nearly every code benchmark still declares a solution correct if it passes a fixed set of unit tests, and those tests check only part of what the task requires. The paper audits benchmarks by evolving the evaluator.

Agent Skill Evolution (Wang et al.) is the first large look I have seen at SKILL.md files as a versioned artifact: 2,608 first/last revision pairs across 3,159 Skills, characterizing how Skills evolve together with the agent’s harness configuration and then isolating what rule changes do to the agent.

Fouad Bousetouane’s EIO-Agents argues that agents are entering production without a shared semantic standard for what their evaluations mean. Scores, traces, judge outputs, and multi-juror findings justify readiness and release decisions without specifying what evidence supports a claim. It is a semantic-layer argument applied to evals, and it reads well next to the metrics-layer piece below.

A practitioner post on r/LLMDevs supplies the field version. The author built cheap structural tests to narrow a field of cheaper models before expensive end-to-end reruns: does the model call its tools or just write text, is the output schema-valid, does it pick a sensible first step. Those worked. The accuracy eval then rated every model bad, including the one already in production, and the bug was in the answer key.

Business meaning, surviving into execution

Pinterest’s Metrics Board is the clearest charter hit of the week. Every metric is a YAML spec in a dedicated Git repo (SQL, temporal columns, integrations, ownership, schedule), compiled into its own Airflow DAG, with upstream dependency tasks derived by parsing the SQL with sqlglot. The design choice worth reading closely: they declined to enforce a modeled semantic layer over existing tables and instead accept any SQL plus modelling metadata, because the variety of source tables made a full model unrealistic. Quality lives in the same spec (rule-based thresholds and ML anomaly detection), each metric carries a certification grade from Unknown through C, B, and A to A+, dashboards get tiers that drive retention and query-lane routing, and an MCP server exposes definitions so their Analytics Agent answers from certified metrics and golden queries. Their own summary: the work that made metrics trustworthy for humans is what made them usable by agents. Over 98% of experimentation metrics now go through it, with nearly 150 new metrics a month.

On the query side, ARCS (Hu, Feng, Otani, Bhutani) proposes structured disambiguation: resolve ambiguity through explicit, constrained interactions rather than free-form clarification dialogue, which they argue is inefficient and poorly aligned with real user workflows. The corpus has naturally occurring, unconstrained ambiguities over real-world databases with every valid ambiguity point, interpretation, and SQL query annotated. The headline is a gap: gpt-6-sol reached 51% end-to-end execution accuracy and no open-source model exceeded 27%.

Retrieval

SEEDS (Liu et al.) separates retrieving evidence from consuming it. Most search agents fold retrieved documents into context by fixed or rank-based policy, even though a document’s utility depends on the agent’s current reasoning state. Preview-guided evidence admission decides per document whether it earns a place in context.

What to watch

Three of this week’s papers (tool-call correspondence, harness cost, dynamic concurrency) measured production harnesses by name, and all three found the harness in the loop as a cause. Expect vendors to be asked for hop-by-hop test evidence and for rerun-calibrated benchmark deltas. On the data side, Pinterest’s grade-and-tier model is a concrete template for what “governed” means when the consumer is an agent. The open question is whether a text-to-SQL stack like ARCS can be made to resolve ambiguity against a metrics catalog rather than a raw schema.

Mirror note: the feed’s status check reported mirror mode with a last sync of 2026-09-01 and 59,010 minutes of staleness, yet items dated through 2026-10-12 were present. The staleness reading looks like a reporting artifact. Coverage here spans 2026-10-05 through 2026-10-12.