Most tool-agent failures end at a harness limit, not a wrong answer

Of 20,408 exactly graded tool-agent episodes in ToolBottleneckBench, 3,725 failed, and 91.5% of those failures ended at a harness limit: a completion cap, an episode budget, a call ceiling, an infeasibility stop. Only 5.5% ended in a wrong conclusion. That split frames the last day or two. Across seven items, the thing being measured, repaired, or governed is rarely the model’s reasoning; it is the contract wrapped around it.

ToolBottleneckBench (Xu, Dong, Onizuka and Xiao, Osaka and Nagoya, 15 Oct) holds synthetic enterprise tasks fixed and varies one pressure at a time: the registered catalog (32 to 2,048 tools), the visible menu (8 to 128), the certified-workflow depth, and the encoding of returned records. Of the three pressures that act on the model, only the visible menu carries a large penalty, and that penalty tracks truncation at the completion cap with a Pearson correlation of -0.95. Raising the cap sixteenfold shrinks the pooled menu penalty in the six truncation-heavy configurations from -43.7 to -2.7 points. Workflow depth fails in a different way: verified success runs 91.2%, 87.5% and 85.0% at 2, 6 and 14 certified calls, then 0.0% at 30, because the episode budget runs out before the chain of handoffs completes. The repairs that helped changed which tools reach the model (a connector-first router lifts required-tool availability at 2,048 tools from 19.4% to 77.8%) or who issues the calls, while prompts that coach the model gained 1.7 and 0.1 points. The recommendation is modest and should become standard practice: report pressure effects together with the harness budgets under which they were measured.

HPCAgent-Bench (Budanaz and colleagues, ETH Zurich, 14 Oct) reaches the same place from performance engineering. It has 689 kernels in three tracks, each specified once in NumPy with derived C and Fortran references, judged by an external harness on hidden fuzzed inputs so agents cannot learn the test. Agents reach up to 4.2x speedup over automatic parallelizers, and 93 to 99% of device-agnostic answers stay correct when moved to another machine. The sentence that matters to anyone paying per token: across 40 matched setups, adding a tool or skill changed speedup significantly only twice, but changed token cost by up to 3.0x. They define intervention efficacy as a paired-run metric over speedup, success and cost, and they publish the token weights, because whether a saving counts as significant depends entirely on how cached input is weighed against output.

Two coding-agent papers go after a failure that lives inside the context window rather than at its edges. OPRD (Luo and colleagues, Tsinghua and Shanghai AI Lab, 15 Oct) starts from a diagnostic on 200 SWE-rebench V2 tasks, where Qwen3.5-4B triggers 446 error events across four categories (tool-feedback handling, scope and source, scenario, post-edit control), and most are cases where the relevant information was already in context and the agent did not act on it. The fix is a training recipe rather than a prompt. A teacher re-evaluates the student’s own failed prefixes with reminders attached, a global one built from final-test diagnostics and turn-local ones that apply only to the next decision after an observation, and the student learns to match that distribution without ever seeing a reminder at inference. Successful rollouts become self-imitation targets. The 4B student gains 11.3 points averaged over SWE-bench Verified, Pro and Multilingual, and error events fall from 446 to 222. At 9B the recipe reaches a 55.9 average, up 11.5 over the open checkpoint.

Felix Woolford’s transparent-compact (Hokkaido, 14 Oct) attacks the same loss from the compaction side, for the lightweight models people run locally. The method is deterministic and almost dull: never let a model or an algorithm rewrite user or assistant prose, drop tool output and reasoning, flag exactly where something was removed, and give the agent a lookup tool to recover it. Against Pi’s default model-written summary on planted facts it recalls 97.8% to 75.3% on Qwen3.8-27B, and the gap roughly doubles on the two weaker models (95.1 to 55.0 on Luna, 93.3 to 51.5 on Gemma). Against pi-vcc, a deterministic method without verbatim retention, commitments the user set mid-conversation survive 75.6% to 54.8% on Qwen, and confident violations drop from 41 to 56% of replies down to 15 to 37%. On the frontier model the summary wins the commitment test by five points, which the author reads as a strong model compensating for an ambiguous summary. The cited prior result, constraint violations rising from 0% to 30% after a compaction that dropped the policy, is the number to carry into any harness review.

Netflix’s Observability Ontology (Vijayanathan and Sanchez-Silva, 16 Oct) is the day’s practitioner piece on semantic governance, and it concerns operational rather than business data, which makes the mechanism easier to see. The problem statement is the semantic-layer one: Atlas, Radar, incident.io, PagerDuty and Spinnaker each hold a coherent model and none agree on what an application is, so a four-hour investigation spends most of its time reconstructing relationships the systems already knew. The answer is RDF, OWL, SHACL and SPARQL over a homegrown quad store, twelve namespaces, no generic Service type, and every ops:Application validated against a 49,000-entry catalog before it may exist. The part worth copying is how they get determinism out of LLM enrichers: the model may write any prose, but its output must fit a closed set of predicates, so a stochastic step yields a typed triple or nothing. Dozens of enrichers run sequentially, each reading what the last wrote, and the AutoRCA agents on top share one vocabulary, the graph’s. Correlation chains carry their confidence and the records that produced them, anything under 0.6 is discarded, and the ranked hypotheses are, by the authors’ own account, sometimes wrong. The next step is integration with Upper, the domain-modeling language behind Netflix’s Unified Data Architecture, so incidents and deployments join the business domains in one semantic layer that engineers and agents query the same way.

RCM (Nguyen and colleagues, Monash, 16 Oct) addresses the accountability gap in decentralized orchestration: when agents delegate recursively, who owns the failure? Each delegation is coupled to an explicit commitment, so an agent keeps its upstream responsibility while locally orchestrating downstream ones, and a responsibility-oriented protocol lets agents establish, manage and terminate commitments without a controller. The commitment graph evolves as agents act in parallel, and coordination toward the shared mission emerges from it. Reported gains run up to 18.7 points in task performance and 10.2 in a Responsibility score over centralized and decentralized baselines across reasoning, coding and executable environments, with better failure and harm detection and no task-specific training. Only the abstract is in the feed so far; the thing to check when the full text lands is whether Responsibility is scored independently of task success.

Librarian Agents (Singh, Zhang, Lee, Nair, Kumar and Finn, Stanford and CMU, 15 Oct) inverts retrieval. Instead of an answering agent searching a raw corpus per query, a builder agent reorganizes the corpus ahead of time into a hierarchical filesystem of synthetic files and reference links, trained with reinforcement learning to maximize the downstream answerer’s accuracy. The motivating observation is sharp: harnesses like Claude Code make agents good at navigating an existing filesystem but never train them to construct one for later use. On BrowseComp-Plus the built workspace beats the strongest retrieval and precomputation baselines by 33 points; on BRIGHT, zero-shot to unseen domains, the average gain is 31 points, and at ten questions per corpus the RL builder spends 33.8% fewer tokens per correct answer than an untrained one. The amortization argument holds only when queries repeat over a stable corpus, which describes enterprise knowledge bases and codebases far better than the open web.

What to watch: whether benchmark authors pick up the ToolBottleneckBench and HPCAgent-Bench habit of publishing harness budgets and token weights beside every score, and whether Netflix’s Upper integration yields the first public account of a single ontology serving both operational agents and business analytics.

Feed note: the code-intel mirror’s status endpoint reported its last sync as 2026-09-01, yet the feed carried items dated through 2026-10-18; the status timestamp appears stale rather than the data.