Agent robustness tracks the harness, not the backbone model

Inject a truncated response or a corrupted tool-call field into the HTTP layer between an agent and its LLM provider, and pass@1 falls by as much as 50 percentage points. That number comes from AgentChaos, a runtime fault-injection framework that intercepts LLM API responses at the shared HTTP interface, so it needs no source modification to the agent under test. The size of the drop is less interesting than its stability. Across 65 fault configurations spanning several agent systems, benchmarks, and backbone LLMs, the ranking of systems by robustness holds as you swap the backbone. Which model you run decides how well the agent does the task; how the system handles a malformed response decides whether it survives a bad afternoon at your provider.

The taxonomy is worth stealing even if you never run the tool. AgentChaos separates crash faults, omission faults, and value faults, and applies each to two distinct fields: message content and tool calls. A corrupted tool-call argument is a different failure than a truncated assistant message, and the two propagate through downstream agents in different ways. The framework also verifies that each injected fault actually fired before counting the task, which filters out runs where the fault landed on a code path the agent never took. Skip that check and you dilute your own results toward optimism.

The second half of that paper is the part production teams should read twice. Existing fault-diagnosis methods identify the fault type with under 53% accuracy and the fault step with under 56%. An agent that degrades because your provider returned a half-written JSON blob will, in most current setups, not be able to tell you which call broke or what kind of break it was. Retries paper over the symptom without touching the diagnosis.

That is the gap Long-Horizon Agent Trajectory Attribution goes after directly. The authors define trajectory attribution as a task in its own right, then build a benchmark for it: heterogeneous trajectories from AgentDojo and from the Stage and Canary settings of Agent3Sigma, normalized under one component schema, yielding more than 1,300 annotated traces that cover task-aligned actions, unsafe actions, and safety refusals. Two evaluation tasks come out of it, localizing the primary attribution component and recovering the full attribution chain, with baselines built on incremental trajectory contribution and component-level leave-one-out perturbation. Baseline performance varies substantially between local attribution and long-range attribution, which is the honest result: figuring out that step 3 of 5 caused the failure is a different problem than figuring out that a memory write forty steps back did. They also ship the annotation procedure as a reusable skill so trajectories from new models can be standardized into the same frame, which is a more useful contribution than another leaderboard.

Attribution matters because harness optimization is now a real engineering loop and it needs a signal to optimize against. ADIAS reformulates automated agent design around that. Existing automated-design methods are candidate-centric: each round produces a new candidate agent, and what got repaired stays implicit in the candidate’s history. ADIAS carries an explicit persistent issue state instead, with stable issue identities, lifecycle status, supporting evidence, and a record of which interventions were tried against each issue and what happened. Optimization then proposes repair targets and revision directions from that state rather than re-deriving them from scratch each round. It beats the strongest baseline by 25.2% on average across five interactive benchmarks and holds across four backbone models; ablating the persistent issue state or reverting to candidate-centric revision costs up to 40.7%. The mechanism is close to what a careful human maintainer does with an issue tracker, which is either reassuring or slightly deflating depending on your priors.

A practitioner on r/LLMDevs arrived at the same conclusion from the other end. Escapement is a repository-native harness built around a Specify → Route → Execute → Verify → Persist loop, currently 35 skills and 143 passing unit tests, but the interesting discipline is the ablation practice. The author removes harness components and reruns the eval corpus. One removal dropped routing from 22/22 to 13/22. Another produced no measurable change at all, and the post treats that as an equally useful result: either the component does nothing, or the routing benchmark does not measure what the component was built to improve. Most harness projects never run that experiment, which is why most harness projects cannot tell you which of their rules is load-bearing. The stated limitation is worth copying too. Real-use validation is on Claude Code, Codex packaging exists, and the author declines to call it equivalent support until the evidence is equivalent.

Two evaluation papers from the same window push on a related idea, that the consumer of an artifact should define its quality metric. SONAR evaluates source-code summaries without reference summaries by regenerating code from the summary and using the reconstruction as the quality signal, scored on correctness, abstraction, conciseness, and fluency. Tested against LLM performance on four downstream SE tasks, correctness and abstraction correlate with downstream performance at up to 14 times the best baseline, while conciseness and fluency, the two dimensions human developers reliably prize, come out mostly insignificant to an LLM consumer. If your pipeline generates summaries for agents to read, you have been optimizing at least one dimension for the wrong reader.

FinRank makes the same move in retrieval, where a numerically correct answer grounded in the wrong disclosure is still wrong. The benchmark is 1,185 hand-authored question-answer records over 10-K and 10-Q filings from 22 companies, each with gold supporting passages and hard negatives curated from confusable passages within a filing, across reporting periods, and across comparable firms. The baselines are sobering: a 7B instruction-tuned embedder reaches 44.8% Recall@10 on the pooled corpus, sub-billion encoders gain at most 3.5 points over BM25, and one finance-adapted embedder trails BM25 by 9.7 points. Swapping random negatives for the curated ones costs 13.0 to 20.5 percentage points of pairwise accuracy, which tells you roughly how much of the reported progress in domain retrieval is an artifact of easy negatives.

What to watch: whether anyone connects these. AgentChaos generates controlled failures, trajectory attribution localizes them, ADIAS-style persistent issue state consumes the localization and proposes repairs, and the ablation discipline from Escapement checks that the repair earned its place. That is a closed loop and every piece of it now exists separately. The open question is whether the diagnosis step gets good enough to drive the rest, because 53% fault-type accuracy is not a foundation you build an automated repair loop on.