Frontier models localize agent runtime faults 22% of the time

DeepSeek-v4-pro correctly names the type of an injected runtime fault in an agent execution trace 24.8% of the time. Local detectors up to 14B parameters land between 13.6% and 19.2%. Ask any of them to name the fault type and point at where it happened, and the ceiling across every model tested is 22%. Those numbers are the headline result of AGENTCHAOSBENCH, posted yesterday by Chenkai Zhang and colleagues, and they describe a capability that most production agent stacks already assume they have.

The construction is worth reading closely, because it is the part that transfers. Five heterogeneous applications coordinate agents over the Agent-to-Agent protocol and reach tools through MCP. Into that the authors inject ten operational fault types at four boundaries: unavailable or slow tools, corrupted or oversized tool responses, delayed, looped, or misrouted delegations, and bypassed guardrails. The dataset is 275 sanitized traces, 250 faulty and 25 no-fault controls, and every faulty trace is paired with a clean execution of the same input so a diagnosis method can be given a reference. Guardrail bypass is the case that stays close to unsolved: it is defined relative to what should have happened, and a single trace does not contain that. Handing the model the aligned no-fault run helps some relative faults and still does not crack it. If your incident review process for agent runs is a human reading one trace, this is the benchmark that says how far that gets you.

A companion result posted this morning argues the expensive approach was the wrong one. AFANet, from Ting-Wei Li and coauthors at Illinois, treats failure attribution as a graph problem instead of a reading-comprehension problem: model the trajectory as step-level semantic signals plus agent-level relationships, run a lightweight GNN over it, and skip long-context LLM inference entirely. It matches or beats LLM baselines including models fine-tuned on in-domain data, holds up across GNN architectures, and improves further with cheap test-time adaptation on the out-of-distribution benchmark, at near-zero inference cost. The finding underneath is that the signal identifying a bad agent lives in the shape of the interaction rather than the prose inside it, which is a good reason to stop paying frontier-model prices to reread your own logs.

Both papers take the trace as given. Agent-native telemetry attacks the format instead. Jun He and Deying Yu start from the observation that operational telemetry is serialized for human reading, so an agent consuming it spends context parsing lexical syntax rather than reasoning about state changes, with no cryptographic claim on provenance or collection completeness. Their Agent Telemetry Protocol structures operational facts into four primitives, Transitions, Observations, Relations, and State Checkpoints, under content-addressed schemas, and quarantines uncurated text behind digest-verified opaque references. Producers sign and hash-chain batches for atomic append. On AIOpsLab and the OpenTelemetry Astronomy Shop, ATP cuts raw wire payload and modeled cloud query scan cost by 96.4% against OpenTelemetry JSON, LLM context tokens by 88.8%, and query operations by 66.2%. It detects all 500 tested adversarial storage mutations and records zero successful prompt injections across 50 adversarial trials per configuration. That last number follows directly from the opaque-reference design: text that an attacker controls never enters the agent’s reasoning path as text.

The same architectural instinct shipped as a runtime today. DeepSeek released a developer preview of DeepSeek Harness (dsh), covered by InfoQ, an open-source execution runtime built as a micro-kernel with modular plugins for each functional unit and an append-only event log for execution activity. The event log is the interesting commitment. Append-only execution history is the substrate every result above depends on, and a harness that makes it a first-class kernel concern rather than an observability add-on is starting from the right place. InfoQ is right that adoption turns on plugin ecosystem stability and API maintenance, which is where most micro-kernel bets go to die.

While the diagnosis side gets sharper, the measurement side keeps finding that agent reliability is thinner than the leaderboards suggest. A Jagged Frontier, from a Colorado State, UIUC, CMU, and Microsoft group, rewrites the codebase surrounding a SWE-bench issue into semantically equivalent form using control-flow rewrites, dead-code injection, and identifier renaming, then re-runs the agent. Two scaffolds, mini-SWE agent and OpenCode, each backed by Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, or Qwen 3.6-27B, across SWE-bench Verified and Pro, with multiple runs per variant so the perturbation effect separates from the agent’s own stochasticity. Mean resolve rate drops up to 6.7 points, statistically significant in 6 of 16 configurations. The result that should change how people write robustness claims is that no model ranking survives a change of scaffold: Qwen is among the most robust under mini-SWE agent on Verified and the most brittle under OpenCode. The simpler scaffold is the more robust one. Robustness is a property of the model-and-harness pair, not of the model, and a leaderboard column cannot represent it.

On the Fragility of Self-Improving Agents, from Qinyuan Ye and colleagues at Salesforce, applies the same discipline to memory-based agents that learn from a task stream. Re-evaluating two published methods across multiple runs and randomly shuffled task orders, they find that evaluation noise in complex multi-step environments is already high and that a self-improving loop stacked on top amplifies it, and that improvement depends heavily on task order, because the default orderings in prior work impose an implicit curriculum that functions as a hidden prerequisite. Their diagnosis is task and environment underspecification: inspecting the memory banks by hand, they add rubrics and environment feedback into memory construction, which partially closes the degradation and leaves a gap they cannot yet account for. A single-run, fixed-order improvement curve is not evidence of learning.

The constructive item of the day makes an agent write down what it knows before acting. Spec-Driven Test Generation, from Michele Tufano’s group at Google, instructs a test-generation agent to first reason about and explicitly document pre-conditions, post-conditions, and undefined behavior, then generate tests against that semi-formal intermediate. On production bugs from Google, that scaffold delivers 9.8 percentage points more bug detection (p = 0.0352) and 2.5 points more branch coverage (p = 0.0034) than a direct-prompting baseline. Under LLM-as-a-judge, the spec-driven suites beat the baseline in 77.8% of cases and human-authored tests in 56.7%. Forcing an explicit contract before generation is cheap, inspectable, and the artifact it produces is exactly the reference that a fault-localization method wants later.

Watch whether the telemetry side converges. AGENTCHAOSBENCH shows diagnosis is hard from a raw trace, AFANet shows structure carries most of the signal, and ATP proposes emitting that structure at the source. If those three lines meet, the practical shape is an append-only, hash-chained event log emitted in an agent-readable schema and diagnosed by something small and cheap, with the frontier model reserved for the cases the graph cannot resolve. The open question is whether anyone standardizes the schema before every harness ships its own.