Keeping failure lessons out of context beats runtime intervention by 37%

Putting a playbook of failure lessons into an agent’s context makes the agent worse. That is the controlled result in Sentry, from Changxiu Ji, Amy Lu, Qizheng Zhang and Kunle Olukotun, posted to arXiv today. Their argument is that failure knowledge is conditional: a lesson about a malformed tool call only helps while that failure is happening, and sitting in context the rest of the time it misfires. Sentry runs as a layer beside the agent. It detects a failure, retrieves matching lessons from an external playbook, checks whether the agent actually recovered without reading task rewards, and stores a new lesson only when it did. The playbook never enters the agent’s prompt. Across several agentic benchmarks it beats the strongest runtime-intervention baseline on every one, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both run. Lessons transfer to held-out tasks, and the ablation is the part to remember: exposing the full playbook lowers performance even when the relevant lessons are still available on demand. If you maintain a growing CLAUDE.md or an evolving skills file, this is evidence that the file’s size is a cost in its own right, separate from whether its contents are true.

The second paper of the day is a position piece with a sharper claim than most. Tracking State Footprints: How Agents Can Transact, from Oto Mraz, Rares Serban, Kyriakos Psarakis, Burcu Kulahcioglu Ozkan and Asterios Katsifodimos, argues that multi-agent orchestrators have recreated the pre-transaction database era. Agents run plans in parallel, write code, change infrastructure and call services, and the orchestrator tracks none of the state they read or write, so lost updates and stale reads show up even on simple coding tasks. The proposal is to describe each agent by its state footprint, the reads and writes across its own context, the orchestrator’s state and external systems, and then borrow isolation guarantees from databases. The differences from classic transactions are real: agents do not read from a fixed schema or an isolated snapshot, and a failed agent cannot be replayed deterministically. The upside is that an agent can resolve a conflict semantically instead of aborting. Anyone who has merged six green branches from six parallel agents into one broken app will recognise the problem; the paper’s contribution is naming it as concurrency control and sketching transactional interfaces for external systems to join an agentic transaction.

Two coding-agent papers attack the same failure from opposite ends: the agent satisfies the examples it was shown and nothing more. GTDD, by Masahiro Kato, reformulates test-driven development so that a separate testing agent generates fresh inputs after each candidate implementation is fixed, working from a human-written behavioral contract and the history of earlier rounds. A trusted evaluator checks the inputs, hands reduced counterexamples back to the coder, and keeps them as regression tests. The finite-population analysis bounds false acceptance under adaptive candidate selection and shows that fresh random audits after commitment keep that rate controlled across rounds. In a paired experiment on a stateful key-value store, both policies that regenerated tests during development ended with lower mean failure rates than the policy whose tests were generated once by the same model, and letting the tester read the candidate’s source produced no detectable extra benefit. The limitation is in the abstract too: conditions requesting equal test counts could not isolate which feature of the policy carried the gain.

CONTRA, from Zheng Fang, Ge Li and colleagues at Peking University, goes after the earlier moment, when the agent resolves an underspecified requirement with its own assumption and says nothing. The method is training-free: generate candidate clarification questions, drop the ones unrelated to required behavior or already answered by the spec, then for each survivor synthesize programs under two plausible answers and check whether they behave differently on shared inputs. A question only earns the right to interrupt the developer if the answer changes execution. On ClarifyCodeBench, CONTRA posts the highest F1 with all four coding agents tested, beating the best baseline macro-average by 13.88 points, and with the same model and protocol it reaches higher clarification recall and F1 than Claude Code and OpenHands used as harnesses. The authors ship it as a Claude Code plugin that reviews implementation decisions at the end of an interaction and reports the unresolved choices.

On evals, Passing the Test You Trained On, by Zhuowen Liu, is the audit that should precede any detector purchase. The setup replays ground-truth tool calls from AgentDojo and tau-bench without an LLM, so tool outputs are benign by construction, labels injected outputs by differential replay, and runs fifteen prompt-injection detectors including Meta’s Prompt Guard 2 plus two task-aware LLM judges. Rankings do not transfer. The best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector catching 72% on AgentDojo catches 15% on tau-bench. False-positive rates on tool outputs, which range from none to over 90%, do transfer between the two agent benchmarks. The explanation is the form of the training data: the BIPIA leader trained on full BIPIA inputs, while the best detector on both agent benchmarks shares no data with any benchmark and trained on agent-style inputs. Evaluate on your own tool outputs, report detection at a fixed low false-positive rate, and read what the detector was trained on.

Verify Claims, Not Scores, by Ali Atiah Alzahrani, makes the same move at the level of an agent’s architecture. Replacing one component of a plan-act-check-refine agent and watching the aggregate score tells you nothing about whether improvement was attainable, which component lost the value, or what the verifier certifies. The audit instead records each conclusion with its evidence, one of four verdicts, and the boundary within which it holds, using oracle policies, one-at-a-time perfect-component replacement, and a test of whether the verifier’s score bounds the quantity it is read as bounding. On a portfolio-allocation agent in a synthetic market with known regimes, the scenario generator retained 29% of the true bull-to-crisis mean gap where a regime-conditional bootstrap retained 101%, better local fidelity did not improve decisions, and the runtime verifier could be bypassed with no visible change in outcomes. The findings are specific to that agent; the protocol is the contribution.

For the semantic-governance charter, FLINT, from Arijit Sehanobish, Kristen Howell and colleagues, is a production text-to-SQL system for financial databases where concepts live as opaque integer keys, every simple query needs several joins, and general-purpose systems that look strong on Spider and BIRD fall below 50%. The three pieces are each a form of semantic grounding: a lookup agent that resolves natural-language concepts to question-specific reference-table constraints, embedding retrieval over a compact expert-authored bank of query templates, and schema linking that prunes the schema by walking foreign-key chains rather than matching names. Evaluated on 359 questions over production schemas, FLINT beats state-of-the-art baselines on the same LLM and is deployed in a live data-retrieval service. It is a concrete answer to the charter question of how the system knows what the business means when the business stored it as an ID.

Watch for Sentry’s playbook result to be replicated on coding benchmarks specifically, and for the state-footprint framing to pick up a reference implementation; the first orchestrator that ships per-agent read/write tracking will have a lot of merge-conflict data to publish.

Feed note: the item mirror reported its last sync as 2026-09-01 while still returning items dated 2026-10-05; everything above was selected from the items present in the window.