Agent evals moved off task success: atomicity, supersession, and transfer

A knowledge-graph project memory answered supersession questions at 0.98 to 1.00 recall. The production vector-memory tool it was benchmarked against answered between 6% and 27% of the same questions. Same corpus, 835 typed records, and the whole gap sits in one class of query that top-k similarity cannot express: what replaced this decision, is this set complete, what is not true in this project.

That result is from MOOSEDev, which stores architectural decisions, lessons, constraints, and rationales as typed records with lifecycle status, provenance, and supersession links, exposes them to agents over MCP, and queries them symbolically instead of through an embedding index. Read past the headline number to what came out equal: relevance recall and token cost were largely equivalent between the two systems. The paper is therefore not a claim that graphs retrieve better in general. Its narrower claim is the useful one, that the questions an agent asks about its own project history are set-shaped, negation-shaped, and temporal, and dense retrieval is the wrong instrument for all three, because nothing in an embedding encodes “and nothing else.” The query engine is proprietary, so the reusable contribution is the record schema and the question taxonomy rather than the ranker. If you maintain a CLAUDE.md or an ADR directory that agents read, there is a cheap experiment sitting right there: ask your current memory layer which of its own entries have been superseded, and look hard at what comes back.

The most operationally direct thing in the window takes the same suspicion of headline metrics into stateful execution. LegacyWorld evaluates computer-use agents on 28 Windows GUI workflows drawn from legacy enterprise and healthcare systems, each specified with an initial state, a goal state, and a task-specific validator. The framing came out of building the framework, where domain experts kept raising the same objection: a run that fails partway through a stateful workflow can still leave persistent invalid changes in a business or patient record. The run failed and the damage is committed. So the metric is atomicity. A run should either complete the intended workflow correctly, or fail without unintended persistent side effects.

Across six hosted computer-use agents, useful completion, safe failure, and non-atomic side effects came out as three distinct operational profiles rather than three points on one quality axis. An agent that fails often but always fails clean is a different deployment risk from one that succeeds slightly more and occasionally corrupts state, and a scalar pass rate cannot tell you which one you bought. The paper argues that workflow capture, state validators, and atomicity-aware acceptance tests belong in the requirements rather than in the postmortem. Anyone who has written a compensating action for a half-finished agent run will recognize the shape: this is the transaction-boundary problem arriving inside an evaluation harness, and the hard design question it opens is where the rollback boundary lives when the environment is a GUI with no transaction log and no undo.

The evaluation-validity case got sharper in Don’t Claim Benchmark-Oriented Optimization Improves General Coding Capability, whose title is also its thesis. The authors built a Django-based benchmark suite, then evaluated foundation models alongside checkpoints post-trained on SWE-bench trajectories. Benchmark rankings frequently failed to generalize. The post-trained checkpoints showed little cross-task transfer, and SWE-bench optimization produced limited or no gains on the Django tasks or on LiveCodeBench. Fine-tuning on individual Django modalities failed to transfer either, which is the control that matters, since it rules out the comfortable reading that Django is simply harder.

Their prescription is differentiated evaluation: holistic assessment for frontier models, multi-task suites for research claims, human-in-the-loop studies for narrow applications, plus a capability taxonomy and sustained benchmark maintenance instead of one-off releases. The maintenance ask will get less attention than it deserves and it is the one that compounds. A benchmark that is optimized against and never revised stops being a measurement and becomes a target, and the community’s revealed preference so far is to publish a new suite rather than maintain an old one, because the new suite is a paper and the maintenance is not.

The system-level framing behind all three got a book-length treatment this morning, with a conflict to flag first: Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model went up on arXiv today and is written by this site’s author, so weigh the summary accordingly. It argues that coding agents are evaluated as models and deployed as systems, and that their reliability depends on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. The synthesis draws on 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 case records from operated systems, and produces a versioned catalog of 206 reliability records: 193 gated practices, 56 of them developed in depth, plus 13 research leads.

Two of its claims are load-bearing and both are testable. Many apparent model failures originate elsewhere in the system. And improvements at one layer often fail to propagate to end-to-end outcomes, which is the same transfer failure the Django study measured, observed at the training layer instead of the operations layer. The dependency-chain framing follows: weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions, so a clean number at the top of the stack is worth what the layer you did not inspect is worth. Stated limits, which are worth repeating: the review is structured rather than exhaustive, evidence strength varies by topic, results depend on workload and configuration, and the methods record which search lanes went unexecuted.

Two papers in the same window attack the legibility of agent behavior directly. ATLAS recovers finite-state models from agent trajectories by combining trace abstraction with automata learning, on the premise that raw traces and task-success rates say little about strategy while a learned automaton exposes recurring behaviors, decision points, successful completion paths, and failure loops as named structures you can query. The proof of concept runs on an LLM penetration-testing agent across 12 vulnerable machines, and the authors also demonstrate symbolic knowledge transfer from a frontier model down to a compact one by way of the learned model. Failure-loop detection is the part worth stealing: if you operate a fleet, “this run entered a known unproductive cycle” beats watching a token count climb.

Second Thought chases efficiency rather than reliability, off an observation about the ReAct loop: while the agent serializes an action and waits for the environment, its reasoning is frozen. The framework forks four auxiliary reasoning branches the instant each Thought phase ends, decodes them concurrently with the main loop, and merges them back when the observation arrives. Across three agentic benchmarks and three reasoning models, average turn count dropped in all nine model-benchmark pairs, and main-thread decoding fell in six of them by up to 43%, roughly 20% on average where it applied. Pass@1 showed no significant change in seven of nine pairs, and the two significant differences were +12.4 and +10.2 points. Against a compute-matched control that spends the same budget on the main thread’s own reasoning, it reached strictly higher Pass@1 with 1.3 to 3.2 times less sequential decoding. Training-free, which makes it a harness change rather than a model change.

From the practitioner side, a thread on running Claude Code unattended walked four options and landed somewhere that rhymes with the research. Plain cron plus claude -p works for one or two tasks, then you find yourself hand-building scheduling, secret handling, and run history, and it dies when the laptop sleeps. The official GitHub Action fits event triggers like a PR opening, less so “every morning at 8, go research this and message me,” which is most of what people want. Workflow tools like n8n push you into expressing agent work as deterministic nodes and fight you the moment the task is fuzzy. The author settled on running the agent as a GitHub repo with Actions as the runtime, so cron, secrets, and logs come from the platform.

The reason they trusted it is the line that matters: every run is a workflow log and every change is a commit, so when it does something dumb you go read exactly what happened instead of asking the agent what it did. That is the requirement ATLAS formalizes with automata and the observability layer the monograph puts in its dependency chain, reached from the opposite direction by someone who wanted a morning brief.

Four independent groups landed in about a day on measuring something other than task success: side-effect containment, cross-task transfer, memory query shape, behavioral structure. The open question is which of these becomes a gate rather than a paper. Atomicity has the clearest path, since a state validator is a thing you can write this afternoon and fail a build on. Watch for whoever ships one first.