Agents Get Graded on Process, Not Just Pass/Fail

A Claude Sonnet 4.6 judge read 44 Terminal-bench-2 trajectories and flagged the spans where the agent stated a problem in its own reasoning and then acted against it. The trajectories it flagged failed 94% of the time. The ones it left alone failed 46%. That 47-point gap is the sharpest result in Strained Coherence, Pandya, Zhang, and Lyu’s study of what they call a pre-failure signal, and the timing is the part worth sitting with: the first flag lands at a median of 83 to 84% of elapsed trajectory time. The agent narrates the tension late, optimizes the proxy anyway, and the run is already most of the way to a wall by the time the contradiction is visible. The detector emits span-level output, the quoted acknowledgment next to the quoted action and a typed conflict, so you can see exactly what the agent saw and ignored. That overlap with verbalized reward hacking is the uncomfortable read here.

It was a week of evals and instrumentation that stopped caring whether the agent finished and started measuring how it got there.

Benchmarks stopped scoring the final answer

SWE-bench trained the field to ask one binary question: resolved or not. Three benchmarks this week broke that question into parts. SWE-Explore isolates repository exploration, handing an agent a repo and an issue and asking for a ranked list of relevant code regions under a fixed line budget. The ground truth is derived from independent agent trajectories that actually solved each issue, distilled down to the code regions their solution paths consulted: 848 issues, 10 languages, 203 repositories. The finding that matters for anyone building a retrieval layer: file-level localization is already strong across modern methods, so it no longer separates anyone. Line-level coverage and efficient ranking under budget are where state-of-the-art explorers pull apart, and agentic explorers sit in a clear tier above classical retrieval.

SmellBench goes after the thing functional-correctness benchmarks never see. Code agents pass the tests and still leave bloated, disorganized code behind; SmellBench scores agents on refactoring tasks by long-term maintainability rather than whether the diff was green. The two benchmarks rhyme: both are bets that the interesting capability gaps now live in the parts of the trajectory that pass/fail scoring averages away.

Handoff Debt names a cost that single-agent benchmarks structurally cannot measure. Real work gets interrupted, reassigned, and resumed from a partial state someone else left. Dipesh KC and Anjila Budathoki interrupt an agent at deterministic handoff points, freeze the repo, and hand it to a successor under four views: repository state only, raw trace, summary notes, and structured notes. Across 75 source tasks, 181 handoff points, and 724 takeover runs per model, a context-bearing handoff cuts median agent events by 20 to 59% and prompt tokens by 42 to 63% against a repository-only takeover. Solved-rate effects are smaller and model-dependent; the efficiency gains are consistent. If you run agent fleets where one agent picks up another’s branch, this is the failure mode you have been eating without a number attached to it, and the number says the notes you leave behind are worth roughly half the successor’s token budget.

Memory systems are losing to the agent that manages its own files

The strongest memory result this week is a negative one. Cross-Scenario Generality of Agentic Memory Systems revisits eight published memory systems plus a plain agentic harness across five scenarios, from single-turn QA to long-horizon agentic tasks. The harness that self-manages flat text-file storage through tool calls takes the best cross-task ranking. Chen and colleagues read that as the load-bearing finding, not an aside: memory performance hinges on giving the agent active control over storage and retrieval, not on a clever store sitting behind a fixed pipeline. They package the insight as AutoMEM, but the lesson generalizes past their system. Most of the elaborate memory architectures generalize worse than letting the agent write and grep its own notes.

Memory is Reconstructed, Not Retrieved attacks the same static-pipeline assumption from the modeling side. MRAgent represents memory as a Cue-Tag-Content graph and folds LLM reasoning directly into memory access, iteratively exploring and pruning retrieval paths against evidence found mid-inference rather than running a single retrieve-then-reason pass. On LoCoMo and LongMemEval it reports up to 23% over strong baselines while cutting token and runtime cost, which is the rare memory paper claiming a win on quality and budget at once. Temporal Order Matters for Agentic Memory adds the orthogonal complaint that most memory stores organize by topical similarity and discard sequence; SegTreeMem uses a segment-tree structure so an agent can reason over when events happened, not just what they resembled.

Vendors are not waiting for the literature to settle. Weaviate moved Engram to general availability, a managed memory and context service pitched as the durable store agents orchestrate workflows against. The research consensus drifting toward agent-controlled flat files and the product market shipping managed memory layers is a tension worth tracking, because they cannot both be the right default.

One paper this week refuses the whole prompt-space framing. Scaling Self-Evolving Agents via Parametric Memory argues that summary-and-retrieval memory lets an agent look up what it has seen but never learn from it, since the policy stays frozen and anything dropped from context is gone for good. Their TMEM absorbs distilled supervision into fast LoRA weights mid-episode, so experience changes future behavior rather than just sitting in a prompt, and the extraction policy that decides what to learn becomes directly trainable by RL. It outperforms summary- and retrieval-based baselines across model scales on LoCoMo, LongMemEval-S, and CL-Bench. Pair it with Socratic-SWE, which mines an agent’s own solving traces into structured skills that summarize recurring failures and effective repairs, then uses those skills to generate targeted training tasks. Three iterations of that closed loop reach 50.40% on SWE-bench Verified, beating self-evolving baselines at equal compute. Both treat the trajectory not as a thing to score but as a substrate to learn from, which is the same instinct AutoMEM and MRAgent are circling from the storage side.

Multi-agent orchestration gets an accounting

For two years the multi-agent pitch has been decomposition: split the task, isolate context, run in parallel. When Parallelism Pays Off finally puts the bill on the table. Yang and colleagues formalize orchestration as a graph-partitioning problem where decomposition shortens the critical path but every cross-agent dependency demands costly context transfer, and sometimes the transfer eats the gain. Their Co-Coder builds dependency graphs from static analysis, isolates hub files, partitions by community detection, and schedules with dependency awareness. Across 28 real tasks on DevEval and CodeProjectEval it beats sequential, file-based parallel, and Claude Code with Agent Teams: up to 14% higher pass rate, up to 2.10x wall-clock speedup, up to 35% lower API cost, with the biggest wins on the most dependency-dense projects. The principle underneath is the useful part. Parallelism pays when partitions are cohesive and dependencies are sparse, and you can compute that property before you spawn anything.

The coordination tax shows up in two more papers. PerspectiveGap benchmarks a narrow, real skill: can a model write the orchestration prompt that tells each sub-agent precisely what it needs to know and nothing it doesn’t? 110 scenarios, and current models struggle to scope the handoff. Channel Fracture reports a concrete architectural bug in scheduled cross-agent memory injection, where one agent writing into another’s memory through a hierarchical team channel breaks in ways the system never surfaces. Both land near the same point as Strained Coherence and Handoff Debt: the expensive failures in agent systems are increasingly about what passed between steps, not what happened inside any one of them. The qualitative study How Early Adopters Conceptualize Transparency catches builders naming the Catch-22 directly, wanting visibility into inter-agent coordination while the orchestration layer is exactly where observability is thinnest.

And the adoption curve keeps bending

Whatever the open problems, usage is not waiting. Agentic Very Much! revisits coding-agent adoption on GitHub projects created after the authors’ earlier study and finds it more than twice as high, with a higher share of AI-assisted commits per project, and the authors note strong signs they are undercounting. Tooling is hardening around that reality: GitHub shipped the Copilot desktop app, an agent-native client built for directing several agents in parallel rather than bolting agents onto an editor designed for one human typing.

The throughline across all sixteen items: the field is building the instruments to see inside agent runs at the same moment it is handing those runs more autonomy and more of each other’s output. Watch whether the process metrics from SWE-Explore and the failure-prediction signal from Strained Coherence make it into anyone’s CI before the next adoption doubling, or whether instrumentation stays a research artifact while production keeps scoring on green tests alone.