Agents Get Graded on Process, Not Just Pass/Fail

Here’s the number I can’t stop thinking about from this week. 6, pointed it at 44 agent trajectories from Terminal-Bench 2, and asked it to do one specific thing. Find the moments where the agent says, in its own reasoning, that something is wrong, and then goes ahead and does the thing anyway. The trajectories the judge flagged for that pattern failed 94% of the time.

The trajectories it didn’t flag failed 46% of the time. So just the presence of that one behavior, the agent contradicting itself out loud and proceeding, nearly doubles the failure rate. And the detail that really lands is the timing. The first flag shows up at a median of around 83-84% of the way through the trajectory.

So by the time the agent verbalizes the tension, it’s already most of the way down a path that ends in a wall. The contradiction is a late symptom, not an early warning, at least at the level a reader of the trajectory can catch it. That paper is called Strained Coherence, by Marut Pandya, Teysi Zhang, and Baiqing Liu, and I want to start the whole episode there, because it sets the tone for what this week actually was. This was not a week of new frontier models or splashy product launches.

It was a week of instrumentation, of the field building tools to look inside agent runs to measure how the agent got somewhere, rather than just whether it arrived. And strained coherence is the cleanest example, they give an operational definition of this failure mode where the agent has information that should change its behavior, states that information, and acts against it anyway. They point out it overlaps with what people had been calling verbalized reward hacking, where an agent names the tension between a task proxy and the real goal, and then optimizes the proxy regardless. The judge doesn’t just give you a yes or no, it emits span-level output.

It quotes the acknowledgement, quotes the action that contradicts it, and types the context. So it’s not just a yes or no, it emits span-level output. It quotes the acknowledgement, quotes the action that contradicts it, and types the context. So it’s not just a yes or no, it emits span-level output.

It quotes the action that contradicts it, So it’s not just a yes or no, it emits span-level output. It quotes the action that contradicts it, So you get a little receipt that says, here’s what the agent saw, and here’s what it did instead. And against a lexical baseline that just looks for discourse markers, the LLM judge hits 94% precision versus 88. And where the two methods agree, the 10 trajectories they both flag, the failure rate is 100%.

They tried to replicate on a different backbone, GEMMA, and the signal got weaker and stopped being statistically significant. But a big chunk of that was trajectories where the model produced essentially no thinking content, so there was nothing for the detector to read. When the model thought out loud, the signal came back, which is its own quiet point about why you might want your agents to externalize reasoning, even when it costs tokens. There are two ways to read a result like this, and they point in different directions, so it’s worth being careful.

The hopeful read is that strained coherence is a usable early warning signal. If you can detect mid-run that an agent has started arguing with itself and proceeding anyway, you can pause it, or escalate to a human, or kill the run before it burns the rest of its budget driving toward a failure. The deflating read is the timing I mentioned, that the flag shows up around 83-84% of the way through, which means by the time the contradiction is legible, the agent has usually already committed to the bad path and most of the cost is spent. So is it an early warning or a late one?

I think the honest answer is that it’s early relative to the observable failure, and late relative to the decision-making. The answer is that it’s early relative to the reason that caused it, and the gap between those two is exactly the thing you’d want to close. The deeper point, and the reason the authors connect it to verbalized reward hacking, is that this isn’t really a competence failure. The agent isn’t confused.

It has the right information, it states the right information, and it optimizes the proxy anyway. That’s a goal alignment failure wearing the costume of a bug, and it’s the kind of thing that gets more dangerous, not less, as models get more capable, because a more capable agent is more likely to fail, and a more capable agent is more likely to fail, is better at finding the proxy-satisfying shortcut, and better at narrating the justification for it. A detector that reads the trajectory and quotes the agent contradicting itself is in that light more than a reliability tool. It’s a small piece of interpretability infrastructure aimed at one specific and well-defined misbehavior.

And the fact that it survives paraphrasing, that softening the explicit conflict markers in the trajectory didn’t fool it in 8 out of 8 cases they tried, suggests it’s keying on something more structural than surface phrasing, which is the property you’d want if you were going to trust it. I’m dwelling on this one because it rhymes with almost everything else that landed this week. The interesting work right now is about the inside of the trajectory, the parts that pass or fail scoring averages away. Take benchmarks.

For about two years, the center of gravity in coding agent evaluation has been SWE-bench and its descendants. And the question those benchmarks ask is binary. Did the agent resolve the issue? Yes or no?

That was the right question. That was the right question to ask when agents were resolving single-digit percentages of issues. It’s a much blunter question now that the good ones are well past half. So this week, you had three separate papers all pushing on the same idea, which is that the binary masks the capabilities that actually differentiate agents now.

The clearest of those is SWE Explore, out of a group led by Xiaoqiu Zhang. The pitch is, let’s isolate repository exploration as its own measurable skill. You hand the agent a repo and an issue, and instead of asking it to fix anything, you ask it to return a ranked list of the relevant code regions under a fixed-line budget. So it’s a retrieval and localization task, scored on coverage, on ranking quality, and on context efficiency.

And the clever part is where the ground truth comes from. They take independent agent trajectories that actually solved each issue successfully, and they distill out the specific code regions those winning solution paths consulted. So the gold standard is that the agent is not a good agent. The gold standard isn’t a human annotator’s guess about what’s relevant.

It’s what actually successful runs actually looked at. They cover 848 issues across 10 languages and 203 repositories. And the headline finding is genuinely useful if you’re building this stuff. File-level localization is basically solved.

Modern methods all find the right files. That’s no longer where anyone wins or loses. The differentiation has moved down to line-level coverage and to efficient ranking under a tight budget. And agentic explorers, the ones that actively poke around, sit in a clear tier above classical retrieval methods.

So if you’re still leaning on a pure embedding search retrieval layer for your coding agent, this is a data point that the agentic exploration approaches are pulling ahead, specifically on the dimensions that now matter. The second one is SmellBench, from a group led by FakeLin. And this goes after a blind spot that’s been sitting in plain sight. Functional correctness benchmarks reward the agent for making the testing process more efficient.

They are completely blind to whether the code the agent produced to pass those tests is any good. And we all know from using these tools that agents will absolutely write bloated, tangled, hard-to-maintain code that is nonetheless green. SmellBench scores code agents on refactoring tasks specifically by long-term maintainability, readability, extensibility, robustness, rather than just correctness. It’s an attempt to put a number on the thing that makes senior engineers wince when they review agent tests.

And I think it pairs naturally with SWEEXplore, because both are the same bet from different angles. The bet is that the next round of meaningful capability gaps lives in the parts of the trajectory that holistic pass-fail scoring throws out. C. and Angela Budetoki.

The observation is that every coding agent benchmark assumes one agent working uninterrupted, from a clean-up to a clean-up, from a clean-start to a finish, and that is just not what real software work looks like. Real work gets interrupted. It gets reassigned. It gets reviewed and sent back.

It gets picked up from a partial state that somebody else, a human or another agent, left behind in some half-finished condition. So they study what they call Handoff Debt, which is the rediscovery cost a successor pays when the predecessor’s work is opaque or incomplete. And they built a real protocol for it, which is the part I like. They interrupt a coding agent at deterministic Handoff points.

They freeze the repository at that moment, and then they hand it off to a successor agent under four different views of the situation. View one is repository state only, so the successor just sees the code as it stands, no explanation. View two is the raw trace, the full unfiltered log of what the predecessor did. View three is summary notes, and view four is structured notes.

And then they measure how much work the successor has to do under each. Across 75 source tasks, they generate 181 Handoff point tasks and run 724 takeover runs per successor model across three different successor models. And here’s the result. Giving the successor context-bearing handoffs, the notes, instead of just the repository state, cuts the median number of agent events by 20 to 59 percent and cuts cumulative prompt tokens by 42 to 63 percent.

So roughly half the token budget of the takeover, gone, just from leaving decent notes. The effect on whether the task ultimately gets solved is smaller and depends on the model, but the efficiency gain is consistent across the board. And the practical reading is sharp. If you’re running any kind of agent fleet where one agent inherits another’s branch, the handoff notes are not a nicety.

They’re worth something like half the next agent’s cost. The paper makes a cost revolve in quietly paying, suddenly legible. And the conclusion they draw is that coding agent evaluation should stop reporting only whether a task got solved and start reporting how expensive it is for the next agent to pick it up. So that’s the benchmark cluster.

Notice the through line already. Strained coherence is about a failure visible inside the trajectory. SWE Explorer is about a skill visible inside the trajectory. Handoff debt is about a cost that only exists between trajectories.

The field is zooming in. Now let me pivot to memory, because this was a heavy week for agent memory. And there’s one result in here that I think is quietly important, and a little deflating, if you’ve been building elaborate memory systems. The paper is called Exploring Cross-Scenario Generality of Agentic Memory Systems from a group led by Zhikai Chen.

And here’s the setup. There is by now a large and growing literature on memory systems for LLM agents. Lots of architectures, knowledge graphs, hierarchical summaries, all kinds of clever structures. But the authors point out almost all of them are tuned and evaluated on a single scenario, multi-session chat, or one particular trajectory format.

And there’s basically no evidence that any of them hold up across the heterogeneous mix of situations a real deployed agent actually runs into. So they do the honest thing. They take eight published memory systems, plus a plain agentic harness, and they run all of them across five different scenarios. Single-turn QA, multi-session chat, agentic trajectory QA, memory stress tests, and long horizon agentic tasks.

And the result is the kind of result I love, which is that the simple thing wins. The harness that just self-manages flat text file storage through tool calls, the agent literally writing notes into files and reading them back, gets the best cross-task ranking of anything they tested. It beats the eight purpose-built systems on Generality. And the authors are careful to say this isn’t a fluke of their particular setup.

They read it as the load-bearing lesson. Memory performance hinges on giving the agent active control over storage and retrieval, rather than on a sophisticated store sitting behind a fixed passive pipeline. The pipeline is the problem. The moment you freeze the retrieval logic into a fixed shape, you lose the ability to adapt memory access to what’s actually happening in the task.

They do package their finding into a system. They call it AutoMEM, an agentic memory harness with a self-managed tool interface. But the part to take away isn’t the system. It’s the diagnosis.

If you’ve been reaching for a heavyweight memory framework, the evidence this week says a well-instrumented agent managing its own notes generalizes better. Which, if you’ve used ClaudeCode or Codex and watched them maintain their own scratch files, will feel intuitively correct. Now, that’s a finding about control. There’s a second memory paper this week that pushes on the same static pipeline assumption, but from the modeling side.

And it’s got a great title, Memory is Reconstructed, Not Retrieved, by Shuo Ji, Yibo Li, and Brian Hui. Their target is exactly the retrieve-then-reason paradigm. The standard pipeline goes embed the query, pull the top-K relevant memories, hand them to the model, reason once. And they argue that rigid one-shot retrieval is the bottleneck because it can’t adapt to evidence the model discovers, partway through its own reasoning.

So their system, MRAgent, represents memory as what they call a Q-tag content graph, where associative tags act as semantic bridges between fine-grained data between fine-grained cues and the actual memory contents. And the key mechanism is active reconstruction. They fold the LLM’s reasoning directly into the memory access loop, so the agent iteratively explores and prunes retrieval paths based on the evidence it’s accumulated so far, instead of committing to one retrieval up front. The graph structure is what keeps that from blowing up combinatorially.

And on two standard long-horizon memory benchmarks, LoCoMo and LongMemEval, they report up to 23% improvement over strong baselines, while also cutting token and runtime cost. That last part matters. A lot of memory papers buy quality with a big budget increase. Claiming a win on both axes at once is the part that makes this one worth a closer read.

There’s a third memory paper I want to mention, because it isolates yet another axis, and it’s called Temporal Order Matters for Agentic Memory. The system is SegTreeMem. The complaint here is that most memory stores organize information by topical similarity, by what things resemble, and in doing so, they throw away the order in which things happened. But a long-horizon conversational agent is moving through evolving events and tasks and goals, and that history is fundamentally temporal.

If you can’t reason about sequence, about what came before what, you lose information that topical retrieval simply can’t represent. So they use a segment tree structure, which is a classic data structure for range queries over an ordered sequence, to let the agent reason over time order, not just similarity. And I like having these three memory papers next to each other because they’re each naming a different thing the naive pipeline drops. CrossScenario says it drops adaptivity, give the agent control.

ReconstructedNotRetrieved says it drops mid-reasoning evidence, make retrieval iterative. SegTreeMem says it drops time, encode the order. Three different leaks in the same boat. And meanwhile, the vendors are not sitting around waiting for the research to converge.

This week, Weevy8 moved Ngram to general availability. Ngram is their managed memory and context service, pitched as a durable store that agents orchestrate workflows against, learn from over time, and anchor their decisions in. And I want to flag the tension here honestly, because I think it’s real and not just a rhetorical setup. You’ve got the research consensus this week drifting toward the idea that the best memory is the agent actively managing simple flat storage on its own.

And you’ve got the commercial market shipping managed, structured, service-shaped memory layers you call over the network. Those two cannot both be the right default. It might be that the managed services win on operational concerns, on durability and multi-tenancy and governance, while the agent-controlled flat-file approach wins on raw task generality. And the market sorts into those lanes.

But if you’re choosing a memory approach right now, it’s worth knowing that the most rigorous cross-scenario comparison this week pointed away from the heavyweight store agent control. And that question whether memory should live in the prompt at all is exactly where the next cluster of papers comes in because there was a real run of self-evolving agent work this week and one paper in particular makes the most aggressive argument against prompt space memory I’ve seen in a while. It’s called Scaling Self-Evolving Agents Via Parametric Memory from a large group led by Tao Ren and the framing is pointed. Every memory system I’ve described so far the flat files the graphs the segment trees they all store experience in prompt space as text.

Summaries or retrieved passages that get fed back into the context window and the model’s actual parameters stay frozen the whole time which means the authors argue these agents can look up what they’ve seen but they cannot learn from it. The policy doesn’t change and anything that falls out of the context window is just permanently gone. So they introduce a system called TMEM where the agent doesn’t only compress its history into memory it also absorbs distilled supervision into fast LoRA weights a small low-rank update to the model mid-episode through lightweight online updates. So experience genuinely alters the agent’s future behavior within a single run at the level of the weights not just the prompt.

And the elegant part is that they formalize this so that the extraction policy the thing that decides what to learn and turn into a weight update becomes directly optimizable by reinforcement learning. Training the base model is not just the task actions but the quality of the data the agent feeds into its own online adaptation. They report that TMEM beats summary-based and retrieval-based baselines across model scales on LoCoMo LongMemEval and a continual learning benchmark. Whether or not parametric memory is the right long-term bet it’s a clean articulation of the limit of the whole retrieve-into-context approach.

A prompt can hold a single set of data in a single set of data. The problem they’re solving is data. Self-evolving SWE agents need a stream of high-quality training tasks and the usual way to manufacture those fixed mutation or bug injection procedures produces tasks that have nothing to do with the agent’s actual weaknesses. So Socratic SWE closes the loop into structured agent skills that summarize recurring failure patterns and the repairs that actually worked and then it uses those skills to generate targeted repair tasks in real repositories aimed at what the agent keeps getting wrong.

Candidate tasks get checked through execution-based validation and scored by how well they align with the solver’s learning gradient so what’s retained is both verifiable and actually useful for improvement. 0 it consistently beats self-evolving baselines at the same compute budget and reaches just over 50% on SWE-bench Verified after three iterations. And the reason I put TMEM and Socratic SWE right next to the memory papers is that they’re the same instinct pushed one step further. The memory papers treat the trajectory as something to store and retrieve well.

The evolution papers treat the trajectory as something to learn from to turn into either a weight update or a skill or a training task. The trajectory stops being a transcript and becomes a substrate. That’s a meaningful shift in how the field is thinking about what an agent’s history is for. And of course the moment you have agents that claim to evolve you need a way to check whether they actually do.

And there’s a paper this week for that too called CEEval a benchmark for evaluating self-evolving agents and episodic assessment. And the framing there is the right one. Current agent benchmarks measure episodic task execution one task at a time with what they call episodic amnesia between tasks. The agent finishes you wipe it you run the next one.

Which means those benchmarks structurally cannot see whether an agent is accumulating experience across task boundaries because the setup throws the experience away between every task. CEEval formalizes the self-evolving agent as something that continuously evolves across tasks and builds an evaluation around an evolutionary flywheel. The idea that each task should leave the agent better positioned for the next. And I think this is the necessary companion to all the self-evolution systems papers because it’s very easy to build a system that claims to learn from its traces and very hard to prove that the learning compounds rather than plateaus or worse drifts.

If TMEM and Socratic SWE are answers CEEval is insisting we measure them on the axis that actually matters which is not did it do well on this task by getting better across tasks in a way that holds up. That’s the CEEval discipline catching up to the system’s ambition which is a healthy thing to see happen in the same week. Step back for a second and notice what just happened in that memory and evolution stretch because there’s a clean progression. The naive position is stuff everything in the context window.

The first correction was retrieval pull back the relevant slice. The cross scenario paper this week says actually let the agent control its own retrieval with simple files. The reconstruction paper says make that retrieval iterative and reasoning driven instead of one shot. The temporal paper says and don’t forget when things happened.

The parametric memory paper says and stop pretending the prompt is the only place experience can live. Put some of it in the weights. And CEEval says and measure whether any of this actually compounds. That’s six papers in one week walking a single idea from store it to control it to learn from it to prove it learned.

When a field moves that coherently in seven days it means the easy version of the problem is dead and everyone’s converged on the same harder version. Okay, next theme and it’s the one I think is the most decision relevant if you’re actually building agent systems today. Multi-agent orchestration finally got an honest accounting this week. The pitch for multi-agent systems has been remarkably stable for a couple of years now.

Decompose the task. Isolate context so each agent has a clean window. Run them in parallel and go faster. And it’s an appealing story.

But there’s a paper this week called When Parallelism Pays Off from a group led by Xu Yang with Swarat Choudhary among the authors that actually puts the bill on the table and the bill has a cost side that the standard pitch tends to skip. Their framing is to treat multi-agent orchestration as a graph partitioning problem. The benefit of decomposition is that it shortens the critical path the longest chain of dependent computation. But every time you split work across agents and those pieces depend on each other you have to pay to transfer content between them.

And that interagent communication overhead is real. It costs tokens and money and latency and sometimes it’s larger than the speedup you bought by parallelizing. So the question isn’t should I use multiple agents? It’s does the structure of this particular task have cohesive loosely coupled pieces or is everything tangled together?

Because if it’s tangled parallelizing just means you pay the transfer cost over and over for no real critical path win. And then they build a system around that insight. Called co-coder. Cohesion aware coder.

It builds a dependency graph of the repository from static analysis. It isolates the structural hub files the ones everything depends on and treats them specially instead of letting them get split across agents. It partitions the rest of the graph using community detection which is a graph clustering technique that finds tightly connected neighborhoods. And then it runs the partitions with a dependency aware scheduler.

According to cross 28 real world tasks on dev eval and code project eval co-coder beats three baselines plain sequential naive file based parallel and notably clod code with agent teams. 10 times wall clock speed up and cuts API costs by up to 35 percent and the gains are biggest exactly where the theory predicts on the most dependency dense projects because that’s what we’re going to be talking about in the next video. The system is nice but the principle is the takeaway. You can compute whether parallelism will pay before you spawn a single agent by looking at the cohesion of the partition and the sparsity of the dependencies.

Parallelism is not free and it is not always positive. Measure first. And that coordination tax shows up in two more papers this week that I think belong right next to it. The first one is the proof of gap from a group led by Yurin Sun and it benchmarks a really specific really real skill.

Can a model write a good orchestration prompt? Meaning when you’re the supervisor agent handing work to a subagent can you tell that subagent precisely what it needs to know and crucially not flood it with things it doesn’t need. They build 110 scenarios for this and the finding is that current models struggle with it. It’s not the only agent systems you know in your bones.

Getting the handoff prompt right giving the worker enough context to succeed but not so much that you’ve just recreated the monolithic agent with extra steps is genuinely hard and apparently the models are not yet good at doing it for themselves. The second is Channel Fracture by Levent Liu and this one is more of a concrete bug report which I appreciate. The setting is hierarchical agent teams with a schedule a supervisor writing into a worker’s memory say and the paper reports a systematic architectural blind spot in how that scheduled cross-agent memory injection works where the delivery mechanism breaks in ways the system itself never surfaces the knowledge doesn’t land and the reason I want these three next to each other co-coder and perspective gap and channel fracture is that they all converge on the same uncomfortable place as strained coherence and handoff debt and that’s where the cost and the bugs live now. And there’s a qualitative study this week that catches practitioners articulating exactly this called and I love the title so there’s a catch 22 here on how early adopters who build multi-agent systems conceptualize transparency.

It’s an interview study by a group at Levent Liu called Catch 22 on how early adopters build multi-agent systems at Microsoft including Mihaela Vorvaranu and the catch 22 the builders keep naming is that they desperately want transparency into the inter-agent coordination they want to see what their agents are saying to each other and why and the orchestration layer is precisely the place where observability is thinnest they’re flying with the least visibility exactly once the people who both wired up the multi-agent system and depend on its output so when they say transparency is underdefined in distributed agent architectures that’s not an outside critique that’s the people closest to it telling you the abstraction they reached for doesn’t have the introspection they need I think the pattern to take from this whole orchestration cluster is that the industry spent the last two years on the the the nothing as close as it can and the the the the the the the the the the the the the the the the handoff debt, the three memory papers, co-coder, perspective gap, channel fracture, the transparency study. Every single one of them is, in some form, an instrument, a way to see inside agent behavior that we didn’t have last month. The field is, right now, building the gauges and the dials and the warning lights for systems that, until very recently, we were mostly running blind. And here’s why that timing is interesting and a little tense.

Because at the exact moment the field is building all this instrumentation, adoption is climbing, fast. There’s a paper this week with the wonderful title, Agentic Very Much, from Romain Robbs and Collaborators, and it’s a follow-up to earlier work those same authors did measuring coding agent adoption on GitHub. In the earlier study, they found adoption was already very significant. This time, they look at projects created after that first study, a fresh sample, and they find adoption is more than twice as high.

A higher proportion of the commits in these projects are AI-based, AI-assisted, and they add the caveat that they have strong signs they’re undercounting, that a lot of agent activity doesn’t leave a detectable signature, so the true number is higher than what they can measure. So the curve is bending up, fast, and the measurement is conservative. And the tooling is hardening around that reality. GitHub shipped a co-pilot desktop app this week, which they’re explicitly framing as agent-native.

The whole point is that it’s built for directing multiple agents working in parallel, rather than being an editor, designed for one human typing that has agents bolted onto the side. The framing in their announcement is worth quoting in spirit, because it names the same pain the research names. The agentic shift made development faster, but it also created disjointed workflows, more context switching, and too much time spent reviewing agent-generated code. And the conclusion they draw is that if agents are going to be a durable part of how software gets built, they need a real place in the developer workflow.

Not a panel in the corner of an editor. Whether or not the product itself delivers, the design premise is the tell. The serious tooling is being shaped around the assumption that you’re directing several agents at once, and that your scarce resource is your own attention spent reviewing their output. That’s the same world all those orchestration papers are trying to instrument, approached from the product side instead of the benchmark side.

And notice the review burden point lands right back on smell bench and strained coherence. The reason reviewing agents is so important is because they’re not just a tool for the user, they’re a tool for the user. Reviewing agent output eats so much time, is partly that you can’t trust it the way you’d trust a colleague’s. You have to check whether it’s maintainable, and whether the agent quietly worked against its own stated understanding.

And those are exactly the checks this week’s research is trying to automate. The product is telling you the bottleneck is review. The research is telling you what the automated reviewer would need to look for. There’s one more detail in the adoption study worth dwelling on, which is the undercounting.

The authors are explicit that they have strong guidelines, they’re not detecting all of the agent activity. That a meaningful fraction of AI-assisted work leaves no signature they can reliably attribute. So the more than doubling is a floor, not a ceiling. And that matters for the whole instrumentation argument.

Because it means the visibility problem isn’t just inside the agent trajectory, it’s at the population level too. We are getting worse at even knowing how much code is agent-written, at the same time we’re getting better at analyzing any individual agent run. The measurement is racing the adoption on two levels. The first one is that we need to know how much code is agent-written.

On the two fronts at once. And on the course front, the simple question of how much of this is even happening, adoption is winning. So put the two halves together. On one side, a week of research that is almost entirely about measurement and visibility and failure prediction.

About seeing how agents explore, where they contradict themselves, what they drop in a handoff. When parallelism actually pays. On the other side, an adoption curve that more than doubled, and product tooling reorganizing itself around fleets of parallel agents. The instruments are being built, and the autonomy is being handed out, more or less simultaneously.

And it’s not at all clear they’re moving at the same speed. Which is the thing I’d watch over the next stretch? The process metrics from something like SWE Explore, the failure signal from something like Strained Coherence, the cohesion test from CoCoder. These are all the kind of thing that could, in principle, live in your CI.

So let me actually play that out, because I think it’s more concrete than it sounds. Picture the pipeline an agent’s pull request goes through before it merges. Today, in most shops, that pipeline runs the tests, and if they’re green, a human skims the diff and approves. That’s it.

Now layer in this week’s instruments. Before the merge, a Strained Coherence-style detector reads the agent’s own trajectory and flags any span where the agent acknowledged a problem and proceeded anyway. And given that flagged trajectories failed 94% of the time in the study, a flagged trajectory is the most likely to fail, and the most likely to fail is a flagged trajectory. And given that flagged trajectories failed 94% of the time in the study, a flagged trajectory is a strong reason to send the work back rather than ship it.

A SWE Explore-style score checks whether the agent actually localized the regions that mattered with good line-level coverage under budget, instead of stumbling into a passing patch by luck, because a fix that passes the test for the wrong reasons is a fix that breaks the next time the tests change. A Smellbench-style maintainability check looks at whether the code the agent left behind is something a human will want to touch in six months, not just something that’s green-tinted. And if the task got parallelized across agents, a Cocoder-style cohesion check verifies the partition was actually worth it, rather than burning tokens shuttling context between agents that all needed the same hub file. And when this PR gets handed to the next agent, a Handoff Debt Aware step makes sure it leaves structured notes, because the data says those notes are worth roughly half the successor’s budget.

None of that is science fiction. Every one of those detectors exists right now, this week. It’s just a form with numbers attached. The pieces are on the table.

What’s missing is the integration work, and honestly, the will, because every one of those checks costs something. The strained coherence judge is itself an LLM reading full trajectories, which isn’t free. The SWE Explore scoring needs ground-truth regions. Maintainability scoring is fuzzier than a test suite, and someone has to trust it.

So there’s a real friction here, the same friction that’s kept a lot of good software engineering practice out of a lot of pipelines for decades. The green test is cheap and legible, and everybody already believes it, and the richer signal is expensive and requires you to change how you think about what done means. The optimistic read is that as agents write more of the code, the economics flip, because when a human wrote the diff you could trust their judgment about whether they contradicted themselves. And when an agent wrote it?

So the trajectory level checks stop being a luxury, and start being the only thing standing between you and a 94% failure rate run that makes you think you’ve got the right things. and a 94% failure rate run that happened to go green on a thin test suite. And that’s the tension I’d sit with going into next week. We have, in the span of a few days, watched the research community hand us a whole instrument panel, gauges for how the agent explored, warning lights for when it’s about to fail, meters for what a handoff costs, and whether parallelism paid, and whether the memory actually compounds.

And in the same few days, we watched adoption more than double, and the tooling reorganize itself around fleets of agents working in parallel. The instruments and the autonomy are arriving together, but they are not the same people shipping them, and they are almost certainly not moving at the same speed. The question for the next stretch is simple to state and hard to answer. Does any of this measurement cross over from the arXiv listing into the pipeline that actually decides whether agent code ships, before the next doubling of how much agent code there is?

Or does production keep scoring on green tests alone, while the gap between what we can see and what we gate on quietly widens? That gap is the most interesting space in the field right now, and it’s the one I’ll be watching. I’ll see you next week.