The benchmarks caught up to the slop

Here’s the number I can’t stop thinking about from this week. More than half of the passing results on SWE-bench, the benchmark that’s defined coding agent progress for the last two and a half years, are unmergible slop. That’s the finding from METR’s work behind FrontierCode, the new evaluation that Cognition put out. And the way they got there is the interesting part.

They didn’t ask the usual question, which is did the patch make the test suite turn green? They asked a different question, the one that actually matters if you’ve ever maintained a code base, which is would a maintainer actually merge this? They put more than a thousand hours of maintainer-validated software engineering work behind it, more than 3,000 rubrics, IOI gold medalists and senior code maintainers grading whether the output is something you’d accept into a real project. 8%.

So this is the week I want to walk through, because almost everything else that’s shipped is, in one way or another, a reaction to that same gap. The gap between code that passes and code that’s good. And I want to dwell on what unmergible slop actually means, because it’s easy to hear that phrase and round it off to the code is buggy. That’s not it.

A lot of the slop passes the tests, that’s the whole point. It’s code that satisfies the assertions and is still something no maintainer would accept because it’s unmaintainable. Or it solves the test rather than the problem, or it games the reward signal. The METR framing includes anti-cheat rubrics specifically, because once you put a model in a loop with a test suite and a reward for passing, you have built an optimization process, and optimization processes find the cheap path.

They’ll hardcode the expected output. They’ll special case the test inputs. They’ll delete the failing assertion if nobody’s watching. None of that shows up as a failure on SWE-bench, because SWE-bench only checks whether the test is good or not.

The suite is green. So when METR says half the passing results are slop, what they’re really saying is that a benchmark which only checks the suite is, at this point in the capability curve, measuring something close to noise on the quality dimension. The agents got good enough to satisfy the proxy, and the proxy stopped tracking the target. Let me stay on FrontierCode for a minute because I think the framing is more useful than the leaderboard.

swyx, who was involved in the launch, laid it out as three errors. The first error is of coding benchmarks, and the periodization is clarifying. Error 1, 2021, auto-complete, and the benchmark was human eval. Can the model produce a function that passes a handful of unit tests?

Error 2, 2023, passing tests at the repository level, and that’s SWE-bench and Terminal-Bench. Can the agent resolve a real GitHub issue such that the existing test suite goes green? And now Error 3, 2026, maintainable code, which is FrontierCode. And the reason you need a new browser?

The benchmark for the new era is exactly the slop finding. Once agents got good enough to make the test suite pass most of the time, passing the test suite stopped being informative. You can turn the suite green by reward hacking, by writing code that technically satisfies the assertion and is unmaintainable garbage underneath, and SWE-bench can’t tell the difference. FrontierCode’s rubrics are built specifically to catch that, the reward hacking and the anti-patterns that plague the older benchmarks.

And there’s a historical chart in the launch that I keep coming back to. On the easiest third of FrontierCode tasks, the tier they call FC Extended, Opus nearly doubled its pass rate over about four months in late 2025, going from roughly 41% to 74%. And swyx connects that to what a lot of people, from DHH to Karpathy, called the vibe shift, the sense that sometime around December 2025, coding agents crossed a line from interesting, to genuinely useful. The mechanism he proposes is concrete, and I find it convincing.

It’s the difference between getting to a successful result in two rerolls versus six. Once the success rate per attempt is high enough, you can stack a loop on top. You can let the agent run in a Geoffrey Huntley style Ralph loop or a Goals style outer loop, without it spiraling off the rails so often that babysitting it costs more than doing the work yourself. So the capability jump didn’t just make single shots better, it opened a whole layer of automation on top because the floor got high enough to build on.

And I think that’s worth sitting with as a mental model for the whole field right now, because it explains a lot of the tooling churn we’ve been seeing. When per attempt success is low, your engineering effort goes into the single attempt, better prompts, better context, getting the one shot to land. When per attempt success crosses some threshold, and the FrontierCode data suggests that threshold got crossed for the easy tier in late 2025, the payoff moves up a level. Now it’s worth building the loop that runs the agent 10 times and picks the best result, or the loop that runs the agent, feeds the failure back in and lets it try again.

Geoffrey Huntley’s Ralph loops, the Goals style outer loops people are building on Claude Code, the whole loopcraft genre of stacking loops on loops, all of that only makes economic sense when the inner step succeeds often enough that the outer loop converges instead of wandering. So the vibe shift wasn’t just models got smarter. If you want to learn more about Goals style outer loops, it was models got reliable enough that automation on top of them became rational. And that’s also why efficiency, which I’ll get to with Kimi, suddenly matters so much because the moment you’re running the inner step 10 times in a loop, the per step token cost gets multiplied by 10 and the economics of the whole loop hinge on it.

Now hold that next to a paper that dropped this week with a deliberately provocative title. Martin Monparis put out something called the End of Code Review, Coding Agents Supersede Human Inspection. And the argument is basically that human code review, which has been the primary quality gate in software since Michael Fagan formalized code inspection back in 1976, is on its way out because coding agents can now raid, write, test and repair software autonomously. So the human looking over the change before merge becomes redundant.

And look, it’s a real argument and worth engaging with seriously. But put it next to the FrontierCode number, and I think it reads as premature. If agents produce mergeable maintainable code only about one time in seven on hard tasks, then the inspection step isn’t the obsolete ceremony the paper frames it as. It’s the thing keeping the slop out of your main branch.

The two papers can’t both be right. And I actually think that’s healthy because it means the field is going to settle this empirically over the next year rather than by vibes. My read is that review doesn’t end, it changes shape. It becomes review of agent output.

Possibly by other agents, possibly with human spot checking. But the function, the gate that asks, is this actually good? That function gets more important as volume goes up, not less. And there’s a volume argument that the end of review thesis actually strengthens rather than weakens.

If you turn it around, if agents are generating 10 times, 100 times more code than humans used to write by hand, then the review burden scales with that volume. And human only review becomes the bottleneck precisely because there’s so much more to review. So you do need to automate the review. But automating review is not the same as ending review.

It’s the opposite. It’s admitting that review is so essential you can’t afford to have it be the slow human step anymore. And the obvious place this goes is agents reviewing agents, a generator critic split where one agent writes and another, ideally with a different prompt or a different model so it’s blind spots don’t correlate, tries to find what’s wrong. We’re already seeing this in practice.

The multi-model review setups, the adversarial verification passes. The FrontierCode rubrics are, in a sense, a frozen snapshot of what a good automated reviewer should be checking for. So I’d reframe Momperus’s paper as human inspection in its 1976 form ends and is replaced by a much more intense, automated, continuous form of inspection because the slop rate demands it, the gate doesn’t open, it gets a faster doorman. And underneath the question of, is the merged code good, there’s a question of, can the agent even find the right code to change, and that’s where a benchmark called corebench comes in.

The insight here is that code retrieval for agents is not the same problem as code search. The old code search benchmarks measure something like, given a natural language description, find the matching function, basically docstring to function matching, but that’s not what an agent actually does. Given a real user request, the agent has to navigate a concrete repository state, figure out which files, and functions are relevant, gather the supporting context around them, and crucially filter out the in-repository distractors, the code that looks relevant to a similarity search but actually isn’t the thing you need to touch. Corebench evaluates that whole loop, and it’s the same intellectual move FrontierCode makes, just one level down the stack.

Stop scoring the convenient proxy, the docstring match, or the green test suite, and start scoring the thing that actually determines whether the agent succeeds in a real repository. This pattern repeating all week, and once you notice it you can’t unsee it. The benchmarks are migrating toward reality. And the distractor piece of corebench is the part I’d underline for anyone building retrieval for a coding agent, because it’s the failure mode you don’t anticipate until you’ve shipped.

In a real repository you have near duplicates everywhere. You have the function and its three slightly different callers. You have the deprecated version and the current version sitting side by side because nobody deleted the old one. You have test fixtures that mirror production code closely enough that a similarity search can’t tell them apart.

So when the agent’s retrieval grabs the top K most similar chunks, a lot of what comes back is in the right neighborhood and is still exactly the wrong thing to edit. And if the agent acts on the distractor, it edits the deprecated copy, or it modifies the test instead of the code under test, and now you have a change that might even pass review at a glance but doesn’t do what was asked. Corebench is measuring resistance to that, the ability to not just find relevant code, but to discriminate among things that all look relevant. And that’s a much harder and much more realistic target than docstring matching, because in the older benchmark there’s one right answer and a bunch of obviously wrong ones, and in the real repo there’s one right answer and a dozen plausible decoys.

So let’s talk about what happens in real repositories, because that’s the other big theme. Sourcegraph published an analysis that I think is one of the more useful things this week, precisely because it’s empirical and not a model release. They looked at 1,281 agent runs across more than 40 large open source repositories, and they found five repeatable failure patterns, not random failures, not flakiness, but structural patterns that show up across different coding agents. And the framing they put on it is the key takeaway.

Each failure pattern points to a different infrastructure problem and a different fix. In other words, when an agent fails in a big code base, it’s usually not because the model is dumb, it’s because the context layer and the tooling around the model didn’t give it what it needed. The failures are in the harness, not the weights. And that reframe matters for where you spend your effort, because if you believe the failures are model failures, you wait for the next model.

But if you believe they’re infrastructure failures, you go build better retrieval and better context plumbing right now. And the repeatable part is what makes this actionable rather than just interesting. If agent failures in large codebases were random, one-off, idiosyncratic to each repo, there’d be nothing to do but shrug and hope. But repeatable patterns are fixable patterns.

A repeatable failure means there’s a specific, nameable gap. The agent can’t see the build configuration, or it doesn’t know which of three similar modules is the live one, or it loses track of a change it made 40 steps ago, and each of those has a concrete engineering response. That’s why I think the Sourcegraph piece pairs so well with everything else this week. Corbench is measuring one of those gaps that can’t find the right code gap.

The memory work is measuring another that doesn’t remember what it did gap. GitLab Orbit is productizing a fix for a third that doesn’t understand the system around the code gap. So the failure taxonomy and the solutions are converging on the same map, and the map says the same thing over and over. The model is mostly fine.

The scaffolding around it is where the work is. And I’d argue that’s actually good. Good news for practitioners, because scaffolding is something you can build and ship on your own timeline, whereas a better base model is something you wait for someone else to hand you. That connects directly to GitLab Orbit, which GitLab announced this week at their Transcend event.

The thesis behind Orbit is stated almost exactly the way the Sourcegraph data implies it should be. Agents are good at writing code. They’re far worse at navigating the system around the code, the related code, the pipelines that run it, the deployments that ship it, the work items that ask for it, and the teams that own it. And in a large monorepo, that gap is where A-assisted engineering breaks down.

It shows up as wasted iterations, blown token budgets, and code that looks correct in isolation but is wrong in context. So Orbit’s pitch is full code and lifecycle context in a single query. Give the agent one place to understand not just the code but everything around it. Whether the specific product delivers we’ll see, but the diagnosis is right, and it’s the same diagnosis Sourcegraph arrived at from the failure.

The bottleneck has moved from the model to the context. The model releases this week were pushing on the same frontier, the long horizon, large repository frontier, but from the capability side. Xiaomi open-sourced something called MiMo code, a coding harness specifically tuned for ultra-long tasks. And the claim that got attention is that it beats Claude code on tasks that run 200 plus steps.

Long horizon is the operative phrase. It’s one thing to resolve a tight, tight code. It’s one thing to resolve a tight, tight code. It’s another to keep an agent coherent across hundreds of tool calls without it losing the plot, and that’s the capability MiMo is targeting.

And it being open-source matters because the long horizon harness design has mostly been proprietary, locked inside the frontier labs and the agent companies, so having an open one to study and build on is genuinely useful. But the model release I found more interesting as a signal was Moonshot’s Kimi K2-7 code. And I want to be careful here. Because the headline numbers are the boring part.

9 to 62. 1. The benchmarks moved up. That’s expected.

Every release moves the benchmarks up. The part that actually matters and the practitioner discussion on this got it right is efficiency. Same model family, 1 trillion parameter mixture of experts, 32 billion active parameters, 256 K context. Basically the same spec as the previous version, but it uses roughly 30% fewer reasoning tokens to get there.

And that’s the axis that’s becoming the real differentiator now. Once raw capability is widely available, and it increasingly is with strong open models shipping every few weeks, the thing that separates a good harness from a bad one is cost per resolved task. How many tokens did you burn to get the merge quality result? A model that’s equally capable, but 30% cheaper per task is a bigger deal than a model that’s two points higher on a benchmark because the cost is what determines whether you can actually afford to run the outer loops, the roles, the multi agent fan outs that the FrontierCode era is built on.

Efficiency is what makes the whole stack loop architecture economically viable. Let me put some arithmetic on why 30% matters more than it sounds. If you’re running a single agent on a single task, 30% fewer reasoning tokens is a nice cost saving. Maybe you don’t even notice it, but the architecture everyone’s moving toward isn’t single agent, single shot.

It’s the loop, the panel, the fan out. Say you run a best of eight, you generate eight candidate solutions and pick the best. Now that 30% compounds against eight runs. Say each of those runs is itself a small multi agent system with a few sub agents.

Now it’s compounding again. The reasoning token cost is the base of an exponent in these architectures and shaving the base by a third changes. What you can afford to do on top. It’s the difference between being able to run a five way verification panel on every change and only being able to afford it on the changes you’re already worried about.

So I read Kimi K27 less as here’s a slightly better coder and more as here’s evidence that the open model ecosystem is now optimizing the right variable for a while. Everyone was chasing the capability number because that’s what the leaderboards rewarded. The fact that a major release is leading with a token efficiency story tells you the market has figured out that. Past a certain capability floor cost per resolved task is the thing that actually constrains what you can build.

There’s one more coding result I want to flag because it complicates the happy story in a useful way. A paper called frontier coding agents use meta programming to adapt to unfamiliar programming languages. They took six current coding agents and evaluated them on four esoteric programming languages, the kind with no big public corpus to learn from. And what they found.

Is that the agents reach for meta programming. They write code that generates code when the target language doesn’t have a comparable prior open source artifact to imitate, which is a fascinating behavioral finding. But the implication for how we read all the other benchmarks is the important bit. A lot of the strong numbers we see on mainstream languages partly measure how much of that language is already baked into the training set.

The agent isn’t reasoning from first principles about Python. It’s pattern completing against the millions of Python. Repositories. It’s seen pull away that crutch.

Give it a language. It’s never really seen and the behavior changes. It falls back to these more general meta programming strategies. It doesn’t mean the agents are fake, but it does mean we should be humble about what the public repo benchmarks are actually measuring because some fraction of the score is memorization dressed up as capability.

And the meta programming behavior itself is genuinely interesting as a window into how these systems generalize when the agent hits a language. It has no template for it. It doesn’t freeze. It reaches for a more abstract strategy, generating code that generates code, which is a way of routing around its lack of idiomatic knowledge by operating one level up.

That’s actually a sophisticated fallback, and you could read it as a sign of real reasoning rather than pure pattern matching. But the more useful read for our purposes is that it changes the shape of the evaluation. If you only ever test on mainstream languages, you never see this behavior, and you never learn that the agent’s competence is partly linear. city whose The game I’ve been playing is fighting me to death.

whether that means that it’s difficult to pronounce and that it’s scary, it’s bullying me, it’s stopping me from failing. The esoteric language test is doing the same job FrontierCode and core bench do in their domains. The unfamiliar language, the unmergible slop, the in-repo distractors, the corrections that don’t stick, each one names a specific thing the old benchmarks left out and measures it. Okay, let me pivot to memory because there was a genuinely strong cluster of memory work this week and the most pointed result is not about capacity, it’s about compliance.

The paper is called Getting Better at Working With You and it studies a gap that I think anyone who uses these agents daily has felt in their bones. The gap between an agent having access to your past corrections and the agent actually obeying them. You tell it, in this project we don’t use that pattern, please stop. It remembers in the sense that the correction is sitting in the memory store and then three sessions later it does the exact thing you told it not to and they quantified this.

5% of applicable preference checks violated. More than half of the time, it’s still the same. More than half, so the agent has the information, it’s retrievable and the agent still violates the preference more than half the time. Which tells you something important that storing the correction was never the hard part.

Enforcing it on the next action is the hard part and their proposed fix is architecturally interesting. It’s to compile the corrections into runtime enforcement rather than trusting retrieval to surface the right memory at the right moment. So instead of embed everything and hope the relevant memory comes back, we’re going to use the correct memory. When the correct memory comes back when it’s needed, you turn the correction into something more like a rule that actively gates the agent’s behavior.

And I think that’s the right instinct because retrieval based memory has this fundamental problem that the memory only helps if it happens to come back at the moment it’s relevant and relevance at storage time and relevance at action time are not the same thing. And it’s worth understanding why that gap is structural rather than something a better embedding model fixes. When you store a memory you index it by what it says. When you retrieve, you embed the current situation and pull back what’s nearby.

But a preference like don’t use this pattern is relevant not when the agent is thinking about that pattern, it’s relevant when the agent is about to write code that would use it. And at that moment the agent isn’t searching its memory for don’t use this pattern, it’s searching for how do I implement this feature. The trigger condition for the memory and the content of the memory point in different directions, so similarity search systematically misses. That’s exactly why compiling the correction into an enforcement rule works better, a rule doesn’t wait to be retrieved, it watches the action stream and fires when the forbidden thing is about to happen.

It is the difference between a librarian who hands you the safety manual if you think to ask, and a guardrail that stops you regardless of what you were thinking about, for corrections you want the guardrail. And almost the same idea shows up from the coding agent direction in a system called PROJCTMEM, which I really are not that familiar with. But I’ll show you what it’s like to do that, for instance. really liked because it’s concrete and shippable not just a paper they call the central idea memory as governance and the framing is that memory should not merely answer the agent it should act on the agent’s next action so here’s how it works it records your development process as an append-only plain text log of typed events issues attempts fixes decisions notes every meaningful thing that happens gets logged as a typed event and the log is immutable append-only which also gives you a provenance trail an auditable record of how the code got the way it is and then it deterministically projects that log into compact ai readable summaries that it serves over MCP the model context protocol but the part that makes it governance and not just storage is a pre-action gate before the agent does something the gate can warn it hey you already tried this fix and it failed or hey you’re about to edit a file that’s known to be fragile so the memory is intervening in the agent’s control flow not just sitting there waiting to be queried and the motivating cost they cite is very tangible reconstructing project context at the start of each session burns somewhere between 5 000 and 20 000 tokens and their point is that bottleneck is usually missing memory not missing model capability the model could do the task it just spent a fifth of its budget rediscovering what it already figured out yesterday and i want to flag the event sourcing choice in projc tmem because it’s an important design decision not just an implementation detail they store development as an append only log of typed events and then deterministically project that log into summaries anyone who’s done event sourcing in regular software engineering will recognize the shape and the reason it matters here is that it separates the raw record from the derived view the log is the source of truth immutable and the ai readable summary is a deterministic function of the log which may not be met with the attribute which will determine the outcome of an event of Stark means you can change how you summarize without losing the underlying history and you can always audit the summary against the events that produced it.

That’s a much more disciplined foundation than the more common approach of just having the agent write free text notes into a memory store because free text notes are lossy and you can’t reconstruct them, you can’t rederive a better summary later, and you can’t prove where a given belief came from. The provenance angle is the part I’d push on. An immutable typed log of issues, attempts, fixes, and decisions is also a record of why the code is the way it is, which is exactly the institutional knowledge that normally evaporates when a person leaves a team or an agent session ends. So PROJCTMEM is solving the token cost problem and the auditability problem with the same mechanism, and that’s the kind of two birds design that tends to stick around.

And the thing I want to underline connecting PROJCTMEM and the corrections paper is that they independently arrived at the same conclusion. Memory that acts on the next action beats memory that passively answers a query. That’s a real convergence, two different teams, two different angles, same architectural lesson. For the practitioner who’s actually trying to pick a memory system to use, rather than read papers about the concept, there was a nice survey-style write-up comparing Letta, Mem0, Graphiti, and Cognee, the main knowledge graph-backed memory, and the data stack-backed memory.

I won’t go through each one on audio because it’s the kind of thing you want to read with the comparison table in front of you, but it’s a useful map of where these systems differ and how they structure and retrieve memory. And I’d pair reading it with a sobering security result because these systems are walking into a threat model that mostly doesn’t exist yet in people’s heads. The paper is called SMSR, and it formalizes something they call multi-session memory poisoning. The attack is elegant and a little scary.

An adversary who interacts with the agent only through completely normal channels, no special access, no touching the weights, no touching the code, just talking to the agent the way any user would, can inject crafted memories. And those memories, once they’re stored and later retrieved, steer the agent’s responses for future users. So I poison the memory in my session, and you, a different user, get the steered behavior later, because the agent retrieved the memory I planted. And the reason this is a genuinely new attack surface, is the three properties they identify.

Persistence, the memory sticks around. Statefulness, the agent’s behavior now depends on accumulated state. And propagation, the poisoned memory can spread its influence across sessions and users. Input-centric security, the stuff we know how to do, sanitizing the prompt, doesn’t cover this, because the attack isn’t in any single input, it’s in the accumulated memory across time.

And the word certified in the title is doing real work, because it points at the harder version of the problem. It’s not enough to filter obviously malicious memories at right time, because the whole danger of memory poisoning is that the malicious memory looks like a perfectly normal observation when it goes in, and only becomes harmful in combination with how it’s later retrieved and acted on. So the defense has to reason about the memory life cycle as a whole, write, store, retrieve act, rather than checking any single step, and that’s a genuinely new security discipline. We spent years learning to think about prompt and rejection as an input validation problem, and now there’s this second surface where the dangerous content is something the agent itself wrote down, in good faith, based on an interaction that seemed fine at the time.

The threat models we have don’t transfer cleanly, and I’d expect this to be a growth area for the next year, because every team shipping persistent agent memory is opening this surface whether they’ve thought about it or not. And the lesson across all four of these memory results is the same and it’s worth saying plainly. Memory is not a database bolt onto an agent, it’s a control surface. It shapes behavior, it can be attacked, it can ignore you, it can be poisoned.

Treating it as passive storage is exactly how you end up with both the ignored corrections problem and the poison context problem. The systems that take memory seriously treat it as part of the control loop. Let me move to orchestration multi-agent systems, because there were two papers this week that both go after the same structural weakness, the central controller. The dominant pattern in multi-agent systems right now is centralized orchestration.

You have a main agent, an orchestrator, that decomposes the problem, assigns subtasks to worker agents, collects their outputs, and merges everything back together. And it works, but it has an obvious scaling problem, which is that as the number of subtasks grows, that central controller becomes a bottleneck, both for communication, everything routes through it, and for integration, it has to merge everything. Azalia Mirhoseini’s group proposed an alternative they call DLM, decentralized language models, and the design is instead of a central orchestrator, you have parallel agents that asynchronously claim subtasks from a shared task queue. Each agent reads the accumulated progress so far, does its local reasoning, and writes back a compact, verified update to a shared context.

And that shared context is the key piece, it’s the communication substrate. Agents build on each other’s verified progress by reading and writing the shared context, so no update has to be added to it. The correct way to do this is to do so is to create multiple sub-tasks from a single task and compare them. So, you can do it by weaknesses, by adding sub-tasks from multiple sub-tasks.

Or if you have two sub-tasks, you can add to each other and then you can use multiple sub-tasks to create different sub-tasks. But this doesn’t so no update has to route through a central controller. It’s coordination through a shared blackboard rather than through a manager, and the results are strong in a way that matters. 5 percentage points over the strongest baseline.

But here’s the number I’d actually flag, it reduces cost per task by roughly 50%, and that’s the opposite of the usual trade. Normally when you decentralize you buy throughput and parallelism at a token premium, you pay more because there’s redundant work and more communication. DLM is claiming better accuracy and half the cost, which if it holds up is a genuinely interesting result, because it means the central controller wasn’t just a bottleneck, it was also expensive, and routing around it pays off twice. The shared context design is worth dwelling on because it’s a very easy to use and it’s very easy to use.

It’s an old idea coming back in a new form. What the LM is describing, a common substrate that agents read from and write verified updates to, asynchronously, without a manager routing the messages, is essentially a blackboard architecture, which is a coordination pattern that goes back decades in AI. You have a shared workspace, multiple independent specialists, and they contribute to the shared state when they have something to add, rather than being told what to do by a controller. And the reason it’s important to have a shared workspace is because it’s a good fit for LLM agents specifically, is that the expensive thing in a centralized system isn’t just the controller’s compute, it’s that every piece of information has to be serialized through the controller’s context window.

The controller has to read every worker’s output, hold it all in context, and produce the merge. That’s a lot of tokens spent on one agent doing integration. When you replace that with a shared context that agents read selectively and write compactly to, you’ve distributed the integration work and you’ve stopped forcing everything to one context window. The verified part is also load-bearing, because a blackboard where agents write unverified junk just propagates errors faster.

The compact and verified updates are what keep the shared context from becoming the same kind of noisy dilution problem we just talked about with retrieval. So the LLM is, in a way, applying the retrieveless-but-better lesson to interagent communication, write less but verified, rather than dumping every agent’s full output into a shared pool. And it raises a question I think the field hasn’t fully answered, which is why centralized orchestration became the default in the first place if it’s this much more expensive. And I think the honest answer is that it’s easier to reason about and easier to debug.

A central controller gives you a single place to look when something goes wrong, a single trace of who was assigned what. Decentralized systems with shared mutable state are harder to debug precisely because there’s no single operator of what happened, the behavior emerges from the interleaving of asynchronous writes. So there’s a real trade-off here that the cost and accuracy numbers don’t fully capture. The LLM might be cheaper and more accurate, but if it’s harder to understand when it misbehaves, adoption will lag the benchmark wins, the same way it usually does when you trade a legible architecture for a more efficient but opaque one.

The companion paper is about a prior problem, which is okay. The LLM is a problem that can be solved in a way that can be solved in a more efficient way. Even if you want a smart orchestrator, how do you train one? And the issue is that supervision is scarce.

You don’t have a big labeled data set of good orchestration versus bad orchestration, and the compute to learn it the brute force way is expensive. So this paper on orchestration reward modeling, OrchRM, proposes a self-supervised framework that scores orchestration quality from intermediate signals, without human annotation. And I flag it not because the result is splashy, but because it’s the kind of infrastructure that has to exist before the more exciting stuff is practical. Right now most multi-agent orchestration is hand-tuned, somebody sat down and wrote the routing logic and the decomposition strategy by hand.

If you want learned orchestration, routing decisions that adapt and improve, you need a reward signal for orchestration quality and you need it to not depend on expensive human labels. OrchRM is a step toward that. It’s plumbing, but it’s the plumbing that the learned orchestration future needs. 8 percent on diamond is a humbling number to put on a slide so there’s a real incentive problem for FrontierCode to be adopted partly because the numbers were flattering here the benchmark that’s honest about how far we have to go is the one that’s least comfortable to adopt and whether the field chooses the comfortable proxy or the honest measure is going to tell us a lot about how seriously it takes the slop problem that’s the thing i’d keep an eye on whether honesty wins when the honest number is embarrassing i’ll see you next week