Agents remember corrections and still violate them

5%. That’s the fraction of applicable preference checks that still get violated when a coding agent runs with Mem0, a well-regarded memory layer, even though the relevant correction is sitting right there in memory. The agent can recall that you told it to use conventional commits or to never touch the lock file, and then it goes ahead and violates the preference anyway. 13174, and it gives a name to something a lot of us have felt without ever measuring, the gap between preference access and preference compliance.

It’s June 12th, and today’s window turned out to have a real spine to it. Five or six items landed in the last 36 hours that are all, in one way or another, about the same problem. The proxies we’ve been using to build and evaluate agent infrastructure are misspecified for what agents actually do. Memory systems optimize recall when the failure is compliance.

Retrieval benchmarks test snippet matching when agents need repository navigation. Code review assumes a human gate when the throughput has moved past what humans can gate. Let me walk through them. Start with that memory paper.

The system is called TRACE, which stands for Test Time Rule Acquisition and Compiled Enforcement, and the design is refreshingly unglamorous. It mines your corrections out of chat history, rewrites your code, and makes it easier to use. The system is called TRACE as an atomic rule, and then compiles those rules into runtime checks that have to pass before the agent is allowed to finish a task. The rules aren’t injected into the prompt as suggestions.

They run as gates, and the agent cannot finish the task while one of them is failing. 6% on in-distribution tasks. On out-of-distribution tasks, they drop to 2%. Sit with that second number for a moment, because it’s the second number.

The enforcement generalizes better to tasks the rule was never mined from. And the reason makes sense once you see it. A compiled check runs no matter what, while a memory retrieval has to win a relevance contest against everything else in the context. The further the task drifts from where the correction happened, the less likely retrieval surface is it, but the check doesn’t care.

Anyone who runs a hooks layer in their coding agent harness already believes this thesis in their bones. What’s new is an evaluation that quantifies exactly how much memory it needs to leave on the table, and the answer is, more than half. The authors also shipped the thing as a deployable skill, not just experiment code, which tells you where they think it belongs, inside the harness as standard infrastructure. The same day brought a second memory paper that attacks the problem from the other end, and honestly, it might be the sharper of the two.

It’s called Learning What to Remember. 12945. And the core observation is about timing. The forgetting decision, what to consolidate deeply, and what to let go, gets made before the future query exists.

You decide at consolidation time. The query arrives later. So when production memory systems score memories by semantic similarity to recent context, or by recency itself, they’re using signals that are misspecified by construction, for the decision they’re making. You cannot score similarity to a question that hasn’t been asked yet.

Their answer is a value function over seven interpretable factors, borrowed from copy and paste. The answer is a value function over seven substantive psychology, things like reliability, emotional intensity, goal relevance, self and user relevance, usage history. The weights get learned from a downstream objective with a gradient-free optimizer, and one scaler then controls encoding depth for debt risk, and retrieval rank uniformly. 77 of gold evidence.

657. 518. 368. Every one of those gaps holds up under bootstrap confidence intervals.

But the part of this paper I’d actually forward to people is the methodological aside. 98. Read that again. Letting the consolidation step peak at the eval question, even indirectly, nearly solves the benchmark.

Which means any published memory result that leaks the query into the scoring step is measuring retrieval, not forgetting. And the two regimes differ by more than 20 points. If you maintain a memory eval harness, that critique applies to you today. And it’s worth an afternoon to check which regime you’re actually in.

I suspect several memory system leaderboards are due a correction if anyone reruns them blind. The misspecification theme carries straight into code retrieval. 11864, comes out of Alibaba’s embedding group, and the argument will sound familiar to anyone who’s watched a coding agent flail in a big repository. The classic code search benchmark is docstring to function matching.

Here’s a natural language description. Find the snippet. But that isn’t the retrieval problem agents have. An agent gets a user request and a concrete repository state.

And it has to be a concrete repository state. It has to locate the right files and functions, pull in supporting context. And this is the hard part. Reject the near-duplicate distractors that every large code base is full of.

Six functions that all look like they handle authentication, one of which is the one that actually runs. So they built a benchmark with three levels. Code understanding, issue-to-edit localization, and broader context retrieval, assembled from curated code search tasks and SWE-bench instances. The scale is real.

Over 180,000 queries, and 106,000 broader context relevance labels. And the headline finding is a sharp drop. Embedding models that look strong on traditional code search fall hard in the agentic setting. The encouraging part is that simple supervised fine-tuning on this kind of data recovers a lot of the gap, which suggests current code embedders aren’t at some capability ceiling.

They’re just underfit to the task that matters. Nobody had built the training signal before because nobody had built the benchmark. Now put empirical flesh on that from the industry side. Sourcegraph published an analysis of 1,281 agent runs across more than 40 large open-source repositories, asking why coding agents fail in big code bases.

The answer? The failures are not random. They cluster into five repeatable patterns. And each pattern points to a different infrastructure problem with a different fix.

I want to dwell on the framing rather than the taxonomy, because the framing is the contribution. When you watch a single agent run fail, it looks stochastic. The model just went off the rails somewhere. When you watch a thousand, structure appears.

And once failure has structure, the remedy stops being better prompting and starts being better context infrastructure, better retrieval, better repository state representation, better navigation tools. That’s the same conclusion CoreBench reached from the benchmark side in the same week from a completely different methodology. When the eval people and the production people converge on the diagnosis independently, that’s usually the signal worth trading on. One for the orchestration crowd.

13598, with Salesforce researchers on the author list, goes after a real bottleneck in multi-agent systems. How do you train the orchestrator? The thing deciding which sub-agent does what. The obvious supervision signal requires running full sub-agent rollouts to find out whether an orchestration decision was good, and rollouts are brutally expensive.

Their move is to build win-lose preference pairs from the intermediate artifacts that multi-agent executions already produce, then train a Bradley-Terry reward model directly at the orchestration level. No human annotations, no extra rollouts. Training efficiency improves by up to 10 times in token usage, and reward-guided test time scaling picks up as much as 8 points of accuracy, with the gains transferring across mathematical reasoning, web question answering, and multi-hop reasoning. The reason this matters practically?

Learned orchestration has mostly been a lab luxury because the supervision cost was absurd. Reward signal at 10 times cheaper changes who can afford to do it. And then there’s the code review fight, which gave us the most fun pairing of the day. 13175.

The argument has two prongs. First, every stated goal of code review, defect detection, and error detection. Second, every stated goal of code review, defect detection, knowledge transfer, consistency enforcement, can be served by agents at lower cost and higher throughput. Second, and this is the prong I find harder to dismiss, the naive integration everyone currently runs, where agents write the code and humans remain the mandatory reviewers, does not serve them well.

Is a dead end. It doesn’t provide meaningful assurance, because humans rubber stamp at volume, and it doesn’t scale, because agent throughput grows and human attention doesn’t. Whatever you think of the conclusion, the dead end, the claim about the current default, deserves a real answer, rather than a reflexive defense of human oversight. The same day, Cursor shipped numbers that read like a market answer to the academic question.

Their bug bot review agent is now over three times faster, 22% cheaper, and finds 10% more bugs, and there’s a local slash review command that runs before you push. So while the academy debates whether human code review survives, the market is already pricing agent review as a product and competing on its unit economics. Both can be right on different timescales. Review as a human practice may have a long tail.

Review as the binding quality gate is already migrating. Step back, and the day has one shape. Trace says recall is the wrong proxy for compliance. The memory value paper says similarity and recency are the wrong proxies for what to keep.

Corebench says snippet matching is the wrong proxy for agentic retrieval. Sourcegraph says single run debugging is the wrong lens on agent failure. Mon Paris says human inspection is the wrong gate at agent throughput. Five corrections to five proxies, all landing within a day and a half of each other.

The field is going back and re-deriving its evaluation layer from what agents actually do, instead of what assistants used to do. What I’m watching from here, whether enforcement layers like trace get absorbed into the major agent harnesses as standard skill infrastructure, the way Hux went from exotic to assumed over the past year. And whether anyone takes the blind regime critique seriously enough to re-run the published memory benchmarks. 77 gap is sitting there, reproducible, with open source code that runs on a CPU.

If it replicates, a fair amount of agent memory marketing is about to get mark to market.