The bottleneck in agent memory is evidence use, not retrieval

Here’s the number I keep coming back to from the last day or so. A team built a benchmark called MemTrace to study long-term memory in agents, the kind that’s supposed to remember facts about you across sessions. They ran 13 different memory system configurations across four paradigms, and when one of those systems failed to answer a question correctly, they checked whether the evidence it needed was actually available to it. It was.

The supporting fact was sitting in the retrievable set about 10 times more often than it was genuinely missing. So the system had what it needed, and it still got the answer wrong. That reframes a lot of how we’ve been thinking about agent memory. The default assumption is that memory failures are retrieval failures.

You didn’t store the right thing or you couldn’t find it, so you add a better embedding model, a bigger index, a smarter chunking strategy. MemTrace says that for the most part that’s not where the failure lives. The bottleneck is evidence use, not storage and not retrieval. The model had the fact in context.

And didn’t use it correctly. Adding more index doesn’t fix that. What makes the benchmark interesting beyond that one number is how they measured it. Most memory benchmarks take a pile of questions, score each one independently, and average them.

MemTrace says the unit of measurement is wrong. Instead of scoring questions, they score knowledge points, a single typed fact about the user. And then they probe each fact along three controlled dimensions. When is memory age?

How many sessions ago did this fact show? Another is question type. Are you asking about the current state of the fact, an earlier state, or the trajectory of how it changed over time? And the third is evidence condition.

Is the supporting evidence present, missing, or actively contradicted by a false premise in the question? When you slice it that way, similar-looking pooled accuracy scores fall apart into very different failure profiles. Being able to recover a fact’s current state and its earlier state tells you nothing about whether the system is working. The system can track how the fact changed between them.

And a system that safely abstains when it doesn’t know something is not the same as a system that can catch and correct a false premise you’ve slipped into the question. Those are different competencies, and the average score smears them together. Now there is a real retrieval half to this problem, and a paper from two days earlier called T-Mem goes right at it. Their observation is that current long-term conversational memory is reachability-bounded by surface similarity.

Whether you’re using keyword matching or data matching, it’s not the same thing. If you have dense vectors, you can only reach a memory if the query looks like the stored content. That works fine when the query and the memory share wording or named entities. They call that the descriptive case.

But there’s a second case where the query and the memory don’t share any surface features and are connected only by a latent semantic arc. They call that associative, and they show that prevailing memory systems basically fail on it as a class. Their fix is clever. At right time, when they store a memory, they don’t have to worry about it.

They don’t have to worry about it. They don’t have to worry about it. They don’t have to worry about it. They also store rehearsals, what they call triggers.

The idea is to anticipate the future contexts in which you’ll want to retrieve this memory and write down hooks for those contexts now, so the memory stays reachable from both surface-similar queries and meaning-bound ones. They explicitly frame it as the engineering version of what cognitive science calls episodic future thinking, rehearsing a past experience against the situations where you’ll need to recall it. It hits state-of-the-art on the LoCoMo and LoCoMo-possessive side. It hits state-of-the-art on the LoCoMo-possessive side.

And if you put T-Mem and MemTrace next to each other, you get a clean two-part program for agent memory. First, make the right evidence reachable, which is what T-Mem is about. Second, get the model to actually use the evidence once it’s reachable, which is the gap MemTrace exposes. Both halves matter, and most systems are only working on the first one.

That same measurement problem the one MemTrace raises for memory shows up one level higher at the harness. There’s a new evaluation environment called C-Gym, posted today, aimed at self-evolving agents. These are agents that improve not by retraining their weights but by rewriting their own scaffolding, their prompts, their memory, their tools, their control flow, the whole structured layer around the base model. The problem C-Gym points at is that people evaluate these systems with a single accuracy curve over evolution rounds, and a single curve hides almost everything you care about.

Did an update produce improvement that actually generalizes, or did it overfit the most recent batch of tasks? Did it raise cost? Did it quietly break behavior that used to work? So C-Gym splits the evaluation into separate views.

There’s the training batch, a frozen update validation set, held out in distribution and out of distribution transfer sets, replay diagnostics, and cost records. They run three different self-evolution methods on Terminal-Bench 2 and on HLE under one shared protocol, and the findings are a caution for anyone running a self-improving loop in production. Frequent updates often fail to move held-out performance at all. And intermediate snapshots that look like real progress during the run can collapse later.

The lesson is that evolution you don’t measure against held-out and replay views isn’t evolution, it’s overfitting with extra ceremony. We also got some concrete model numbers worth knowing. 8 across 917 shared coding agent scenarios, drawn from registry skills, and they scored each task both with and without the relevant skill loaded so they could separate the model’s contribution from the scaffolds. 8 scored 92.

9 point difference. 25 per task versus 74 cents for Opus, so you’re paying roughly a 73% premium for that fraction of a point. But the number I’d actually take away is the skill effect. Loading the relevant skill added about 17 points for both models, and almost all of that lift was in instruction following, not in task completion.

In other words, both models could usually get to the right answer on their own. What the skill changed was how reliably they followed the project’s conventions, its workflows, its specific requirements. That’s the model versus scaffold split again, and it keeps telling us the same thing. The scaffold is carrying more of the load, than the leaderboard gap between two frontier models would suggest.

One footnote that stuck with me, Fable refused 26 tasks that Opus completed, and several of those were security review tasks. Now, while all these benchmarks are arguing about quality, the plumbing underneath multi-agent systems is breaking in ways no task score will ever catch. There’s a paper this week on concurrency anomalies in multi-agent LLM systems, and what I like about it is that it’s not hand waving. It models the way agents share state, through memory stores, vector indices, tool registries, as long-running red-generate-right operations, and then formalizes four specific anomalies in TLA+.

Stale generation, phantom tool, causal cascade, and tool effect reordering. These are structural cousins of the classic database isolation anomalies, except now there’s a non-deterministic generation step sitting in the middle of the read and the write. And they ground it in real life. It’s a very simple system.

They reproduce a silent lost update in ByteDance’s DeerFlow framework and formalize the fix as a verified refinement. They show tool effect reordering happening in LangGraph’s tool node on unmodified output and remove it with a commit order sequencer. The detectors are proved sound and complete with 274 proof obligations in Veris, zero assumptions, zero admits. The takeaway for anyone building with these frameworks, the moment you have two agents reading shared memory, generating against what they read, and writing back concurrently, you have a textbook isolation problem, and most frameworks ship without the locking that situation demands.

There’s an adversarial version of bypassing task scores too. Tenet Security published an attack they call agent jacking. They plant malicious instructions inside a fake Sentry error report and get a coding agent to execute them during ordinary debugging. The way in is the Sentry DSN, which is a write-only credential that’s commonly embedded right in a website’s front-end JavaScript and treated as public.

If an attacker grabs that DSN, they can submit a forged error event and control the message, the stack trace, the tags, even a fake resolution section that contains an NPX command. When the agent pulls that error through Sentry’s MCP server, it reads like legitimate diagnostic guidance, and the agent runs the command with the developer’s own local permissions, downloading and executing the attacker’s password. Tenet reports an 85% success rate, more than 100 agent executions in testing, and 2,388 organizations whose DSNs accepted injected events. The structural point is the sharp one.

The MCP server only fetches data, the command line access is a separate tool, and the exploit only exists in the combination of the two. So you have to audit an agent’s permissions as a whole set, not one integration at a time. And to close on something, more straightforwardly encouraging, retrieval quality is still improvable on its own terms. A paper called AlignCoder goes after the misalignment between a code completion query and the actual target code.

1% improvement in exact match on CrossCodeEval across five different backbone models. If there’s one thread running through all of this, it’s that the easy metrics have stopped being informative. Pooled accuracy, a single evolution curve, a one-point leaderboard gap, a green task run, every one of those hides the thing that actually decides whether the system works in practice. So what I’m watching next is which benchmarks start reporting the structure of their failures instead of just the mean, and which agent frameworks start treating shared-state correctness as a property they prove to be true.

So what I’m watching next is which benchmarks start treating shared-state correctness as a property they prove rather than a surprise they discover in production. That’s where the signal is going to be.