Memory only helps your agent when it's looking at a near-duplicate

Let me start with a result that a group of researchers seemed almost determined to prove wrong about their own work. The paper is called Git of Thoughts, it went up in the last day and the headline pitch is a clean one. Store an agent’s reasoning the way we store everything else in software in Git. Every scored thought becomes a commit, the scores live as Git notes, outcomes are tags, and when the agent wants to recall something, retrieval is just a Git log over its own history.

Replayable, diffable, mergeable across agents at basically zero engineering cost, nice idea, but then they ask the harder question, the one most memory papers skip, which is, does any of this actually make the agent more accurate? And they tested it properly, five different memory substrates, no memory at all, markdown, a vector store, a knowledge graph in their Git approach, two benchmarks, two model scales, and pre-registered replications, which is the part that matters because it means they committed to what counts as success before they saw the numbers. For novel problems, the answer is no. No memory format reliably helped.

They found there’s a threshold they call the copyability threshold. 8, accuracy jumps sharply. Below that line, nothing happens, and here’s the sting, the gain is answer retrieval, not method transfer. The agent isn’t learning a technique from a worked example and applying it to a new situation.

It’s finding a problem that’s almost identical and copyable. The answer is no. No memory format reliably helped. They found there’s a They even ran a bigger model, four and a half times larger, and it doubled the near duplicate payoff but still could not extract a transferable method from an example.

So what survives, the case for Git as the substrate is an accuracy. It’s auditability, provenance, and the ability to merge reasoning across agents, all at accuracy parity with everything else. And to their credit, they document a result they had to retract and a hypothesis they refuted, just to set the bar for how they want this kind of work evaluated. It’s rare to see a memory paper tell you exactly when its own mechanism does nothing.

Now hold that thought next to a second paper from the same window, stream and bench. This one builds a benchmark out of egocentric video, the egolife streams, where you’re watching the world from a person’s point of view over time. Around each piece of evidence in the stream they construct a two-step task. The first step tests whether the agent actually uses something it observed.

The second step, the follow-up, tests whether it reuses feedback and interaction experience from that first step. They run eight memory systems across two different backbone models and what they find rhymes with Git of thoughts. The systems routinely fail to use the evidence they observed, and they fail to turn feedback into reliable follow-up behavior, even in the cases where the evidence was stored correctly and the feedback was incorporated locally. So storing worked, using didn’t.

That’s the throughline I want to pull on today. If you’re interested in learning more about Git, please subscribe to my YouTube channel, and I’ll see you in the next video. Thanks for watching. If you’re interested in learning more about Git, please subscribe to my YouTube channel.

Thanks for watching. Thanks for watching. Thanks for watching. Thanks for watching.

Thanks for watching. Thanks for watching. And the uncomfortable implication is that a lot of the memory benchmark scores we’ve been quoting were measuring the half of the problem that was never hard. Here’s where it gets interesting because the same separation insight is showing up on the coding side, just framed as a system design choice instead of a negative result.

There’s a paper from Microsoft called Fast Context, and the observation behind it is something anyone who’s watched the video will not. decoding agent work has felt in most agents the same model explores the repository and solves the task so it grips around it opens files it reads things that turn out to be irrelevant and all of that exploratory junk stays in the solver’s context window polluting it costing tokens fast context pulls exploration out into its own dedicated sub-agent small models 4 billion to 30 billion parameters trained specifically to explore they’re bootstrapped from strong reference model trajectories and then refined with task grounded rewards for three things broad first turn search multi-turn evidence gathering and precise citation meaning it tells you exactly which file and which line range you invoke it on demand it fires off parallel tool calls and it hands back concise file paths and line ranges nothing else no transcript of its rummaging then they drop it into mini-SWE-agent and run it across SWE-bench multilingual SWE-bench pro and SWE-QA an end-to-end resolution goes up by as much as five and a half percent while token consumption drops by as much as sixty percent the lesson is the same one stream membench and git of thoughts are circling from the memory side exploration is a different skill from solving mixing them hertz and a small specialized model handling retrieval beats the frontier model trying to solve the problem in the same way it does in the same way it does in the same way it does trying to do it in line separate the explorer from the solver the fourth paper turns the same skepticism onto evaluation itself it’s called dialogue SWE bench and the premise is almost obvious once you say it out loud every SWE benchmark we trust scores coding agents as fully autonomous systems you handed an issue it goes away it comes back with a patch you check if the patch passes but that’s not how anybody actually uses these tools we use them interactively in a back and forth so the authors built a persona grounded user simulator a fake user with a consistent personality and goals and they score agents on resolving real software issues through dialogue with that simulated user they layer automatic dialogue quality metrics on top of the does it resolve check and the finding is the one i’d write on a sticky note better coding models do not always make better dialogue models dialogue capability is a distinct axis and it’s currently understudied they build a schema guided agent that improves dialogue over strong baselines by three to fourteen percent but the real takeaway is that the leader board you’ve been trusting for autonomous resolution tells you very little about how that same model behaves when it has to ask a clarifying question or handle a vague request two different skills and we’ve only been measuring one the fifth one is about the harness the scaffolding around the model it’s called harness x and the argument is that the runtime harness the prompts the tools the memory the control flow before again comes down to the standards of the covers i mentioned in the level 2 the variables are the ones who make the models and the hints of the sorta choose theener sondern the is itself something you should compose and evolve, not hand-build from scratch every time a new model drops. They have a set of typed harness primitives that you assemble through what they call a substitution algebra, and then a trace-driven evolution engine that watches the agent run and feeds those trajectories back into both harness updates and model training signal. So it closes the loop.

5% up to 44%, and the gains are biggest exactly where the baselines were weakest. The claim underneath is worth sitting with as a strategy question. Agent progress doesn’t have to come from a bigger model. Evolving the runtime interface from execution feedback is a separate lever, and it’s one most teams are leaving completely untouched because they treat the harness as a fixed thing they wrote once.

And then the counterweight because all of that is fairly optimistic and this last paper is not. It’s called When Errors Become Narratives, and it’s a longitudinal post-mortem of a single real agent runtime, a personal assistant system that’s been running continuously since March. Around 40 scheduled jobs, 8 LLM providers, a tool governance proxy, a knowledge-based memory plane, defended by over 4,000 unit tests and over 800 governance checks. Over eight weeks the author logged 22 incidents with full, root cause write-ups and one pattern dominated, a failure whose error signal never reaches a human in a form they can act on.

That showed up at least 28 times. The taxonomy has five classes and the one that’s unique to LLM systems is the dangerous one. The author calls it chained hallucination or fail plausible. The system doesn’t just fail silently.

The model takes a real failure and transforms it into a fluent plausible story and hands that story to the user. The observer isn’t blind. The observer is being convincingly lied to by the failure itself. Three numbers stuck with me.

About 70% of the silent failures were caught by a human looking at the output, not by any test or audit. A retrospective audit of 15 incidents found 0% were prevented ahead of time, but 87% were blocked from recurring, so audits are regression engines, not prediction engines. And the time-to-detection, anywhere from 13 hours to 60 days, tracked the failure mechanism, not how complex. The code was the longest-lived bugs lived in the seams between components where no single test runs.

So if I pull all six together, the thread is pretty clear. The right side of memory, the autonomous resolution leaderboard, the green test suite, those are the parts we know how to measure, and they are not where agents are actually failing. The hard parts are application, dialogue, the evolving harness, and the seams between components. What I’m watching now is the next wave of memory.

The benchmark starts scoring reuse instead of recall, and whether anyone builds the instrumentation to watch those seams that the production post-mortem says no test will ever cover. That’s the gap. That’s where the next year of real work is.