Agentic Memory Systems

Plain-language abstract SkillEvolBench is a benchmark designed to test whether AI agents can turn their own task experience into reusable procedural skills — not just replay what they did once, but extract general instructions a future agent can follow on related tasks. It contains 180 tasks across six real-world work domains and evaluates ten large language model configurations under three agent frameworks, comparing skill-based conditions against baselines that use no skills or raw trajectory replay.

Motivation LLM agents accumulate detailed records of how they solved past tasks, but it is unclear whether that experience can be distilled into compact, reusable procedures rather than task-specific patches. Prior work either studied skill use (given curated skills) or experience reuse (replaying trajectories), leaving a gap: can agents themselves convert noisy one-off episodes into external skill artifacts that help on harder, related tasks they have not yet seen?

Methodology The benchmark organizes 180 tasks into six environments covering code modification, API orchestration, data processing, document transformation, research synthesis, and communication operations, with five task families per environment. Each family follows a six-role arc: three acquisition tasks (canonical, enriched, variant) used to build the skill library, and three frozen deployment tasks (context-shift, adversarial, composition) where no further skill updates are allowed. Agents operate in Self-Generated or Curated-Start settings; a separate Skill Author step uses compacted trajectories and structured verifier feedback to decide whether to write a new skill, revise an existing one, or leave the library unchanged. Controls include a No-Skill condition and a Raw-Trajectory condition that provides the original episode directly.

Results Across ten model configurations and three agent harnesses, current agents adapt locally but rarely form robust reusable skills. Raw-trajectory reuse frequently outperforms distilled skills, indicating that current abstraction procedures discard contextual and procedural cues that remain useful for future tasks. The always-update curated variant showed the strongest average gain (+0.78 percentage points in frozen evaluation success rate), but benefits were model-specific: GPT-5.4 improved by +6.7 percentage points and Opus 4.5 by +4.4 percentage points, while Gemini 2.5 Pro was harmed by most variants (average -3.70 percentage points). Adding more skills or larger resource libraries did not reliably help and often introduced episode-specific drift and procedural clutter without improving deployment success.

Plain-language abstract VOYAGER is an AI agent that plays Minecraft indefinitely on its own, continuously learning new skills without any human guidance or changes to its underlying model. It uses a large language model (GPT-4) to decide what to do next, write executable code to carry out actions, and build up a growing library of reusable skills — all through text prompting alone.

Motivation Prior AI agents using reinforcement learning or imitation learning struggle to explore open-ended environments systematically, generalize across tasks, or accumulate skills over time without forgetting earlier ones. There was no agent capable of lifelong, self-directed learning in a complex open-world setting without human intervention or model retraining.

Methodology VOYAGER has three components that work together: an automatic curriculum in which GPT-4 proposes progressively harder tasks based on the agent's current inventory and world state; an iterative prompting mechanism that feeds environment feedback, execution errors, and self-verification results back to GPT-4 to refine the generated action code; and a skill library that stores successful action programs indexed by text embeddings, enabling retrieval and composition of past skills for new situations. The agent interacts with GPT-4 purely through blackbox API queries, requiring no gradient-based fine-tuning.

Results VOYAGER obtains 3.3 times more unique items, travels 2.3 times longer distances, and unlocks key tech-tree milestones up to 15.3 times faster than the prior state of the art. It also demonstrates zero-shot generalization: after learning skills in one Minecraft world, it can transfer that skill library to a new world and solve novel tasks from scratch, while baseline methods fail to generalize.

Plain-language abstract This paper introduces Agent Workflow Memory (AWM), a method that teaches language model-based web agents to extract reusable task routines — called workflows — from past experience and apply them to guide future actions. Rather than treating each task in isolation, AWM lets agents build up a growing library of skills, from simple steps like finding a place by name to more complex multi-step procedures composed from earlier workflows.

Motivation Current language model agents struggle with long, complex web navigation tasks because they process each task independently and do not learn from past successes or failures. They lack the ability to extract and reuse common task patterns across similar contexts, making them brittle when the task environment changes even slightly.

Methodology AWM operates in two modes: offline, where it extracts workflows from annotated training trajectories before test time, and online, where it induces workflows on the fly from self-generated predictions judged correct by an evaluator. The method was evaluated on two major web navigation benchmarks — Mind2Web (1000+ tasks across 200+ domains) and WebArena (execution-based evaluation covering travel, shopping, social media, and other domains) — using language model agents that maintain and grow a memory of induced workflows over time.

Results AWM improved baseline success rates by 24.6% relative on Mind2Web and 51.1% relative on WebArena, while also reducing the number of steps needed to solve WebArena tasks. It outperformed methods that use human-expert-written workflows by 7.9% on WebArena. In cross-task, cross-website, and cross-domain generalization evaluations, online AWM surpassed baselines by 8.9 to 14.0 absolute percentage points, with the performance gap growing as the train-test distribution divergence increased — reaching as high as 22.5 points after rolling over only tens of examples.

Plain-language abstract This paper introduces SEA-Eval, the first benchmark designed to evaluate AI agents that improve themselves over repeated tasks. Most current benchmarks treat each task in isolation, resetting the agent's memory each time — which makes it impossible to measure whether an agent is genuinely learning from experience or just repeatedly solving problems from scratch. SEA-Eval is built around a new category of agent called a Self-Evolving Agent (SEA), which maintains persistent memory across tasks and uses past experience to get faster and cheaper over time.

Motivation Existing AI agent benchmarks evaluate each task as an isolated episode, resetting memory at every task boundary. This design structurally prevents them from detecting cross-task learning. Two agent frameworks can achieve identical success rates while differing dramatically in whether they are genuinely evolving — one accumulating efficient strategies, the other simply re-doing zero-shot reasoning every time. Current benchmarks produce a 'capability illusion' and cannot distinguish these fundamentally different behaviors.

Methodology The authors formalize the Self-Evolving Agent concept and its core architecture — the Evolutionary Flywheel, a closed loop of execution, experience distillation, and augmented re-execution. They construct a dataset of 32 atomic tasks across three difficulty tiers, composed into three types of sequential task streams (correlated, orthogonal, and implicit intent) under two noise conditions. Two primary metrics are used: success rate (SR) and token consumption (T), where the trajectory of T across sequential task repetitions is the key signal of genuine evolution. They then empirically evaluate two leading agent frameworks, OpenClaw and GenericAgent, using this benchmark.

Results Both frameworks achieved 100% task success rate in static evaluation, but their token consumption differed by roughly sevenfold in aggregate — OpenClaw consumed 27,364K tokens across all tasks while GenericAgent consumed only 3,918K tokens, with differences reaching up to 31.2x on individual tasks. In sequential task analysis, GenericAgent showed genuine evolution: token consumption for one academic database retrieval task fell monotonically from 520K to 117K tokens. OpenClaw showed pseudo-evolution: its token consumption fluctuated without converging and collapsed on certain sequential tasks. These results confirm that success rate alone is insufficient and that sequential convergence of token consumption is the necessary criterion for detecting real learning.

Plain-language abstract This paper studies self-evolving LLM agents — AI systems whose surrounding infrastructure (prompts, skills, memories, tools) is automatically updated based on experience, while the underlying model weights stay fixed. The authors separate two distinct capabilities: how well a model produces useful updates to that infrastructure, and how well a model actually benefits from those updates when solving tasks. They find these two capabilities are largely independent of each other and of a model's general task-solving ability.

Motivation Prior evaluations of self-evolving agents measure only end-to-end performance gains, making it impossible to tell whether improvements come from better infrastructure updates or from the agent using updated infrastructure more effectively. This conflation leaves open the practical questions of which models are worth investing in as updaters versus task-solvers, and which capability bottlenecks most limit weaker models.

Methodology The authors evaluated seven LLMs spanning open-source and closed-source families across capability tiers — including Claude Opus 4.6, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-OSS-120B, Qwen3-235B, Qwen3-32B, and Qwen3.5-9B — on three agentic benchmarks: SWE-bench Verified (software engineering), MCP-Atlas (tool use over real MCP servers), and SkillsBench (skill-based execution). They systematically varied which model acted as the evolver (producing harness updates) and which acted as the task-solving agent, measuring harness-updating gain and harness-benefit gain independently.

Results Harness-updating capability is flat across model capability tiers: the best and worst evolver differ by at most 3.1 percentage points on any benchmark, and even the smallest evolver (Qwen3.5-9B) matches Claude Opus 4.6 in downstream gains. Harness-benefit is non-monotonic: mid-tier models such as GPT-OSS-120B and Qwen3-235B benefit most (up to 19.3 percentage points on SWE-bench), strong models gain little due to a ceiling effect, and weak models gain little for different reasons — they fail to invoke relevant harness artifacts (Qwen3-32B loads the harness only 25% of the time versus ~96% for strong models) or fail to follow the harness instructions faithfully over long tasks. The findings recommend allocating capability budget to the task-solving agent rather than the evolver, and treating harness invocation and long-horizon instruction following as first-class targets in agent training.

Plain-language abstract This paper introduces Skill1, a framework that trains a language model agent to simultaneously improve at three linked tasks: choosing a useful skill from a stored library, applying that skill to solve a new task, and distilling new skills from experience. Unlike prior work that handles these capabilities separately, Skill1 uses a single reinforcement learning signal to evolve all three together, building an agent that gets better at finding, using, and creating reusable strategies over time.

Motivation Language model agents trained with reinforcement learning absorb successful strategies only implicitly into their parameters and cannot explicitly reuse them across tasks. While skill-library approaches address this by storing reusable strategies, existing methods optimize skill selection, utilization, and distillation in isolation or with inconsistent reward sources, creating optimization bottlenecks where improving one stage does not benefit the others and conflicting signals prevent coherent improvement.

Methodology Skill1 trains a single policy using the GRPO reinforcement learning algorithm on the ALFWorld household-task benchmark and the WebShop online-shopping benchmark. For each task, the policy generates a natural-language query to retrieve candidate skills from the library via semantic similarity, re-ranks those candidates to select the best match, executes multi-turn environment interaction conditioned on the selected skill, and then distills a new skill from the resulting trajectory. All three stages are credited from a single binary task-outcome reward: the low-frequency moving average of outcomes over skills credits selection quality, and the deviation of each new outcome from that trend credits distillation quality.

Results Skill1 reaches 97.5% average success rate on ALFWorld, surpassing the prior best method (RetroAgent) by 2.6 percentage points and ranking first on five of six task types. It also achieves the best performance across all methods on WebShop. Compared to the strongest reinforcement-learning-only baseline (GiGPO), Skill1 improves by 6.7 points on ALFWorld. Ablation studies show that removing the skill library causes the largest single drop (to 80.9%), and removing any individual credit-assignment signal degrades all three capabilities, confirming mutual dependence among selection, utilization, and distillation.

Plain-language abstract SkillClaw is a framework that lets AI agents improve their reusable skills automatically by pooling experience across many users. Rather than each user rediscovering the same solutions independently, the system collects interaction traces from all users, identifies recurring patterns, and updates a shared skill library that every agent benefits from—without requiring any extra effort from users.

Motivation LLM-based agents like OpenClaw rely on a fixed set of reusable skills that remain static after deployment. As a result, similar workflows, tool-usage patterns, and failure modes are repeatedly rediscovered across users, and no mechanism exists to convert heterogeneous cross-user experience into reliable skill improvements. A single user rarely produces enough signal to distinguish a generalizable fix from an idiosyncratic one, so aggregating evidence across users is necessary for stable evolution.

Methodology SkillClaw operates in a closed loop: independently deployed agents record full session trajectories—capturing prompts, tool calls, intermediate feedback, and final responses—and upload them to a shared evidence base. A centralized, autonomous evolver periodically processes these trajectories to identify recurring behavioral patterns, then refines existing skills or adds new capabilities to a shared skill repository. Updated skills are synchronized back to all agents, so improvements discovered in one context propagate system-wide. Experiments were conducted on WildClawBench, a real-world agent benchmark, using Qwen3-Max as the underlying model.

Results Experiments on WildClawBench demonstrate that SkillClaw yields substantial improvements across tasks with limited interaction and feedback, highlighting the effectiveness of multi-user driven collective evolution for building continuously improving agent systems in real-world environments.

Plain-language abstract This paper presents EvoLib, a system that lets large language models get better at solving problems over time — without retraining or human-provided feedback. As the model works through a stream of tasks, it builds up a shared library of reusable knowledge extracted from its own reasoning, and that library keeps improving with each new problem solved.

Motivation Large language models normally treat every problem in isolation: any strategy discovered while solving one problem is discarded before the next. Approaches that share memory across problems typically store raw past trajectories, which are noisy and hard to generalize from. The gap EvoLib addresses is the absence of a principled way for black-box models — where weights cannot be modified — to accumulate and refine knowledge across problems without any external supervision signal.

Methodology EvoLib maintains a weighted library of two kinds of knowledge abstractions extracted from the model's own outputs: modular skills (reusable functions or sub-task workflows) and reflective insights (natural-language lessons about common mistakes). For each new task, the system samples relevant abstractions from the library via embedding-based similarity, uses them to guide solution generation, then extracts new abstractions from the resulting solution. A credit-assignment mechanism based on Information Gain and Future Information Gain updates abstraction weights to jointly reward immediate usefulness and potential to spawn better future abstractions; a consolidation step merges similar abstractions into more general ones. The whole process is self-supervised — the model evaluates its own solutions with no ground-truth labels. Experiments used GPT-4o and o4-mini on five benchmarks: HMMT math competition problems, BigCodeBench Hard and LiveCodeBench v6 Hard for code generation, and ScienceWorld and PDDL planning tasks for multi-turn agentic settings.

Results EvoLib achieved the strongest average performance across all benchmarks compared to base sampling, best-of-N, Recursive Self-Aggregation, ExpRAG, and Dynamic Cheatsheet baselines. Over base sampling, it improved accuracy by 11–20%. On BigCodeBench Hard, GPT-4o with EvoLib reached a pass rate of 40.8%, surpassing the strongest model on the public leaderboard (Gemini-Exp-1206 at 40.5%). EvoLib also proved more token-efficient than test-time scaling baselines, yielding 6–16% larger gains under half the compute budget. In continual learning experiments on PDDL with mixed task orderings, EvoLib outperformed Dynamic Cheatsheet by up to 9 percentage points, showing less sensitivity to task order.

Plain-language abstract This paper introduces MUSE-Autoskill, a framework that lets AI agents build up a personal library of reusable "skills" — chunks of code or instructions that encapsulate how to do something — and continuously improve those skills over time without human intervention. Rather than treating each skill as a one-off artifact, the system manages a full lifecycle: creating skills during task execution, storing them with accumulated experience, evaluating them with automated tests, and refining them when they fail.

Motivation Large language model agents increasingly tackle complex, multi-step tasks, but raw model reasoning alone is insufficient at scale. Existing approaches to automatic skill creation treat skills as static, isolated outputs, leaving four practical gaps: skills are created without access to the agent's runtime context, there is no per-skill memory that accumulates experience across tasks, skills are never validated or refined through unit tests, and long conversation histories cause context windows to overflow. The paper argues that skills must be managed as long-lived, testable assets rather than disposable generation outputs.

Methodology The authors built the MUSE-Autoskill agent framework around a unified five-stage skill lifecycle — creation, memory, management, evaluation, and refinement. Skills are created on demand via a built-in skill_create tool invoked from within the agent's reasoning loop. A multi-level memory system includes short-term, long-term, and a novel skill-level memory that accumulates per-skill experience across tasks. An evaluation subsystem validates skills through unit tests and runtime execution feedback, automatically triggering refinement when tests fail. A structured context manager with adaptive compression and cross-session state persistence handles long-horizon tasks. The framework was evaluated on SkillsBench, a benchmark of 51 real-world tasks graded by automated verifiers in standardized Docker environments, using three GPT-5.5-backed agents (MUSE-Autoskill, Codex, and Hermes) as comparators.

Results MUSE-Autoskill achieved the best with-skills accuracy in 3 of 4 benchmark super-domains and overall (68.40%, a +15.21 percentage point lift over its no-skills baseline), compared to Codex at 67.3% and Hermes at 61.2%. On the 35 tasks where the agent successfully generated its own skills from prior trajectories, accuracy reached 87.94%, surpassing the ceiling set by human-authored skills. Skills generated by MUSE-Autoskill also transferred to a different agent (Hermes), raising its accuracy by +10.51 percentage points and closing 79% of the gap between Hermes with no skills and Hermes with human skills.

Plain-language abstract AutoManual is a framework that lets AI agents learn to navigate unfamiliar environments on their own by building and refining a structured set of rules through hands-on interaction, then compiling those rules into a human-readable instruction manual. Given just one simple demonstration, the system achieved a 97.4% task success rate using GPT-4-turbo and 86.2% using GPT-3.5-turbo on standard household task benchmarks.

Motivation LLM-based agents typically depend on elaborate expert-written prompts and fixed in-context examples to function in specific domains, which limits their ability to adapt when conditions change. Prior methods that save successful experiences as skills suffer from a 'Path Dependence' problem — the agent blindly replicates past solutions rather than reasoning about new situations — and offline rule extraction cannot respond to distributional shift encountered during deployment.

Methodology AutoManual uses three cooperating agents in an alternating online loop. A Planner agent writes executable code plans guided by the current rule set and interacts with the environment. A Builder agent updates those rules after each episode using a structured rule management system; to reduce hallucinations, a case-conditioned prompting strategy directs the Builder to first diagnose the type of failure (unrecorded situation vs. failure to follow existing rules) before updating rules. Finally, a Formulator agent reorganizes all optimized rules into a comprehensive Markdown manual. The system was evaluated on the ALFWorld household task benchmark.

Results Starting from a single demonstration, AutoManual achieved a task success rate of 97.4% with GPT-4-turbo and 86.2% with GPT-3.5-turbo on ALFWorld benchmark tasks, substantially outperforming prior methods. The self-generated manual also improved planning by smaller LLMs, showing that knowledge accumulated during training transfers to less capable models at test time.

Plain-language abstract This paper studies whether AI coding agents can benefit from memories collected in one programming domain when working on tasks in a different domain. The authors introduce Memory Transfer Learning (MTL), which pools experiences from heterogeneous coding tasks—such as software engineering, machine learning development, and competitive programming—and makes them available to agents tackling new problems. They test this idea across six coding benchmarks and four different ways of representing memories, from raw execution traces to high-level abstract insights.

Motivation Most memory-augmented coding agents are restricted to reusing experiences from the same task domain, ignoring the fact that diverse programming problems share common infrastructure like Linux shells and programming languages. As gains from scaling training data plateau, self-evolution through accumulated experience offers a path to further improvement, but existing systems fail to exploit knowledge across domain boundaries—leaving a large pool of potentially transferable experience untapped.

Methodology The researchers evaluated MTL across six coding benchmarks spanning software engineering (SWE-level), machine learning, and competitive coding tasks. They used four memory representations of varying abstraction: concrete execution trajectories, code snippets, experiential planning and debugging traces, and high-level abstract insights. A unified memory pool drawn from all heterogeneous domains was made available during retrieval, and performance was compared against single-domain memory baselines.

Results Cross-domain memory transfer improved average performance by 3.7% across the six benchmarks. The primary source of benefit was meta-knowledge—operational know-how such as validation routines and structural inspection strategies—rather than task-specific code. The paper found that abstraction level determines transferability: high-level abstract insights generalize well across domains, while low-level execution traces often cause negative transfer by introducing excessive task-specific detail. Performance gains also scaled with the size of the memory pool and the number of contributing domains, and memory was shown to transfer effectively even between different underlying models.

Plain-language abstract SEARL is a reinforcement learning framework that trains AI agents to simultaneously build up a library of reusable tools and improve their own decision-making policy. Rather than relying on a fixed set of pre-defined tools or simply accumulating raw past experiences, the agent continuously creates, stores, and reuses modular tools organized in a graph structure, growing smarter as it solves more problems.

Motivation Large language models still struggle with complex, multi-step tasks when prompted directly, and existing approaches either use static tool libraries that limit adaptability or rely on large-scale models and multi-agent systems that are expensive to deploy. Reward signals in agentic reinforcement learning are also typically sparse—given only at task completion—making it hard to assign credit to individual steps. There was no framework that jointly improved both the agent's policy and its external tool memory together.

Methodology The authors formalize agent training as a Tool Memory-Enhanced Markov Decision Process, where the agent operates over a dynamic directed graph of tools (nodes) connected by execution dependencies (edges). At each step the agent retrieves relevant tools from the graph, decides whether to reuse or create a new one, and executes actions. Training uses a composite reward combining a sparse binary outcome reward with dense process-level rewards for planning, tool creation, and tool execution. Credit assignment is handled through a two-level advantage scheme: episode-level advantages across full trajectories, and step-level advantages anchored to specific tools, grouping actions that used the same tool across different trajectories. Experiments use 10,000 open-source RL training samples and evaluate on mathematical reasoning (AIME2024, MATH500, GSM8K) and multi-hop knowledge QA benchmarks (HotpotQA, 2WikiMultihopQA, Musique, Bamboogle) against baselines including GRPO, DAPO, REINFORCE++, and ARPO.

Results SEARL achieves the best average rank (1.43) across all benchmarks compared to baselines. It matches the top-performing method on AIME2024 (0.3333) while consistently outperforming or matching baselines on multi-hop QA tasks; on Bamboogle it reaches 0.3040 versus 0.2480 for the next best. The Tool Graph Memory grows from small disconnected subgraphs into interconnected clusters spanning multiple domains as training progresses, with the agent demonstrating higher training rewards and sustained exploration entropy compared to the GRPO baseline throughout training.

Plain-language abstract This paper is a survey of "agent skills" — reusable, stored procedures that tell AI agents not just what tools exist, but how, when, and in what sequence to use them when completing complex tasks. The authors review the research literature on how these skills are represented, learned, retrieved, and updated, and provide a taxonomy and resource collection for the growing field of skill-centric AI agent design.

Motivation Large language model agents can access many tools via APIs and protocols, but simply having access to tools does not produce reliable behavior: agents must still figure out how to coordinate multiple tools, handle failures, and validate outputs from scratch on every task. This "procedural gap" makes agents brittle, slow, and hard to maintain as tasks grow more complex. The survey argues that reusable procedural artifacts — skills — are the missing layer between raw tool access and robust task execution.

Methodology The authors systematically organize existing research around four lifecycle stages of agent skills: representation (how skills are formally encoded), acquisition (how skills are learned from human demonstrations, execution traces, or task-conditioned generation), retrieval (how relevant skills are selected at runtime), and evolution (how skills are updated and governed over time). Representative methods, platforms, benchmarks, and application domains are reviewed within each stage, drawing on work published from approximately April 2023 through April 2026.

Results The survey identifies three main skill acquisition strategies — human-derived, experience-derived, and task-derived — and documents that systems such as Voyager, Reflexion, ExpeL, and Agent Workflow Memory illustrate a spectrum from selecting successful traces, to abstracting lessons, to packaging reusable executable workflows. Eight application domains are covered, including software engineering, web and GUI automation, robotics, and social simulation. Open challenges identified include weak standards for skill schemas and interoperability, lack of lifecycle robustness against API deprecation, and evaluation methods that cannot distinguish genuine skill improvements from stronger test-time compute or task synthesis.

Plain-language abstract SkillDroid is an Android automation system that learns from a task the first time an AI completes it, then replays that learned sequence on later requests without involving the AI again. Instead of reasoning through every step of a repeated task from scratch, it stores a reusable action template and replays it at machine speed, falling back to AI assistance only when something unexpected occurs.

Motivation Current AI-based mobile agents treat every task execution as an independent reasoning episode, calling the language model at each action step. This means a task completed successfully yesterday is re-derived entirely from scratch today, with no gain in speed or reliability. LLM calls account for 75-94% of total agent execution time, and in testing a stateless agent's success rate degraded from 80% to 44% over 150 rounds as instruction phrasing became more varied, because past successes provide no benefit to future attempts.

Methodology SkillDroid uses a three-layer architecture running on top of an open-source Android automation framework. In Layer 1, an LLM guides task execution step by step and, on success, a skill compiler extracts a parameterized template — a sequence of UI actions with weighted element locators and typed parameter slots — stored in a local SQLite database. In Layer 2, a matching cascade using regex patterns and embedding-based semantic similarity routes new instructions to stored skills for replay via Android's accessibility interface with zero LLM calls, using a state verifier to detect UI deviations and gracefully fall back when needed. In Layer 3, a failure-learning component tracks skill reliability and triggers recompilation when a skill's failure rate exceeds 50%, progressively replacing noisy initial compilations with cleaner versions.

Results Over a 150-round longitudinal evaluation spanning 15 task types across diverse Android applications with systematic instruction variation and controlled perturbations, SkillDroid achieved an 85.3% success rate — 23 percentage points above a stateless LLM baseline — while using 49% fewer LLM calls. The skill replay mechanism achieved a perfect 100% success rate across 79 replay rounds at 2.4 times the speed of full LLM execution. Critically, the system's success rate converged upward from 87% to 91% across experimental phases, while the baseline degraded from 80% to 44%, demonstrating that SkillDroid improves with use while stateless agents become less reliable as task variation grows.

Foundational long-term dialogue benchmark (pre-wave).

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Early memory-management benchmark for long conversations.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Neuroscience anchor: active forgetting pathway.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Neuroscience anchor: active forgetting.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Neuroscience anchor: consolidation.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Adversarial false-memory formation in continual learning.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Plain-language abstract This paper deploys a production LLM agent that, instead of writing fresh code for every request, compiles the repeated steps of a standard operating procedure (SOP) into validated, versioned tools ahead of time; at runtime the agent calls those tools directly and falls back to code generation only when a tool is unavailable or fails. It triages alarms in an Amazon fulfillment-center outbound dock against a 44-node SOP over heterogeneous metric backends.

Motivation In the prevailing CodeAct-style paradigm the agent generates and executes fresh code for each request, so when the same workflow repeats against a stable backend it re-interprets the same instruction, rediscovers the same schema, and regenerates similar code, raising latency, cost, and run-to-run variance. In this setting most latency and correctness errors come from translating underspecified SOP text into a concrete query against a production metric backend, which motivates self-evolving agents that build reusable tools before they are needed in production.

Methodology Offline, each SOP node is compiled into a tool: a data-collector sub-agent runs against the live environment via MCP, producing a trace of query code, backend responses, observed schema (field names, datatypes, value ranges), and a graded verdict; a tool-maker LLM writes a candidate tool conditioned on the SOP node text, the node's position in the decision tree, and that trace; and a reflector-tool-maker loop repairs the candidate against labeled cases. Online, the production agent invokes these compiled tools inline, one call per node, falling back to code generation only when a tool is unavailable or fails.

Results In production, tool calls reduce p50 latency by 42%; on 1,500 historical alarms they reduce end-to-end error rate by up to 53% by suppressing run-to-run variance in repeated steps. Because tools return compact structured verdicts, a simpler direct-call architecture reduces p50 latency by a further 62% in a controlled ablation. Both the data-collection trace and the test-repair loop are necessary, residual errors concentrate on underspecified SOP steps where targeted fixes raise pass@1 from 94.5% to 99.9%, and versioned tools improve auditability while exposing specification gaps and upstream data drift.

Plain-language abstract Community SKILL.md files package reusable procedural knowledge for LLM agents, but the public ecosystem is fragmented, redundant, and uneven in quality. SkillCorpus crawls roughly 821,000 skill files from 62 public sources, curates them through a six-stage pipeline into 96,401 skills organised by a 16-class taxonomy and three quality facets (utility, robustness, safety), and pairs the corpus with a fine-tuned retrieval-and-selection stack. Plugged into two agent harnesses across three real-task benchmarks, the corpus consistently raises pass rates, and the paper traces exactly where the gains stop.

Motivation Whether the abundant open skill ecosystem improves agents on real tasks was unsettled: hand-picked oracle skills help heterogeneously, while uncurated community libraries can fail to beat a no-skill baseline. No released corpus was simultaneously broadly aggregated, strictly quality-gated, and licence-cleared, and no prior evaluation served a deployed community corpus through retrieval on real benchmarks.

Methodology A registry of 62 sources across five ingestion mechanisms feeds a six-stage funnel: structural parse and length filters; two-tier deduplication in which exact fingerprints collapse 59.7% of stage input and an LLM judge adjudicates 66,751 borderline semantic pairs; an LLM judge emitting utility, robustness, and safety subscores plus 19 quality flags, five of which (prompt injection, command injection, unsafe execution, auth bypass, CSAM risk) force exclusion; a safety hard-gate plus OSI-permissive licence filter; and indexing with 1024-dimension retrieval embeddings. A fine-tuned recall-and-rerank stack with an LLM selector that reads full skill bodies matches skills to incoming tasks. Evaluation spans SkillsBench, GDPVal, and QwenClawBench on two harnesses (OpenClaw, Raven) and two open backbones, with a frontier robustness check.

Results Integration yields consistent gains on all three benchmarks, largest on SkillsBench (+7.5 pp mean; +13.4 pp on the strongest Raven cell over a 9.2% no-skill baseline; +8.0 pp for the frontier check). Lesion ablations show curation and retrieval each contribute: swapping the filtered corpus for the raw crawl drops the strongest cell from 22.6% to 14.9% pass@1, and swapping the fine-tuned stack for off-the-shelf retrieval drops it to 13.8%. The gains are bounded twice over. Coverage: mean per-task gain climbs from +2.2 to +25.1 pp across retrieval-match bins, and thin-coverage categories floor at zero rather than turning negative. Harness: on tasks Raven passes and OpenClaw fails, traces show OpenClaw writes scripts it never executes, while Raven completes an execute-verify-fix loop, so skill content converts into verifier passes mainly through such a loop.

Plain-language abstract Agent Skills package procedural instructions for coding agents, but a loaded skill's requirements do not all stay active across a long tool-using trajectory. In a white-box code-audit workflow with 24 fixed artifact checks, the same task passes 8/10 runs in a clean 11K-character context and only 3/10 in 299K-character contexts - whether the added material is relevant production text or irrelevant archive. Requirement coverage stays above 92% even in failing runs: agents lose one or two obligations, not the task. An external checklist restating all 24 checks passes 10/10 runs where a generic self-check passes 5/10.

Motivation Long-context research shows a model's usable context is smaller and less uniform than its advertised window, and trajectory studies show failures begin early and hide behind plausible final states. The missing piece is a focused test: hold a skill-based task and its checks fixed, vary only the surrounding context, and identify exactly which stated requirement fails - and whether a simple external check prevents the failure.

Methodology Mandatory instructions from a production-derived audit workflow (an industrial code-audit scanner's Stage 4CD task) are translated into 24 deterministic, equally weighted checks, each validated against positive and negative fixtures before any model runs. Every attempt starts from a sanitized, answer-free workspace; conditions add clean (10,991 chars), relevant-long (299,140 chars of same-workflow production material), or irrelevant-long (equal-length unrelated archive) context. Ten valid runs per condition with Codex + gpt-5.4-mini; failed runs are classified by first visible failure location (lost requirement, editing drift, failed checking, non-agent failure). A mitigation arm compares a generic self-validation prompt against a detailed checklist differing by only 649 characters; scaffold probes compare direct prompting, a retrospectively selected evidence bundle, and tool-using agent shells.

Results Clean passes 8/10; both long conditions pass 3/10 - a 50-point observed drop that remains trend-level (two-sided Fisher p=0.0698) - with no ordering between relevant and irrelevant context (p=1.0). Check coverage stays at 92-94% in the long conditions: typical failures omit one required array or exceed a field cardinality, though one run abandoned two of four audit tasks entirely. The detailed checklist passes 10/10 versus 5/10 for the generic self-check (p=0.0325); the generic validator's spot-checks omit the same field generation omitted. A second task passes every clean and long run, so the evidence supports no universal context-length threshold. Infrastructure is a separate failure source: only 29 of 57 extension attempts produced scoreable output, with first-token disconnects clustered in the irrelevant-long condition. For gpt-5.5, a small evidence bundle passes 3/3 without tools while the full 299K direct prompt passes 1/3, suggesting selective retrieval - not tools per se - explains coding-agent benefit.

Plain-language abstract Procedural skills for LLM agents are normally judged by average improvement in task success, which hides the tasks a skill library breaks. Running the same agent with and without skills across 5,832 paired task-condition runs on two office-automation benchmarks and three model-harness stacks, the paper decomposes each pass-rate change into gains and regressions. Regressions offset 59% of gross gains, and the best libraries lead mainly by regressing less rather than gaining more. Three regression mechanisms are identified from traces - skill-description osmosis, grounding displacement and verification displacement - and the residual failures cluster at grounding and verification, the stages existing skills least often address.

Motivation Systems that generate skills automatically from execution traces are evaluated by average gain, so two libraries with identical averages can differ sharply in how many working tasks they destroy. Methods that detect and drop harmful skills identify which skill to remove but not why it regressed a task, so a mostly useful skill may be discarded wholesale. Those methods also act only when a skill is retrieved or invoked, while a skill's name and description remain in the system prompt at every step and can change behavior without ever being called - a channel selection-based approaches cannot observe.

Methodology Two benchmarks over real artifacts: OfficeQA-Pro (question answering over U.S. Treasury financial documents, where the hard part is finding the right table, vintage and definition rather than the arithmetic) and SpreadsheetBench (workbook manipulation from real Excel-forum problems, graded on exact cell values). Three stacks pair a harness with its usual model: OpenCode / minimax-m2.7, Codex / gpt-5.4-mini, Claude Code / sonnet-4.6. Each stack and benchmark runs under four conditions - none, plus three libraries built by different meta-skill creators from the same analyst-extracted failure signals, so only the library text changes within a stack. Every task falls into gain, regression, residual failure or retained against the no-skill baseline; net effect is gains minus regressions, tested with the exact two-sided McNemar test on discordant pairs with Newcombe 95% intervals. Skill invocation is read from the trajectory so presence-only influence can be separated from body engagement. Each condition is run once per task, so run-to-run variance is not estimated.

Results Regression counts range from 2 to 41 across the eighteen library conditions and never reach zero; 324 regression transitions cancel 59% of 553 gains, with the pattern holding on both benchmarks (66% of gains cancelled on OfficeQA-Pro, 56% on SpreadsheetBench). Even clearly positive cells are close-run: on SpreadsheetBench the openai library broke 41 while gaining 45 on one stack and broke 27 while gaining 30 on another. Five of eighteen conditions reach nominal p < .05, but only three survive Bonferroni correction, all on Claude Code / sonnet-4.6 with SpreadsheetBench - a real but narrow effect. Grounding displacement accounts for 72.8% of the 81 coded OfficeQA-Pro regressions and osmosis for 17.3%; on SpreadsheetBench osmosis accounts for 70 of 243 regressions against 46 with the body engaged and 32 attributable to a grader artifact. Body engagement is set by the harness rather than the skill: the same library is invoked on 73-91% of OfficeQA-Pro tasks but 5-14% of SpreadsheetBench tasks on one stack, and on 99-100% versus 0-1% on another. Correcting the known grader artifact required re-grading 226 treatment task-conditions with a full spreadsheet engine, since the default engine cannot recalculate Excel structured references and scores correct formulas as failures.

Plain-language abstract Enterprises still run large COBOL systems on mainframes, and moving them to Java means proving the new program behaves like the old one. Testing that is hard because real test data is scarce and the corner cases are the point, including behavior that never matched the original product definition but became part of it over the years. This paper describes the Locksmith Loop, used at American Express. The COBOL source and its generated Java target are each instrumented with mocks and run off the mainframe on ordinary hardware, and a deterministic gate runs both on the same inputs and flags any divergence. A search phase drives inputs to reach as many branches as it can; when it stalls, an AI agent proposes a code mutation applied identically to both programs to open a blocked region, and the gate decides whether the mutation is kept. Successful mutations are saved as reusable skills. Across three programs the loop pushes coverage well past the plateau that input search alone reaches, and the Java matched the COBOL reference on every accepted test case.

Motivation Agentic coding systems make exploring and migrating legacy systems increasingly plausible, but evaluating parity between the legacy and modern variants remains hard, and organizations typically run both systems under strict evaluation until they have enough confidence to cut traffic over. The blocker is test data: only a limited number of real-world scenarios are usually covered, leaving a long tail of corner cases untested, and broken-as-usual behavior that deviates from the original specification has to be preserved anyway in lift-and-shift modernization. The authors observed empirically that established input-space techniques, including combinatorial interaction testing, adaptive random testing, search-based test generation and many-objective variants, fuzzing, and quality-diversity algorithms such as MAP-Elites, all converge to similar coverage ceilings, which indicated that the remaining branches were structurally unreachable without modifying state outside the input-search space.

Methodology The COBOL source passes through a deterministic Migrator to produce the Java target and through a deterministic Mock Generator to produce a runnable COBOL mock; a shared test harness exercises both, records a behavioral fingerprint, and enforces the Parity Gate. The loop follows a teacher-student pattern. The student is deterministic: Migrator, Analyzer, Runner and Parity Gate. Witness Search explores the input space, including mocked backend responses and environmental state, running six algorithms per sweep and taking the union of discovered branches with the test-case set, each sweep starting from the best previous set until two consecutive sweeps yield nothing new, which defines the plateau. At the plateau the Analyzer identifies a Locked Paragraph, a COBOL paragraph unreachable by input search alone. The teacher, an Authoring Layer agent, is then invoked to select or propose a Mutation Skill; the student applies the mutation symmetrically to both the COBOL mock and the Java target, and post-mutation Witness Search using UCB1 confirms the coverage gain. A mutation is retained only if it increases branch coverage and passes the Parity Gate, which compares end states across three equivalence axes; successful mutations are captured into the Skill Catalog. Control then returns to a full sweep, and the process recurses until no Locked Paragraph remains.

Results Three case studies span two open-source COBOL programs and one internal production-like program, ranging from 430 to 4,114 source lines. Locksmith consistently improved coverage beyond the input-search plateau, reaching nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like program. On the production-shape run the loop covered 135 of 142 paragraphs (95.1%) and 101 of 146 static paragraph-to-paragraph transitions (69.2%) from 166 test cases, with Witness Search accounting for 54 transitions, agentic mutation for a further 47, and 45 left uncovered. Under deterministic parity checks the generated Java matched the COBOL reference on every accepted test case. The authors flag branch coverage as an imperfect proxy for fault detection, note that acceptance is anchored on cross-language parity rather than coverage, and decline to claim the resulting suite suffices for migration sign-off by itself.

Plain-language abstract A survey of coding agents that update their own framework, memory, skills, tools, models, or workflow structure from prior coding attempts, organized by what evolves, when it evolves, and what software-specific evidence drives it.

Motivation Coding agents now operate as interactive systems inside real development workflows, interpreting requirements, inspecting repositories, editing multiple files, running tests, diagnosing failures and generating patches. But in most systems the base model, prompts, tool interfaces, memory mechanisms and control flow are fixed after deployment, which is hard to sustain when codebases evolve, APIs and dependencies change, conventions vary across repositories, and bug fixing requires repeated cycles of localization, patching, execution and revision. Meanwhile software engineering supplies unusually concrete feedback: unit tests, compiler errors, runtime traces, lint warnings, CI results and human code reviews. An agent that cannot accumulate experience from that feedback repeats its mistakes across tasks and never adapts to project-specific context. Existing work on self-evolving agents mostly targets broad task environments and leaves the software-specific requirements uncharacterized.

Methodology The survey is framed as a guiding synthesis rather than a review of a settled paradigm, organized around three questions: what components evolve and through what mechanisms, when evolution occurs and what evidence drives it, and how such agents should be evaluated beyond the setting they evolved in. It first separates coding agents, self-evolving agents and self-evolving coding agents conceptually, then builds an object-centered taxonomy with five categories: agent framework self-evolution (self-modifying scaffolds and archive-based variant search), memory self-evolution (experience, repository and plan-abstraction memory), skill and tool self-evolution, model self-evolution (adversarial test evolution, coder-verifier co-evolution, self-play policy evolution), and workflow and topology self-evolution. Two orthogonal dimensions cross it: timing (task-time, post-task, stage-wise) and evidence type (outcome, environmental, trajectory-derived). Representative systems are tabulated against all three dimensions plus their software-engineering task domain. Inclusion is limited to work whose central contribution changes an agent component or behavior through coding-specific feedback; benchmark-only datasets and static agent systems are treated as evaluation context rather than as methods.

Results The synthesis finds that executable feedback, repository-level context and coding trajectories give software engineering a distinctive role as a domain for agent self-evolution, because a remembered strategy can be validated by tests rather than merely accumulated, and repository history supplies long-term signal about how a project changes. Memory self-evolution clusters into experience banks over prior issue-resolution trajectories including failures (SWE-Exp), hierarchical pools separating general from repository-specific experience (EvoCoder), subtask-granularity stores that avoid coarse whole-trajectory matching, commit-and-issue-grounded repository memory for code localization, and plan abstraction reused as later context (SAGE). Skill and tool self-evolution is distinguished by actionability: CODESKILL maintains a skill bank at task and event granularity and treats skill management as a learnable policy; gskill learns per-repository skill documents validated by whether agents in isolated environments produce passing patches; Socratic-SWE distills a skill registry from traces that then generates the training curriculum. Tool self-evolution is the least developed, with Live-SWE-Agent the main example of an agent creating and revising its own tools from a bash-only scaffold. The named challenges are feedback reliability, benchmark overfitting, safety, maintainability, cost and generalization. Two are emphasized: memory must be evolved selectively, since noisy logs, misleading tests, brittle patches and repository-specific conventions produce memories that hurt when retrieved uncritically; and framework-level evolution carries the strongest risk, because a harmful modification to the machinery that generates future actions can break the agent loop, degrade tool use, overfit to benchmark feedback or exploit weaknesses in the evaluation harness, so it requires validation, rollback and robustness checks rather than performance-driven search alone.

Plain-language abstract A benchmark and method for a specific skill-retrieval failure: the retriever finds the right capability family but surfaces the wrong member of it, one that shares the vocabulary and procedure shape while pointing at a stale resource, skipping a precondition, or applying the wrong check. The benchmark pairs each query's helpful skill with that query-specific risky sibling inside a large public candidate pool and scores helpful ranking alongside a harmful sibling rate. The method resolves candidates into capability families, scores query-conditioned utility, and exposes one representative per family before the final top-K list.

Motivation Skills have become loadable operational artifacts carrying instructions, scripts, resources and metadata, so the retrieval layer decides which procedural context enters an agent before planning or execution. That makes retrieval failures more specific than broad irrelevance. Public skill audits find weak routing metadata, non-actionable bodies, reusable-artifact defects and technical debt, and gains from curated skills weaken when agents must retrieve from large real collections. Related work has shown top-K skill quality cannot be reduced to independent query-skill relevance because the retrieved set must be compatible as a set, and security work has established that loaded skill packages affect planning, context, permissions, scripts and local resources. Existing positive-skill retrieval benchmarks identify useful or gold skills but assign no query-specific execution-risk siblings, while malicious-skill detection and permission enforcement act after admission rather than at the retrieval-time choice of representative.

Methodology The setting is formalized as same-capability execution-risk retrieval: a collection of queries, each with a candidate pool, an admitted helpful skill, a query-specific execution-risk sibling, and a released family relation. Construction starts from a task-facing skill admitted for a query, and the paired sibling changes exactly the condition that makes the procedure usable, keeping the sibling plausible under ordinary semantic retrieval. Candidate pools add library pressure by ranking the pair among unrelated and partially related public skills drawn from a public SkillRet corpus, so a system must both recover active capability families and choose the right representative within each. Evaluation reports Recall@K and NDCG@K for the helpful skill together with HSR@K, the rate at which the risky sibling appears in the final top-K. The release records source role and admission evidence, risk taxonomy, cue and leakage checks, hashes, query-disjoint splits and held-out outputs. The method, SkillResolve, has three components: a Capability Resolver returning active candidate groups that should compete as alternative representatives (singleton groups allowed, which recovers ordinary ranking); a query-conditioned Utility Scorer trained with admitted helpful skills as positives and confusable library alternatives mined under the same query and pool protocol as negatives, using ordinary retrieval signals plus contract-profile cues; and a Representative Selector keeping the highest-utility member of each resolved group before final ranking. Family sources can also be derived from public metadata or skill text, which trades exposure against recall.

Results Same-capability ambiguity is prevalent in public libraries: 4,716 of 4,997 audited SkillRet queries have at least one non-gold skill in the gold skill's domain/action/object family, and across seven standard retrievers a top-three list surfaces such a sibling for 47.3% of queries, with 36.6% containing both the gold skill and a strict same-family sibling. Generic lexical retrieval, SkillRouter and BGE reranking retrieve relevant skills but expose risky siblings, while an attribution-listwise baseline suppresses them at the cost of helpful retrieval quality. SkillResolve reaches Recall@3 0.766, NDCG@3 0.699 and HSR@3 of 0 under the released family relation, improving over SkillRouter by 0.112 Recall@3 and 0.165 NDCG@3 while reducing HSR@3 from 0.693. Component analysis identifies representative selection as the controlling mechanism: with the same utility scorer but no representative selection, HSR@3 rises to 0.236. The recall-exposure tradeoff depends on the quality of the family source, since a resolver that splits a helpful skill and its risky sibling into different groups lets both survive into the final ranking. Note on versions: SciX holds the June 2026 v1 of this arXiv entry, titled SkillResolve-Bench with 661 helpful/risky pairs and a 7,982-candidate pool; a later revision circulates as SameCapRisk-Bench with a larger unit count and a different baseline table. The mechanism and the metric are unchanged across both; the figures above are the v1 body.

Plain-language abstract Agent skills are prose procedure files loaded into a model's context and interpreted by a tool-calling loop, which means nothing in the runtime makes a mandated step happen. Measured against the procedures their own skills prescribe, a prose agent performs 56% of mandated steps on gpt-4o while still producing deliverables that pass output checks. SIGIL compiles a SKILL.md into an executable harness through a typed intermediate representation that marks each step as owned by code or by the model, raising step compliance to 86%, full-procedure completion from 28% to 65% of runs, and cutting median token cost to 0.58x.

Motivation Skills succeeded as an authoring format because domain experts can write, review and version them, but the property that makes them accessible leaves them unenforced. The document is consulted by the same process that is supposed to obey it, so an agent can satisfy its own reading of an instruction by writing a sentence asserting the outcome. Hand-writing a harness fixes enforcement at the cost of the authoring surface, and leaves two independent representations of one procedure free to drift. The resulting defect class is invisible to artifact-level testing: a financial-compliance report can carry reasonable figures and conclusions and still be invalid if the agent never retrieved the authoritative records, recalculated the totals, or checked the result against policy. Prior runtime supervision work adds monitors and guards that can block a disallowed action, but detection does not make a required action occur.

Methodology Compilation runs in two stages over AG-IR, a typed agentic IR whose nodes carry an owner stating whether code or the model executes the step, a modality recording how binding the source instruction was, and provenance to the mandating sentence. Extraction is the only stage containing a model and admits a rule only when a verbatim quotation supports it; six compile gates then reject an IR that points at content instead of embodying it, omits a mandated deliverable, fails to lower and type check, folds a mandatory rule into another slot's interior, or leaves a human-gated decision to model judgment. Lowering is deterministic, mapping each primitive onto an Object-Spatial Programming construct in Jac: nodes become node archetypes, edges fix walker traversal order, mandatory rules become abilities bound to node entry, model-owned nodes become typed llm() slots whose return type makes an out-of-range answer a type error, and forbidden rules lower to nothing, so the guarantee comes from the absence of a path. STRUCT-COV statically reports, per mandatory rule, whether the emitted module realizes it as a code gate, a dedicated model slot, monolithically inside another slot, or not at all, failing the last two. Evaluation covers 30 skills across document and tooling, software process, and governance and compliance families, with three process-scenario tasks per skill repeated three times per arm in isolated sandboxes, on gpt-4o and gpt-5, scored by Applicable-Mandate Compliance with a judge reading the full run and, for the harness, its node-path trace.

Results On gpt-4o the compiled harness performs 86% of mandated steps against 56% for prose, matching or exceeding prose on all 30 skills with 28 strict wins, 2 ties and no losses, and completing the full applicable procedure in 65% of runs against 28%, a factor of 2.3. Moving to gpt-5 leaves the harness at 86% while prose rises to 68%, so the advantage narrows from 30 points to 17; the harness artifact is byte-identical across both sweeps, which is what makes the guarantee model-independent by construction. Median token cost is 0.58x prose, cheaper on 24 skills and as low as 0.02x on a nearly all-code skill, with the exceptions concentrated in skills organized around a genuine tool-using loop, reaching 7.64x on using-superpowers and 3.09x on systematic-debugging, where the prose agent was cheaper because it truncated or skipped the loop. The compiler realizes 37% of mandates as code-owned gates on average, 32% to 44% by family, and the four skills where the harness loses on gpt-5 are judgment-heavy ones with little to compile.

Plain-language abstract Agent Skills package procedural knowledge as markdown plus scripts, and that format describes how an agent should behave without deciding which behavior may become an action. As Skills move to settings that drive relays and locks, the gap becomes a physical one. The paper names Borrowed Authority: an inter-agent message carrying an instruction plus an unverifiable permission claim, which the receiving agent has no typed way to reject. Edge Skillguard answers it with a typed authority layer inside the Skill artifact - guard predicates over world state, leases and sensor evidence, expressed as a schema-validated policy file with a pure-function evaluator. On a live edge control plane it rejects 60/60 attacks across five variants while preserving all benign requests, holding at 5x scale and across hosts.

Motivation Self-evolving Skill harnesses generate orchestration automatically and report efficiency gains, not safety ones, so generating more Skills scales the advisory layer and leaves the authority decision to the model. Two adjacent attacks are already public: malicious skills distributed through community registries, plus a Claude Code project-file misconfiguration (CVE-2026-21852) that routed a session's API tokens to an attacker before trust was established, and jailbreaks of LLM-controlled robots reaching up to 100% success against deployed commercial platforms including a self-driving LLM, a wheeled UGV and a quadruped. Their intersection - a compromised skill artifact causing physical-state harm - is the open cell of the paper's attack table, and the authors construct and defend it before an in-the-wild incident rather than after.

Methodology An Edge Skillguard artifact is a tuple of orchestration states, typed world state (sensors, user identity, leases, device state, time, shared-state commit), typed envelope events, deterministic actions, guard predicates over state and event, a transition relation, bounded LLM holes, and inter-agent contracts incoming messages must satisfy. Guards are seven typed operators over dot-separated paths into envelope or state, shipped as a JSON-Schema-validated policy plus a pure-function evaluator that emits either an inbox publish or a structured policy_block log naming the failed predicates. It sits on a messaging substrate providing schema-validated typed envelopes, broker-attested sender_id from the connection token, durable per-agent FIFO inboxes, an audit-mirror outbox, and boundary rejection of malformed envelopes. Four conditions are compared - plain Skill, natural-language machine-to-machine with no receiver guard, a lease-only ablation, and the full policy - over 60 Borrowed Authority requests across five variants (stale presence, missing presence source, wrong-grantee lease, expired lease, lease-scope mismatch), 12 each, plus 60 benign requests, run in-process, on a live NATS broker against a Home Assistant deployment of 148 entities, and cross-host over Tailscale. Test subjects are isolated so no device adapter fires; the measurement is whether an unauthorized transition reaches the adapter boundary.

Results The full typed policy rejects 60/60 Borrowed Authority requests and preserves 60/60 benign ones, versus 60/60 wrongful actuation for both the plain Skill and the natural-language machine-to-machine baselines. The 5x run holds 300/300 attack rejection and 300/300 benign success at p95 399 microseconds on the live broker, and the cross-host Tailscale run holds the same correctness at p95 7.9 ms. Median latency is 3.2 microseconds in-process, 273 microseconds on the local broker, 5.7 ms over the mesh, with 0 LLM calls per decision against 1 for the advisory baselines. The lease-only ablation catches 36/60, covering the three lease-bound variants and letting both sensor-bound variants through, so freshness and source predicates carry the remainder. Blocked requests surface the failed predicates rather than a model rationale. The authors do not claim automated policy synthesis; hand-authored guards are the supported workflow, and the attack class is constructed rather than observed in the wild.

Plain-language abstract This paper introduces Agent Workflow Memory (AWM), a method that teaches language model-based web agents to extract reusable task routines — called workflows — from past experience and apply them to guide future actions. Rather than treating each task in isolation, AWM lets agents build up a growing library of skills, from simple steps like finding a place by name to more complex multi-step procedures composed from earlier workflows.

Motivation Current language model agents struggle with long, complex web navigation tasks because they process each task independently and do not learn from past successes or failures. They lack the ability to extract and reuse common task patterns across similar contexts, making them brittle when the task environment changes even slightly.

Methodology AWM operates in two modes: offline, where it extracts workflows from annotated training trajectories before test time, and online, where it induces workflows on the fly from self-generated predictions judged correct by an evaluator. The method was evaluated on two major web navigation benchmarks — Mind2Web (1000+ tasks across 200+ domains) and WebArena (execution-based evaluation covering travel, shopping, social media, and other domains) — using language model agents that maintain and grow a memory of induced workflows over time.

Results AWM improved baseline success rates by 24.6% relative on Mind2Web and 51.1% relative on WebArena, while also reducing the number of steps needed to solve WebArena tasks. It outperformed methods that use human-expert-written workflows by 7.9% on WebArena. In cross-task, cross-website, and cross-domain generalization evaluations, online AWM surpassed baselines by 8.9 to 14.0 absolute percentage points, with the performance gap growing as the train-test distribution divergence increased — reaching as high as 22.5 points after rolling over only tens of examples.

Plain-language abstract SkillClaw is a framework that lets AI agents improve their reusable skills automatically by pooling experience across many users. Rather than each user rediscovering the same solutions independently, the system collects interaction traces from all users, identifies recurring patterns, and updates a shared skill library that every agent benefits from—without requiring any extra effort from users.

Motivation LLM-based agents like OpenClaw rely on a fixed set of reusable skills that remain static after deployment. As a result, similar workflows, tool-usage patterns, and failure modes are repeatedly rediscovered across users, and no mechanism exists to convert heterogeneous cross-user experience into reliable skill improvements. A single user rarely produces enough signal to distinguish a generalizable fix from an idiosyncratic one, so aggregating evidence across users is necessary for stable evolution.

Methodology SkillClaw operates in a closed loop: independently deployed agents record full session trajectories—capturing prompts, tool calls, intermediate feedback, and final responses—and upload them to a shared evidence base. A centralized, autonomous evolver periodically processes these trajectories to identify recurring behavioral patterns, then refines existing skills or adds new capabilities to a shared skill repository. Updated skills are synchronized back to all agents, so improvements discovered in one context propagate system-wide. Experiments were conducted on WildClawBench, a real-world agent benchmark, using Qwen3-Max as the underlying model.

Results Experiments on WildClawBench demonstrate that SkillClaw yields substantial improvements across tasks with limited interaction and feedback, highlighting the effectiveness of multi-user driven collective evolution for building continuously improving agent systems in real-world environments.

Plain-language abstract SEARL is a reinforcement learning framework that trains AI agents to simultaneously build up a library of reusable tools and improve their own decision-making policy. Rather than relying on a fixed set of pre-defined tools or simply accumulating raw past experiences, the agent continuously creates, stores, and reuses modular tools organized in a graph structure, growing smarter as it solves more problems.

Motivation Large language models still struggle with complex, multi-step tasks when prompted directly, and existing approaches either use static tool libraries that limit adaptability or rely on large-scale models and multi-agent systems that are expensive to deploy. Reward signals in agentic reinforcement learning are also typically sparse—given only at task completion—making it hard to assign credit to individual steps. There was no framework that jointly improved both the agent's policy and its external tool memory together.

Methodology The authors formalize agent training as a Tool Memory-Enhanced Markov Decision Process, where the agent operates over a dynamic directed graph of tools (nodes) connected by execution dependencies (edges). At each step the agent retrieves relevant tools from the graph, decides whether to reuse or create a new one, and executes actions. Training uses a composite reward combining a sparse binary outcome reward with dense process-level rewards for planning, tool creation, and tool execution. Credit assignment is handled through a two-level advantage scheme: episode-level advantages across full trajectories, and step-level advantages anchored to specific tools, grouping actions that used the same tool across different trajectories. Experiments use 10,000 open-source RL training samples and evaluate on mathematical reasoning (AIME2024, MATH500, GSM8K) and multi-hop knowledge QA benchmarks (HotpotQA, 2WikiMultihopQA, Musique, Bamboogle) against baselines including GRPO, DAPO, REINFORCE++, and ARPO.

Results SEARL achieves the best average rank (1.43) across all benchmarks compared to baselines. It matches the top-performing method on AIME2024 (0.3333) while consistently outperforming or matching baselines on multi-hop QA tasks; on Bamboogle it reaches 0.3040 versus 0.2480 for the next best. The Tool Graph Memory grows from small disconnected subgraphs into interconnected clusters spanning multiple domains as training progresses, with the agent demonstrating higher training rewards and sustained exploration entropy compared to the GRPO baseline throughout training.

Plain-language abstract This paper introduces Reflexion, a framework that teaches AI language agents to improve through verbal self-reflection rather than by retraining the underlying model. After each attempt at a task, the agent writes a short textual reflection on what went wrong, stores it in memory, and uses that reflection to make better decisions on the next try — much like a person learning from mistakes across multiple attempts.

Motivation Large language models used as autonomous agents struggle to learn efficiently from trial and error, because traditional reinforcement learning requires large amounts of training data and expensive model fine-tuning. There was no lightweight mechanism for an LLM-based agent to carry forward lessons from past failures without modifying model weights.

Methodology Reflexion converts binary or scalar feedback from the environment into natural-language summaries, which are stored in an episodic memory buffer and prepended as context in subsequent episodes. The framework was evaluated on three task categories: sequential decision-making (AlfWorld), knowledge-intensive reasoning (HotPotQA), and programming (HumanEval). Feedback signals included simple binary environment feedback, predefined heuristics for common failure cases, and self-evaluation methods such as LLM-based binary classification or self-written unit tests.

Results Reflexion improved performance over strong baselines by an absolute 22% on AlfWorld decision-making tasks within 12 iterative learning steps, by 20% on HotPotQA reasoning questions, and by up to 11% on HumanEval Python programming tasks. On HumanEval, Reflexion achieved 91% pass@1 accuracy, surpassing the prior state-of-the-art of 80% achieved by GPT-4.

Plain-language abstract This paper introduces Reflective Memory Management (RMM), a system that helps AI dialogue agents remember and retrieve relevant information across many conversations over time. Because large language models are stateless — they forget everything between sessions — the authors built an external memory mechanism with two complementary parts that work together to store and recall user information more accurately and adaptively.

Motivation AI assistants used in customer service, healthcare, or education need to remember what individual users have told them across many past conversations, not just the current session. Existing external memory systems for language models suffer from two problems: they store information at rigid, pre-defined chunk sizes (such as per conversation turn or per session) that do not match natural topic boundaries, and they rely on fixed retrieval mechanisms that cannot adapt to different users or dialogue styles, making personalized long-term interaction difficult.

Methodology The RMM framework combines two mechanisms. Prospective Reflection dynamically decomposes dialogue history into topic-based memory units across multiple granularities — individual utterances, turns, and whole sessions — so that semantically related content is grouped together regardless of where session or turn boundaries fall. Retrospective Reflection treats retrieval improvement as an online reinforcement learning problem: as the language model generates responses, it cites which retrieved memories were useful, and those citation signals are used as unsupervised reward feedback to iteratively refine the retriever without requiring any labeled training data. The system was evaluated on two benchmarks, MSC and LongMemEval, using Contriever and GTE retrievers with Gemini-1.5-Flash and Gemini-1.5-Pro as generators.

Results RMM achieves more than 10% accuracy improvement over a no-memory baseline on LongMemEval, and more than 5% improvement over the strongest prior baseline across memory retrieval and response generation metrics on both benchmarks. The full RMM framework reaches a METEOR score of 30.8% on MSC and Recall@5 of 60.4% on LongMemEval. Citation-based scoring used in Retrospective Reflection was validated as reliable, achieving an overall F1 of 86.7% for identifying useful versus non-useful retrieved memories. Experiments also show that flexible topic-based granularity from Prospective Reflection approaches oracle-level performance compared to any single fixed granularity strategy.

Plain-language abstract CLEAR is a framework that helps AI agents perform better on tasks by generating customized background knowledge before each task, rather than simply retrieving stored instructions from previous runs. It trains a small, lightweight model to produce this task-specific context, which is then fed into the main AI agent's prompt. The system was tested on two benchmarks for agent task completion and outperformed existing approaches.

Motivation Large language model agents typically rely on either their built-in knowledge or retrieval of past experience to guide decision-making. Retrieved context from prior tasks is general-purpose and static, so the agent must figure out how to adapt it to each new situation — a burden that is especially problematic when the agent has limited reasoning ability or when new tasks differ substantially from past ones. There was no method that generated context tailored to the specific task at hand rather than reusing historical guidance.

Methodology CLEAR uses a two-stage training pipeline for a context augmentation model (CAM). First, a reflection agent performs contrastive analysis over past task execution trajectories, comparing successful and unsuccessful runs to produce task-specific summaries; these summaries become supervised fine-tuning data for the CAM. Second, the CAM is further optimized with reinforcement learning, where reward signals come from actually running the execution agent on tasks. The CAM is kept small and model-agnostic, adding minimal overhead, and does not require any modification to the underlying execution agent — making it compatible with proprietary LLMs. Evaluations were conducted on the AppWorld and WebShop benchmarks.

Results On the AppWorld benchmark test set, CLEAR improved the task completion rate from 72.62% to 81.15% compared to the baseline agent. On a subset of WebShop, it raised the averaged reward from 0.68 to 0.74. CLEAR consistently outperformed strong baselines across both benchmarks.

Plain-language abstract This paper tackles the problem of what an AI assistant should remember from past conversations. The authors build a benchmark called BEHEMOTH covering 18 datasets across very different task types, then propose a new method called CluE that learns better memory-extraction rules by grouping similar examples together before updating its strategy.

Motivation As LLM-based assistants become persistent across sessions, they need to decide what information from past conversations is worth storing as memory. Existing systems use fixed, hand-written extraction rules tuned for one narrow use case, but a general-purpose assistant faces many different kinds of interactions. There was no benchmark to measure how well memory extraction generalizes across such diverse tasks, and existing self-evolving prompt optimization methods were designed for homogeneous data and degrade on heterogeneous distributions.

Methodology The authors formalize memory extraction as producing a memory string from a past conversation using an extraction prompt, evaluated by how well that memory helps a separate generation model answer follow-up queries (a utility-driven metric). They construct BEHEMOTH by repurposing 18 existing datasets spanning personalization, problem-solving, and agentic task categories. They then propose CluE (Cluster-based Evolution), a self-evolving framework that batches training examples, clusters them by summarized extraction scenario, performs local analysis within each cluster, and synthesizes cross-cluster insights to iteratively update the extraction prompt.

Results Experiments on BEHEMOTH show that no single static extraction prompt dominates across all task categories. CluE achieves a +9.04% relative gain over a static baseline and consistently outperforms prior self-evolving frameworks (GEPA, ACE, MemEvolve), which were found to degrade under heterogeneous task distributions.

Plain-language abstract This paper introduces TSUBASA, a framework for making AI language models better at personalizing their responses over long stretches of time—for example, tracking a user's preferences and history across many conversations. It combines two components: one that maintains and updates a structured memory of the user, and one that trains the model to internalize that memory more effectively so it can answer personal questions without needing to look up everything at retrieval time.

Motivation Personalized language models struggle with long-horizon tasks—situations where they must reason across a user's extensive history of interactions. Existing memory systems grow linearly and fail at tasks requiring implicit temporal reasoning; retrieval-augmented approaches face a quality-efficiency tradeoff where better answers require retrieving more context at prohibitive cost; and fine-tuning approaches are held back by a train-inference gap because raw conversation data does not prepare models well for the complex personalization tasks they face at evaluation time.

Methodology TSUBASA uses a two-wing design. The first wing handles dynamic memory writing: core factual observations are extracted from raw conversations and a memory manager applies structured operations (ADD, UPDATE, RECONCILE, IGNORE) to keep the memory store compact and conflict-free. The second wing handles internalized memory reading through a self-learning pipeline that applies a teacher-student context distillation objective on synthetic question-answer pairs: a frozen teacher model sees the full session context, while a trainable student model sees only the question, and the distillation loss trains the student to internalize user-specific knowledge parametrically. Evaluations were conducted on long-horizon personalization benchmarks using the Qwen-3 model family ranging from 4B to 32B parameters, compared against memory-augmented baselines including Mem0 and Memory-R1.

Results TSUBASA surpasses competitive memory-augmented systems that rely primarily on memory writing, such as Mem0 and Memory-R1, on long-horizon benchmarks. The framework achieves Pareto improvements over prior approaches—delivering higher personalization fidelity while using a reduced token budget—effectively breaking the quality-efficiency tradeoff. The approach also preserves user privacy by avoiding cross-user training.

Plain-language abstract This paper introduces HiExp (Hierarchical Experience), a framework that improves how AI language models learn to search the web or external knowledge sources when solving complex, multi-step questions. Instead of having the model stumble through trial-and-error searches during training, HiExp teaches it strategic search patterns distilled from its own past successes and failures, making the model faster, more accurate, and more stable to train.

Motivation Current reinforcement learning methods for training AI search agents rely on stochastic exploration: the model tries many random reasoning paths and is rewarded only for getting the right answer. This is inefficient, prone to redundant or irrelevant search steps, and produces unstable training signals — especially for smaller models tackling complex multi-hop questions that require chaining several pieces of information together. There was no principled way to extract and reuse the procedural knowledge embedded in a model's own successful reasoning trajectories.

Methodology HiExp operates in two stages. First, it constructs hierarchical experience from the model's own rollouts: for each training question, multiple reasoning trajectories are sampled, then contrasted by outcome (successful vs. failed) to identify key decision points and reasoning traps. A multi-level agglomerative clustering step abstracts these case-specific insights into higher-level strategic principles. Second, during reinforcement learning training, these distilled experiences are dynamically injected into the rollout stage so the model's exploration is guided by accumulated knowledge rather than pure trial-and-error. Experiments used Qwen2.5-7B and Qwen2.5-32B base models trained on six multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, MoreHopQA, Frames) and six mathematical reasoning benchmarks.

Results HiExp-Searcher outperforms prior reinforcement-learning-based search agents across all evaluated benchmarks. On the 7B model, it achieves an average CEM gain of roughly 9.7 points over the next-best RL baseline across the four primary multi-hop benchmarks. The framework also generalizes to out-of-domain datasets and to mathematical reasoning tasks without task-specific retraining. An ablation on experience source showed that self-distillation (the 7B model generating its own experiences) slightly outperforms using a larger teacher model (Qwen-Max) by about 1.2% on average, indicating that alignment with the student model's own reasoning distribution matters more than raw experience quality. Training stability analysis confirmed that HiExp substantially reduces reward variance and gradient noise compared to the standard GRPO baseline.

Plain-language abstract This paper introduces Mem2Evolve, a framework for building AI agents that can improve themselves over time by combining two types of memory: a store of past task experiences and a store of dynamically created tools and specialist sub-agents. Rather than treating these two growth processes separately, the system makes them work together so each one reinforces the other.

Motivation Existing self-evolving agent frameworks either let agents learn from experience but keep them locked to a fixed set of tools, or let agents create new tools and sub-agents but do so blindly without drawing on past experience. Both approaches are limited: the first cannot expand what the agent is capable of doing, while the second produces unreliable results because it ignores proven strategies and known pitfalls.

Methodology Mem2Evolve maintains two memory components — an Experience Memory that stores trajectories from past task executions, and an Asset Memory that stores dynamically created tools and expert agents. When a new task arises, the system retrieves relevant past experience to guide the creation of new tools or specialist agents, then stores the resulting execution trajectory back into experience memory, enabling a continuous co-evolutionary loop. The framework was evaluated across 6 task categories and 8 benchmarks.

Results Mem2Evolve achieves an 18.53% improvement over standard large language models, an 11.80% improvement over agents that evolve only through experience accumulation, and a 6.46% improvement over agents that evolve only through asset creation, demonstrating that the co-evolutionary approach is more effective and stable than either strategy in isolation.

Plain-language abstract This paper introduces MemCoE, a two-stage framework that teaches AI assistants to better remember and update information about individual users over long conversations. Rather than relying on hand-crafted memory rules, MemCoE learns both how to organize memory (what patterns to follow) and what specific information to store or forget, resulting in more consistent and personalized responses over time.

Motivation Large language models have limited context windows, so they cannot simply remember everything a user has said across many sessions. Existing memory systems use static, hand-crafted rules that cannot adapt to changing user behavior, while reinforcement learning approaches that treat memory updates as learnable actions suffer from sparse, delayed rewards that make training unstable and data-hungry. There was no principled way to jointly learn the organizational structure of memory and the specific update decisions an agent should make.

Methodology MemCoE draws an analogy to human memory schema theory, specifically the functional division between prefrontal cortex (which configures schemas) and hippocampus (which encodes episodic details). Stage one, Memory Guideline Induction (MGI), treats the memory-update instruction prompt as a learnable natural-language parameter and refines it using contrastive feedback over memory-augmented conversation trajectories, interpreted as textual gradients aggregated at the batch level. Stage two, Guideline-Aligned Memory Policy Optimization (GMPO), uses the induced guideline to define structured process rewards and trains a memory-evolution policy via multi-turn reinforcement learning. The framework is evaluated on three personalization memory benchmarks: PersonaMem, PrefEval, and PersonaBench, covering explicit and implicit user preferences across varying history lengths and noise levels.

Results MemCoE consistently outperformed strong retrieval-based and RL-based memory baselines across all three benchmarks. On the PersonaMem benchmark, the MGI-induced guideline achieved accuracy of 53.28 at 32K context and 43.76 at 128K context, corresponding to relative improvements of +10.4% and +11.3% over a manually written prompt baseline. The induced guideline also proved transferable across different LLM backbones: optimizing with one model generalized well to others including GPT-5 and Gemini-2.5-flash. The system showed favorable robustness under longer histories, noisier evidence, and different retrieval configurations.

Plain-language abstract Evo-MedAgent is a memory system that lets an AI medical imaging agent learn from its own past cases at test time, without any retraining. Instead of treating each chest X-ray case in isolation, the agent accumulates three kinds of memory — records of past diagnostic episodes, a bank of distilled diagnostic rules, and a tracker of which tools are reliable — and uses them to do better on each new case.

Motivation Current tool-augmented LLM agents for medical image interpretation solve every case independently: they cannot retain lessons from prior successes or failures, correct recurring reasoning mistakes, or adjust their trust in specialist tools without expensive retraining. Human radiologists naturally improve with experience, but existing agents remain static across an entire evaluation benchmark.

Methodology The authors formalize the setting as sequential inference over an ordered series of chest X-ray cases from ChestAgentBench (2,500 multiple-choice questions across 675 expert-curated cases). After each case, ground-truth feedback triggers a reflection step that updates three memory stores: an episodic store of compressed past cases retrieved by similarity, a procedural store of priority-tagged diagnostic heuristics with per-rule utility tracking, and a tool-governance store that labels each specialist tool as trusted, caution, or avoid based on accumulated interaction statistics. The framework requires no LLM parameter updates — only one retrieval pass and one reflection call per case — and was tested with GPT-5-mini, Gemini-3 Flash, and GPT-5.2 as backbone vision-language models.

Results On ChestAgentBench, adding the full Evo-MedAgent memory raised multiple-choice accuracy from 0.68 to 0.79 on GPT-5-mini and from 0.76 to 0.87 on Gemini-3 Flash. An ablation showed that episodic and procedural memory each contribute independently, and their combination consistently outperforms either alone. Notably, for qualitative diagnostic tasks, the tool-free full-memory configuration outperformed the tool-enabled baseline, suggesting that accumulated reasoning heuristics can substitute for noisy external specialist tools when the base model is strong. GPT-5-mini full-memory accuracy was stable across three random case-order permutations (mean 0.79, SD 0.025).

Plain-language abstract This paper introduces SEA, a self-learning AI agent designed to improve its clinical diagnostic ability over time by accumulating and reusing experience, much like a human physician does. Rather than treating each patient case in isolation, SEA maintains a memory system that stores recent cases and distills them into reusable diagnostic rules, then uses reinforcement learning to improve both its reasoning and its memory management simultaneously.

Motivation Most LLM-based diagnostic agents treat each case independently, which prevents them from learning and adapting through experience the way clinicians do. Prior work focuses on what to diagnose rather than how to improve over time, leaving a gap in long-horizon, experience-driven learning for medical AI. The medical domain is especially challenging because clinical cases are information-dense and useful learning often requires abstraction into generalizable rules rather than verbatim storage.

Methodology SEA uses a dual-memory architecture inspired by human cognition: a short-term memory stores a bounded set of recent concrete cases (up to capacity K), and a long-term memory stores abstract diagnostic rules distilled from those cases via a consolidation step. The agent is trained with a reinforcement learning framework that produces structured reasoning trajectories and applies adaptive rewards optimizing both diagnostic accuracy and effective memory management decisions. The system is evaluated on two datasets: MedCaseReasoning for static diagnostic competence, and ER-Reason for simulating streaming deployment with ongoing feedback.

Results On the MedCaseReasoning dataset, SEA achieves 92.46% accuracy, outperforming the strongest baseline by +19.6 percentage points. On the long-horizon ER-Reason dataset, SEA attains the best final accuracy of 0.7214 and the largest improvement of +0.35 delta accuracy over 100 cases, while standard and RL-trained baselines show only marginal gains or even degradation. Expert clinical evaluation of the agent's consolidated rules yields strong scores for overall trust (4.216 out of 5), rule usefulness (3.946), and clinical correctness (3.865), indicating the induced rules are reliable and practically meaningful.

Plain-language abstract This paper asks whether AI coding assistants should update their own weights nightly based on user interactions, instead of relying on in-context summaries that get repeatedly compressed and discarded. The authors build a three-stage pipeline that extracts knowledge from daily conversations, generates synthetic training data, and fine-tunes a small language model using LoRA on a single consumer GPU overnight. They compare this approach against the context-compaction strategy used by current deployed assistants across ten simulated software development projects.

Motivation Large language models deployed as coding assistants never update their weights per user, so they cannot accumulate preferences, corrections, or project-specific knowledge across sessions. Platforms that carry forward summaries must repeatedly compress prior context at a roughly 6:1 ratio, and each compression cycle degrades what was retained. After only three such cycles, a significant fraction of knowledge is lost — yet the volume of relevant project history grows continuously while the context window does not.

Methodology The authors generated ten realistic multi-turn software development conversations (each 50,000–65,000 tokens, covering projects such as CLI tools, ML frameworks, and distributed systems) using Claude Sonnet 4. For each conversation, a three-stage consolidation pipeline ran: a reflection stage extracted structured memories classified as procedural, semantic, or episodic; a synthesis stage produced roughly 20 paraphrased user-assistant conversation pairs per fact (approximately 18,000 training examples per source conversation); and a training stage fine-tuned a Qwen2.5-7B-Instruct model via LoRA (rank 16, 8 epochs) on an NVIDIA A100. A parallel compaction condition simulated three cycles of 6:1 summarization plus fresh continuation conversations, matching the behavior of deployed coding assistants. Retention was measured on 1,146 test questions judged by an LLM evaluator.

Results Three cycles of cascading compaction retained 36.8 ± 3.0% of knowledge, compared to 80.4 ± 1.3% for weight-based consolidation — a 43.6 percentage-point gain (paired t(9) = 14.8, p < 0.001) that more than doubles what compaction preserves. The gains were largest for procedural corrections (36.3% vs 74.6%) and episodic project facts (31.5% vs 78.2%), while semantic memories were better preserved under both strategies. The paper also reports a methodological finding: median per-token validation cross-entropy tracks LLM-judged task accuracy almost perfectly (r = +0.99), while mean per-token cross-entropy is negatively correlated with accuracy (r = −0.51) due to a heavy-tailed token-loss distribution, making the mean a misleading training signal when the evaluator tolerates surface-form variation.

Plain-language abstract This paper is a survey of how memory systems in AI agents powered by large language models (LLMs) have developed over time. The authors propose a three-stage evolutionary framework — Storage, Reflection, and Experience — to organize and explain the progression of memory designs, and use it to identify the core technical drivers and open challenges in the field.

Motivation LLMs are stateless by nature, meaning they cannot retain information across interactions on their own. This makes it hard for LLM-based agents to stay consistent over long, multi-step tasks or to learn from past mistakes. Prior survey work on agent memory was fragmented, split between engineering-oriented and cognitive-science-oriented perspectives, with no unified framework explaining why and how memory mechanisms have evolved.

Methodology The authors conduct a systematic literature survey organized around a novel three-stage taxonomy they formally define: Storage (faithful recording of interaction trajectories), Reflection (dynamic evaluation and refinement of stored records), and Experience (cross-trajectory abstraction of generalized behavioral patterns). They analyze the field using a 'Why-How-What' structure addressing three research questions about the drivers, path, and outcomes of memory evolution, and they also catalog relevant benchmarks across each stage.

Results The survey maps existing memory mechanisms onto the three-stage framework and identifies the core catalysts driving evolution: the need for long-range consistency, the challenges of dynamic environments, and the goal of continual learning. It highlights that the frontier Experience stage — characterized by active exploration and cross-trajectory abstraction using techniques such as Minimum Description Length compression of trajectory clusters — addresses bottlenecks in agent adaptability. The authors also identify gaps in benchmark coverage, particularly for the Experience stage, and call out distributed shared memory and multimodal memory fusion as critical directions for future work.

Plain-language abstract This paper proposes CoALA (Cognitive Architectures for Language Agents), a conceptual framework for organizing and designing AI agents that use large language models (LLMs) to interact with the world. The authors draw on the history of cognitive science and symbolic AI — specifically production systems and cognitive architectures like Soar — to give a principled structure to a rapidly growing but terminologically fragmented field.

Motivation Researchers have built many different language agents that connect LLMs to memory, tools, and environments, but each work uses its own custom vocabulary (such as 'tool use', 'grounding', 'actions'), making it hard to compare systems, trace how the field is evolving, or build new agents on clean, consistent abstractions. There was no unifying framework to organize existing agents or guide future development.

Methodology The authors propose CoALA, which characterizes language agents along three dimensions: information storage (working memory and long-term memory subdivided into procedural, semantic, and episodic types); an action space (internal actions like reasoning and retrieval, and external actions like interacting with the environment); and a structured decision-making loop consisting of planning and execution phases. They use this framework to retrospectively survey and taxonomize a large body of recent language-agent work, and to prospectively identify open research directions.

Results CoALA successfully organizes a wide range of existing language agents under a common conceptual structure, showing that diverse systems can be understood as varying instantiations of the same memory, action, and decision-making components. The framework also highlights systematic gaps in current agents — particularly in long-term memory management and learning — and outlines actionable directions toward more capable, general-purpose language agents. The paper was published in Transactions on Machine Learning Research in February 2024.

Plain-language abstract SEAGym is a testing environment for AI agents that improve themselves over time by editing their own setup—their prompts, memory, skills, tools, and configuration—rather than by retraining the underlying model. Instead of just checking whether a self-improved agent scores higher on a final task, SEAGym measures the improvement process itself: whether each self-edit actually helps on new tasks, whether gains stick or later collapse, and what they cost.

Motivation Self-evolving agents improve mainly by changing their agent harness—the structured execution layer around a base model. Existing evaluations reduce this to isolated task scores or a single sequential learning curve, which obscures whether a given update produces reusable improvement, overfits the recent tasks, increases cost, or harms older behavior. Most agent benchmarks are built for static evaluation, resetting agent state between isolated episodes and removing exactly the state persistence that self-evolution depends on.

Methodology SEAGym uses an RL-style environment formulation in which the self-evolving agent supplies both the task policy and the harness-update rule, while the environment defines task sampling, feedback, schedules, and snapshot assessment. It converts Harbor-compatible static benchmarks into reusable task sources organized into train batches and frozen evaluation views—update-validation, held-out in-domain transfer, out-of-domain transfer, replay diagnostics, and cost records—and saves agent snapshots and metric artifacts. Explicit schedule parameters (state reset, task reuse, batch size, update timing) let single-task adaptation, online transfer, and epoch-based batch learning be studied under one protocol. Task rollout is separated from method update, so methods such as ACE, TF-GRPO, and AHE connect through thin wrappers while keeping their native update rules; experiments instantiate the environment on Terminal-Bench 2.0 and HLE.

Results The evaluation views provide complementary signals about the evolution process rather than one summary number. Frequent updates may fail to improve held-out performance, validation gains do not always transfer to in-domain or out-of-domain test views, and useful intermediate snapshots can collapse later or recover. Batch size, source diversity, and the rollout model backend all affect harness reliability, showing that self-evolution dynamics depend on the evaluation schedule and setup, not just the update method.

Plain-language abstract AutoMem trains an agent to get better at managing its own memory, the way a person gets better at note-taking with practice. It gives the agent file-system operations (read, write, search, append) as memory actions on equal footing with its normal task actions, then uses a strong meta LLM to review whole episode traces (tens of thousands of steps) and improve both the memory scaffold the agent works within and, separately, the agent's own skill at making memory decisions.

Motivation Long-horizon tasks can run for 10^4-10^5 steps, far past what a human reviewer can practically read through to find where a memory decision went wrong; a single bad memory choice can stay hidden for thousands of steps before it matters. Existing memory systems treat memory as a fixed architectural module rather than a skill the agent can improve, so there's no natural way to optimize it beyond manual tuning.

Methodology Two outer loops wrap a shared inner-loop agent that treats a directory of files as its memory. Loop 1 (structure): a meta-LLM reads full episode traces, diagnoses memory-use failure patterns, and revises the agent's scaffold, prompts, file schema, action vocabulary. Loop 2 (proficiency): a meta-LLM selects the agent's own good memory decisions across many episodes as supervised training data and orchestrates LoRA fine-tuning of a dedicated memory-specialist model, while the task-action model stays frozen. Evaluated on three procedurally generated long-horizon games (Crafter, MiniHack, NetHack) with Qwen2.5-32B-Instruct as the base model.

Results Optimizing memory alone, without touching the model's task-action behavior, improves the base agent's performance roughly 2x-4x across the three games, and the optimized 32B model outperforms Qwen2.5-72B-Instruct on all three, becoming competitive with frontier systems like Claude Opus 4.5 and Gemini 3.1 Pro Thinking. The authors frame this as evidence that memory management is an independently learnable skill and a high-leverage optimization target for long-horizon agents.

Plain-language abstract SelfMem gives an AI agent a set of memory tools and feedback signals, instead of a fixed rule for what to remember and how, and lets the agent figure out its own memory strategy. The idea, as the authors put it, is teaching the agent to fish rather than giving it a fish: rather than forcing a predefined summarization format, the agent inspects the raw conversation, decides what's worth writing to its memory workspace, and can revise that memory when a self-check flags a problem.

Motivation Existing agent-memory systems (Mem0, MemGPT, MemoryBank, A-Mem) rely on manually specified strategies for what to store, update, and retrieve, which are rigid across tasks and need hand-tuning. Different tasks and conversation histories need different memory behaviors, and no fixed schema fits all of them; the authors argue this calls for letting the agent adapt its own strategy rather than hand-crafting one.

Methodology SelfMem keeps the raw transcript as an immutable source of truth, accessible only through read-only tools like a SQL-queryable turns table, and gives the agent a separate memory workspace plus a memory action space: read the transcript, read/write/revise memory, check memory quality, and even extend its own memory toolkit. The agent decides what to store, compress, update, or leave for transcript retrieval. Evaluated on BEAM across conversation scales from 100K to 1M tokens against retrieval, compression, and agent-memory baselines.

Results SelfMem gets the highest official score and Pass0.5 at all three tested scales (100K, 500K, 1M tokens), improving the official score over the strongest baseline by 0.165, 0.141, and 0.134 respectively, and Pass0.5 by 14.3-17.0 percentage points. It's the best performer on 9/10 question types at 100K tokens (8/10 at 500K, 7/10 at 1M), and a model-guided strategy-refinement study shows further gains are possible on top of the base approach.

Plain-language abstract Holding a diagnostic-routed self-evolution recipe (TRIAGE) fixed across 8 programming languages and 3 base models, evolved coding-agent harnesses turn out to mostly compensate for recoverable execution defects (like test collisions or non-compiling code) rather than encode deep language-specific knowledge, and share an abstract playbook but not the concrete machinery across languages.

Motivation Self-evolving coding agents that edit their own prompts, hooks, memory, and tools show large empirical gains, but it's unclear whether these evolved harnesses encode genuine language expertise or are just compensating for fixable execution-environment bugs, and whether what's learned transfers across languages or models.

Methodology A fixed diagnostic-routed self-evolution recipe, TRIAGE, run across a grid of 8 programming languages by 3 base models, analyzing what kind of defect (recoverable execution defect vs. deeper knowledge gap) each evolved harness edit addresses, and measuring how much of each harness's machinery is shared across languages versus ecosystem-specific, plus testing a distilled universal harness against native per-language harnesses.

Results Evolved harnesses primarily compensate for recoverable execution defects (test-file collisions, non-compiling code, skipped test runs) rather than encoding deep language knowledge, with which defect dominates varying by language/model cell rather than following a fixed rule. Harnesses share an abstract playbook across languages but instantiate it with 20-40% ecosystem-specific machinery per harness, and a distilled universal harness recovers only 48-68% of the native per-language gains on some ecosystems.

Plain-language abstract A re-evaluation of memory-based self-improving web agents (Agent Workflow Memory, ReasoningBank) across WebArena, VisualWebArena, and SCUBA, adding multiple runs and randomized task ordering to standard single-run, fixed-order evaluation. Both additions expose that reported improvements are considerably less stable than prior single-run reports suggested.

Motivation Self-improving agents that write and reuse a textual memory bank across a stream of tasks are promoted as a path to reliable long-horizon automation, but their reliability under realistic conditions — repeated runs, unordered task streams — had not been stress-tested. In enterprise deployment, tolerance for early failure is low and mistakes compound silently.

Methodology The authors re-run Agent Workflow Memory and ReasoningBank with a stronger base model and harness, then extend evaluation along two axes: multiple runs of the same setting to quantify variance, and randomly shuffled task orders to test sensitivity to the default (often implicitly easy-to-hard) task curriculum. They then manually inspect agent-written memories to identify failure drivers, and test whether adding rubrics and environment feedback to memory construction narrows the gap.

Results Variance across runs increases in 71% of settings when a self-improving loop is stacked on a baseline agent; on WebArena's GitLab subset, the best/worst run gap widens from 4.4% (no memory) to 7.8% (ReasoningBank). Under the default task order, ReasoningBank gains 1.5% on average; under random shuffling, it degrades by 4.5%. Manual inspection attributes much of this to task and environment underspecification, which causes agents to write plausible but inapplicable memories; adding rubrics and environment feedback partially, but not fully, closes the degradation.

Plain-language abstract A survey of coding agents that update their own framework, memory, skills, tools, models, or workflow structure from prior coding attempts, organized by what evolves, when it evolves, and what software-specific evidence drives it.

Motivation Coding agents now operate as interactive systems inside real development workflows, interpreting requirements, inspecting repositories, editing multiple files, running tests, diagnosing failures and generating patches. But in most systems the base model, prompts, tool interfaces, memory mechanisms and control flow are fixed after deployment, which is hard to sustain when codebases evolve, APIs and dependencies change, conventions vary across repositories, and bug fixing requires repeated cycles of localization, patching, execution and revision. Meanwhile software engineering supplies unusually concrete feedback: unit tests, compiler errors, runtime traces, lint warnings, CI results and human code reviews. An agent that cannot accumulate experience from that feedback repeats its mistakes across tasks and never adapts to project-specific context. Existing work on self-evolving agents mostly targets broad task environments and leaves the software-specific requirements uncharacterized.

Methodology The survey is framed as a guiding synthesis rather than a review of a settled paradigm, organized around three questions: what components evolve and through what mechanisms, when evolution occurs and what evidence drives it, and how such agents should be evaluated beyond the setting they evolved in. It first separates coding agents, self-evolving agents and self-evolving coding agents conceptually, then builds an object-centered taxonomy with five categories: agent framework self-evolution (self-modifying scaffolds and archive-based variant search), memory self-evolution (experience, repository and plan-abstraction memory), skill and tool self-evolution, model self-evolution (adversarial test evolution, coder-verifier co-evolution, self-play policy evolution), and workflow and topology self-evolution. Two orthogonal dimensions cross it: timing (task-time, post-task, stage-wise) and evidence type (outcome, environmental, trajectory-derived). Representative systems are tabulated against all three dimensions plus their software-engineering task domain. Inclusion is limited to work whose central contribution changes an agent component or behavior through coding-specific feedback; benchmark-only datasets and static agent systems are treated as evaluation context rather than as methods.

Results The synthesis finds that executable feedback, repository-level context and coding trajectories give software engineering a distinctive role as a domain for agent self-evolution, because a remembered strategy can be validated by tests rather than merely accumulated, and repository history supplies long-term signal about how a project changes. Memory self-evolution clusters into experience banks over prior issue-resolution trajectories including failures (SWE-Exp), hierarchical pools separating general from repository-specific experience (EvoCoder), subtask-granularity stores that avoid coarse whole-trajectory matching, commit-and-issue-grounded repository memory for code localization, and plan abstraction reused as later context (SAGE). Skill and tool self-evolution is distinguished by actionability: CODESKILL maintains a skill bank at task and event granularity and treats skill management as a learnable policy; gskill learns per-repository skill documents validated by whether agents in isolated environments produce passing patches; Socratic-SWE distills a skill registry from traces that then generates the training curriculum. Tool self-evolution is the least developed, with Live-SWE-Agent the main example of an agent creating and revising its own tools from a bash-only scaffold. The named challenges are feedback reliability, benchmark overfitting, safety, maintainability, cost and generalization. Two are emphasized: memory must be evolved selectively, since noisy logs, misleading tests, brittle patches and repository-specific conventions produce memories that hurt when retrieved uncritically; and framework-level evolution carries the strongest risk, because a harmful modification to the machinery that generates future actions can break the agent loop, degrade tool use, overfit to benchmark feedback or exploit weaknesses in the evaluation harness, so it requires validation, rollback and robustness checks rather than performance-driven search alone.

Plain-language abstract Graph RAG systems answer each query independently and discard the reasoning afterwards, so related later queries start from scratch. LivingRAG adds a writable experience store to a graph retrieval backbone, keeping for each verified answer the entity activation map that worked and a compact reasoning summary, and reusing them on two separate paths: fused into the retriever's starting activation, and inserted into the prompt as a scaffold. Across five online QA streams it improves accuracy over graph RAG baselines while cutting completion tokens 22.7% and estimated cost 12.1%, with grounding and novelty gates admitting only 27.4% of candidate experiences.

Motivation Reuse across an online query stream is not limited to repeated entities. Two questions may visit nearby regions of the retrieval graph with entirely different entities, or follow the same reasoning pattern with different subjects, as when two comparison questions both ask which person was born earlier. A measurement of the streams supports the distinction: direct entity overlap with an earlier query ranges from 4.30% to 76.69% by dataset, while graph-neighborhood overlap runs 80.45% to 99.70%, so an entity-keyed answer cache would miss most of the available reuse. The countervailing risk is that a writable store lets unsupported reasoning be written once and then amplified by every later query that reuses it.

Methodology The retrieval backbone is LinearRAG, unchanged: a passage-sentence-entity graph built with lightweight entity extraction and no LLM calls during indexing, query entity activation, propagation over the sentence-entity graph, and Personalized PageRank over the passage-entity graph. A verified experience stores the query, its embedding, a sparsified final activation vector, a compact summary, the answer, a timestamp and a grounding confidence. Experiences are scored against a new query by a convex combination of query-embedding cosine and activation-map cosine, and the top-K maps are fused into the initial activation vector with softmax weights scaled by stored confidence and deliberately not renormalized. Scaffold selection adds a masked-template similarity term that retrieval omits, because a scaffold only enters the prompt and cannot inject historical entities into graph retrieval. Write-back requires a novelty score computed against the base rather than the fused activation to avoid circularity, and a grounding score equal to the fraction of extracted atomic claims entailed by retrieved passages under an NLI model; novelty is checked first so NLI runs only on survivors, and the grounding score becomes the stored confidence. Evaluation runs 2WikiMultiHopQA, HotpotQA, MuSiQue, MuSiQue-full and WixQA as online streams with an empty initial store, generating with Qwen3.6 Plus, reporting contain-match and LLM-evaluation accuracy.

Results LivingRAG leads all baselines on the four multi-hop benchmarks, for instance 58.42 LLM-evaluation accuracy on MuSiQue-full against 52.71 for LinearRAG, 47.75 for GFM-RAG and 44.06 for HippoRAG2, and 70.25 against 66.00 on WixQA. Prompt tokens rise 3.5% weighted because scaffolds add context, completion tokens fall 22.7%, and total estimated cost falls from $51.05 to $44.87, a reduction on every dataset. The saving is not a late-stage artifact: per-segment traces show the system spending more completion tokens than its backbone in the earliest segments and turning negative once experience accumulates. Realized reuse differs sharply by stream, with graph transfer at 91.20% on MuSiQue and 0.00% on 2Wiki, where realized template reuse is 84.15% instead. Ablations match those traces, since removing activation fusion costs most where graph transfer is realized, removing scaffolds costs most on 2Wiki and cuts token savings from 23.6% to 8.3%, and removing the quality gate lowers accuracy on all three tested datasets. The stated limits are the store's lifecycle: nothing updates, downweights or deletes an accepted experience, and the fixed-corpus benchmarks contain no chronological updates or fact-validity intervals, so staleness cannot be evaluated.

Plain-language abstract This paper introduces Agent Workflow Memory (AWM), a method that teaches language model-based web agents to extract reusable task routines — called workflows — from past experience and apply them to guide future actions. Rather than treating each task in isolation, AWM lets agents build up a growing library of skills, from simple steps like finding a place by name to more complex multi-step procedures composed from earlier workflows.

Motivation Current language model agents struggle with long, complex web navigation tasks because they process each task independently and do not learn from past successes or failures. They lack the ability to extract and reuse common task patterns across similar contexts, making them brittle when the task environment changes even slightly.

Methodology AWM operates in two modes: offline, where it extracts workflows from annotated training trajectories before test time, and online, where it induces workflows on the fly from self-generated predictions judged correct by an evaluator. The method was evaluated on two major web navigation benchmarks — Mind2Web (1000+ tasks across 200+ domains) and WebArena (execution-based evaluation covering travel, shopping, social media, and other domains) — using language model agents that maintain and grow a memory of induced workflows over time.

Results AWM improved baseline success rates by 24.6% relative on Mind2Web and 51.1% relative on WebArena, while also reducing the number of steps needed to solve WebArena tasks. It outperformed methods that use human-expert-written workflows by 7.9% on WebArena. In cross-task, cross-website, and cross-domain generalization evaluations, online AWM surpassed baselines by 8.9 to 14.0 absolute percentage points, with the performance gap growing as the train-test distribution divergence increased — reaching as high as 22.5 points after rolling over only tens of examples.

Plain-language abstract This paper introduces LoCoMo, a dataset and benchmark for testing whether AI language models can remember and reason about information across very long conversations — up to 35 chat sessions and around 300 turns. The authors also build and evaluate several model approaches against human performance on tasks like question answering, event summarization, and multi-modal dialogue generation.

Motivation Prior work on long-term open-domain dialogue only evaluated models over contexts spanning at most five chat sessions, leaving it unknown how well large language models or retrieval-augmented generation systems handle truly extended conversational histories. This gap matters because real human relationships involve months or years of accumulated shared context that a useful conversational AI should be able to track.

Methodology The authors created a machine-human pipeline that uses LLM-based agent architectures to generate long-term dialogues grounded in character personas and temporal event graphs; agents were also given the ability to share and react to images. The generated conversations were then verified and edited by human annotators to ensure long-range consistency and fidelity to the event graphs, resulting in the LoCoMo dataset of conversations averaging 300 turns and 9,000 tokens each over up to 35 sessions.

Results Experiments show that current LLMs struggle to understand lengthy conversations and to track long-range temporal and causal dynamics across sessions. Techniques such as long-context LLMs and retrieval-augmented generation offer some improvement, but all tested models still fall substantially short of human performance on the benchmark tasks.

Plain-language abstract Synthius-Mem is a memory system for AI agents that stores what is known about a person rather than replaying raw conversation history. Drawing on how the human brain organizes memory into specialized subsystems, it decomposes conversations into six typed domains — biography, experiences, preferences, social circle, work, and psychometrics — and retrieves structured facts at query time. On the LoCoMo benchmark it achieves 94.37% overall accuracy and 99.55% adversarial robustness while consuming roughly five times fewer tokens than full-context replay.

Motivation Current LLM memory approaches — sliding windows, summarization, embedding-based retrieval, and flat fact extraction — each reduce token cost but introduce information loss, semantic drift, or uncontrolled hallucination about the user. No published system on the LoCoMo benchmark reported adversarial robustness, meaning the ability to refuse questions about facts the user never disclosed. The paper addresses this gap by arguing that domain-agnostic storage discards the structural advantages that neuroscience shows different memory types require.

Methodology Synthius-Mem uses a multi-stage pipeline: conversations are parsed and chunked, then six parallel domain-specific extractors pull typed facts into separate stores for biography, experiences, preferences, social circle, work, and psychometrics (the last using nine validated psychological frameworks). Each domain undergoes its own consolidation and deduplication pass. At inference time, a planner LLM selects relevant domains and a CategoryRAG component retrieves structured facts with a reported mean latency of 21.79 ms. The system was evaluated on the LoCoMo benchmark (ACL 2024), which contains 10 multi-session conversations and 1,813 questions spanning single-hop recall, multi-hop reasoning, temporal reasoning, open-domain knowledge, and adversarial false-premise questions.

Results Synthius-Mem achieves 94.37% accuracy on LoCoMo, exceeding the prior best published system MemMachine (91.69%) and human performance (87.9 F1). Core memory fact accuracy reaches 98.64%. Adversarial robustness — refusal to hallucinate facts the user never disclosed — reaches 99.55%, a metric no competing system reports. Token consumption is approximately five times lower than full-context replay at 500 messages, where full-context replay itself achieves only 85.46% in the authors' controlled evaluation.

Plain-language abstract LongMemEval is a benchmark for testing how well AI chat assistants remember and reason over information shared across many conversations with a user. It contains 500 carefully crafted questions embedded within long, scalable user-assistant chat histories, covering five memory abilities: extracting information, reasoning across sessions, handling time, updating knowledge, and knowing when not to answer.

Motivation Current AI chat assistants struggle to remember personal details and preferences across extended interactions, yet existing benchmarks for evaluating this mostly use human-to-human dialogues, lack task-oriented exchanges, and cover only short histories of a few thousand tokens. There was no rigorous, scalable benchmark that tested the full range of memory abilities needed for sustained, personalized user-AI interactions.

Methodology The authors built LongMemEval with 500 questions spanning five memory ability categories, embedded within freely scalable synthetic chat histories. They evaluated commercial systems (ChatGPT and Coze) and long-context LLMs on these questions, then developed a unified three-stage framework (indexing, retrieval, reading) and proposed specific optimizations: session decomposition for finer-grained value storage, fact-augmented key expansion at indexing time, and time-aware query expansion to sharpen retrieval scope.

Results Commercial chat assistants and long-context LLMs showed roughly a 30% accuracy drop when memorizing information across sustained interactions compared to offline reading with the full history provided. ChatGPT tended to overwrite crucial information as conversations continued, while Coze often failed to record indirectly provided user information. The proposed memory design optimizations substantially improved both memory recall and downstream question-answering performance on LongMemEval.

Plain-language abstract REALTALK is a dataset of real conversations collected over 21 days from messaging apps, where pairs of people who initially met through those apps exchanged daily messages. The dataset contains 10 unique conversations totaling more than 16,000 words each, and is used to evaluate whether AI language models can hold long-term, emotionally aware conversations the way humans do.

Motivation Most research on long-term dialogue systems uses synthetic conversations generated by AI models rather than real human exchanges, leaving it unclear whether those simulated dialogues capture the emotional nuance and persona consistency of genuine human interaction. REALTALK was created to fill that gap by providing an authentic benchmark that can be directly compared against LLM-generated conversation datasets.

Methodology Ten participants each engaged in two separate 21-day conversation threads with different partners via messaging apps, producing conversations that span approximately 21 daily sessions per pair. The dataset was analyzed for emotional intelligence attributes and persona consistency, and compared against LLM-generated conversations. Two benchmark tasks were then defined: persona simulation, where a model must continue a conversation on behalf of a specific user given prior dialogue context, and memory probing, where a model must answer questions requiring recall of information from earlier in a long conversation.

Results Analysis showed that real-world conversations contain more diverse emotional expressions and greater variation in persona stability than synthetic LLM-generated dialogues. Benchmark experiments found that models struggle to simulate a specific user's conversational style from dialogue history alone, but fine-tuning on that user's own chat history improves persona emulation. Existing models also faced significant challenges in recalling and leveraging long-term context from real-world conversations.

Plain-language abstract This paper introduces LifeDialBench, a new benchmark for testing how well AI memory systems can handle continuous spoken conversations recorded by wearable devices like smart glasses. Unlike existing benchmarks that focus on chatting directly with an AI, this one evaluates memory over the kind of multi-person, everyday dialogues that an always-on microphone would capture. The paper also proposes a new evaluation protocol that respects the time-ordering of events, mimicking how such a system would work in practice.

Motivation Current AI memory benchmarks focus on one-on-one person-AI chat sessions and do not address the emerging use case of wearable devices that continuously record ambient conversations throughout the day. Existing approaches to extending memory — such as feeding ever-longer text into a model — become prohibitively expensive and still underperform. There was no public benchmark designed for the continuous, multi-person, time-stamped dialogue streams that wearable lifelogging devices produce.

Methodology The authors built LifeDialBench from two complementary data sources: EgoMem, derived from real-world egocentric video recordings (transcribed via automatic speech recognition), and LifeMem, constructed through a hierarchical life-simulation framework using a virtual community with human-in-the-loop review. Question-answer pairs cover four types — single-event, event detail, multi-event, and temporal information queries — spanning timescales from 30-second clips to week-level summaries. Crucially, the paper introduces an Online Evaluation protocol that enforces temporal causality, evaluating memory systems incrementally as data is stored rather than all at once after the fact.

Results Experiments reveal a counterintuitive finding: sophisticated memory systems with complex architectures fail to outperform a simple retrieval-augmented generation (RAG) baseline. The results show that over-designed structures and lossy compression hurt performance, and that preserving raw conversational text with high fidelity is more important than elaborate memory abstractions. Temporal retrieval is identified as a universal bottleneck across all systems tested.

Plain-language abstract GroupMemBench is a benchmark for testing how well AI agent memory systems handle conversations involving multiple people — such as workplace channels or group chats — rather than simple back-and-forth with a single user. It generates realistic multi-party conversations and poses questions that require knowing who said what, to whom, and in whose vocabulary, exposing gaps that existing benchmarks never measure.

Motivation AI memory systems and the benchmarks used to evaluate them are built around one-on-one conversations, yet real deployments increasingly involve groups where multiple users talk to each other and to the agent simultaneously. This mismatch leaves three important properties of group memory untested: the complex social dynamics of group dialogue, the need to track beliefs separately for each speaker, and the way vocabulary shifts depending on the audience — for example, the word 'token' means something different to an NLP engineer than to a systems engineer.

Methodology The benchmark is built in two stages. First, a graph-grounded synthesis pipeline creates multi-party conversations by encoding domains, topics, phases, and users in a directed knowledge graph, then generating messages conditioned on per-user personas and target audiences, with controllable reply structure to produce threaded debates and cross-topic shifts. Second, an adversarial query pipeline generates questions bound to specific asking users across six categories — multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention — and iteratively refines them through a Solve-Judge-Refine loop to ensure each query defeats a competent retrieval baseline before being accepted. The resulting corpus contains 120,000 turns across 123 topics.

Results Benchmarking five leading agent memory systems on GroupMemBench revealed a sharp performance collapse: the strongest system reached only 46.0% average accuracy, with knowledge update dropping to 27.1% and term ambiguity to 37.7%. A simple BM25 keyword-retrieval baseline, which performs no memory transformation at all, matched or exceeded most agent memory systems at effectively zero cost, Pareto-dominating four of the five systems. Failure analysis showed that retrieval failures — not reasoning failures — account for 41–79% of errors, meaning memory ingestion pipelines are discarding the structural and speaker-identity information that group memory depends on.

Plain-language abstract This paper introduces ImplicitMemBench, the first benchmark designed to test whether large language models can automatically adapt their behavior based on past experience — without being explicitly reminded to do so. It evaluates 17 AI models on 300 tasks drawn from three types of unconscious memory studied in cognitive science, finding that even the best models fall well short of human performance.

Motivation Existing benchmarks for AI memory focus on explicit recall — directly asking a model to retrieve a fact it was told. But effective assistants also need implicit memory: the ability to automatically apply a learned procedure or avoid a repeated mistake without being prompted. No prior benchmark measured this kind of unconscious behavioral adaptation, leaving a critical gap in understanding how well AI assistants actually learn from experience.

Methodology The benchmark operationalizes three cognitively grounded constructs drawn from nondeclarative memory research: Procedural Memory (one-shot skill acquisition that persists after distracting interference), Priming (theme-driven behavioral bias measured via paired experimental and control instances), and Classical Conditioning (whether stimulus-response pairings shape first decisions). All 300 items follow a unified Learning/Priming–Interfere–Test protocol scored on the first attempt, with passive scenario-driven triggers rather than explicit retrieval cues.

Results No model exceeded 66% overall accuracy. Top performers were DeepSeek-R1 at 65.3%, Qwen3-32B at 64.1%, and GPT-5 at 63.0%, all well below human baselines. Analysis revealed dramatic asymmetries between memory subtypes, with inhibition (avoiding a previously failed action) succeeding only 17.6% of the time versus preference-based priming at 75.0%, pointing to universal architectural bottlenecks that parameter scaling alone does not address.

Plain-language abstract MemFail is a diagnostic benchmark for testing where and why AI memory systems break down. When AI assistants need to remember information across long conversations, they rely on memory systems that compress, store, and retrieve facts. MemFail isolates exactly which part of that pipeline fails — summarization, storage, or retrieval — rather than just measuring whether a final answer is right or wrong.

Motivation AI agents increasingly depend on external memory systems to stay consistent across long interactions, but existing benchmarks only report overall question-answering accuracy and treat memory systems as black boxes. This makes it impossible to pinpoint whether a wrong answer came from the system losing information during compression, storing facts incorrectly, or retrieving the wrong memories — a gap that prevents targeted improvements to memory architectures.

Methodology The authors formalized memory systems as a composition of three operations — summarization, storage, and retrieval — and identified the failure modes each can introduce. They then built five adversarially designed datasets spanning four tasks (Conditional-Facts, Coexisting-Facts, Persona-Retrieval, and Long-Hop) to isolate these failure modes. Four state-of-the-art memory systems — Mem0, A-MEM, SimpleMem, and StructMem — were evaluated across these tasks using multiple underlying LLMs (GPT-4.1-mini, Haiku-4.5, GPT-5.4-mini, Gemini-3.1) and varying retrieval counts.

Results No single memory system dominated: graph-based StructMem excelled at causal reasoning tasks but failed on coexisting-fact retrieval, while Mem0 showed the opposite pattern. Scaling either the number of retrieved memories or the strength of the underlying LLM yielded little or no improvement, and sometimes degraded performance, indicating that current systems are limited by architectural constraints rather than model intelligence. The relationship between token usage and accuracy was task-dependent: tasks bottlenecked by summarization failures improved with more verbose storage, while retrieval-heavy tasks suffered when large memories polluted the embedding space.

Plain-language abstract This paper studies how AI systems can answer personal questions when the evidence they have access to is conflicting or incomplete — and when they should simply refuse to answer. The authors build a benchmark and compare several methods, ranging from simple baselines to large language models, on this task.

Motivation Personal AI agents increasingly maintain memory from multiple sources (calendars, messages, notes, etc.), and those sources often contradict each other. Existing benchmarks do not isolate whether a system fails because of bad evidence or because it cannot resolve conflicts, leaving a diagnostic gap that this work directly addresses.

Methodology The authors construct a benchmark of 34,560 instances built from 18 question templates spanning 8 reasoning types, 480 synthetic personas, and 4 random seeds, with controlled source distortions and deterministic ground truth. They evaluate baselines with no source access, single-source access, structured fusion methods, and frontier large language models, measuring both standard accuracy and selective accuracy under abstention.

Results The best trained fusion resolver achieves 80.3% accuracy overall, compared to 70.0% for the strongest prompt-only LLM baseline. When abstention is allowed, the fusion resolver reaches 85.3% selective accuracy at 78.3% coverage, while the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different model types show different strengths across the eight reasoning types. All data, code, and cached outputs are released for reuse.

Plain-language abstract This paper introduces AgenticAI-DialogGen, a software framework that automatically generates realistic, topic-organized conversations for training and testing AI language models on memory tasks. It also releases a new dataset called TopicGuidedChat (TGC) built from these generated conversations, which encodes both short-term and long-term memory cues for each pair of simulated speakers.

Motivation Existing conversational datasets lack the structure needed to train language models to reliably remember and reference information across a conversation — either they test short question-answering without capturing speaker personality, or they capture free-form dialogue without organizing it around coherent topics or memory. Creating such datasets manually is expensive and hard to do consistently, creating a gap that prevents progress on memory-aware conversational AI.

Methodology The framework takes an unstructured conversational dataset as input — primarily the Multi-Session Chat (MSC) dataset, restricted to 1,001 speaker pairs with four sessions each — and runs it through a pipeline of coordinated LLM-based modules and agents. These components preprocess conversations, extract factual knowledge triples, group them into topics, build per-speaker knowledge graphs, generate speaker personas, simulate multi-turn dialogues using LangGraph-based agents, validate and refine the output for topical adherence and quality, and finally produce memory-grounded question-answer pairs. The resulting TGC dataset stores long-term memory as structured knowledge graphs and short-term memory as simulated conversational turns.

Results Human and automatic evaluations show that AgenticAI-DialogGen improves discourse quality and topic coherence compared to baselines. Lightweight language models fine-tuned on the TGC dataset outperformed larger zero-shot models on memory-grounded tasks, demonstrating that the dataset's structured memory design provides practical utility for training memory-aware conversational systems.

Plain-language abstract MemLens is a benchmark designed to test how well large vision-language models (AI systems that handle both images and text) can remember and reason over long, multi-session conversations. It contains 789 questions that require retrieving and connecting visual and textual evidence spread across conversation histories ranging from 32,000 to 256,000 tokens, and it evaluates two competing approaches to giving these models long-term memory.

Motivation As AI assistants handle longer conversations involving both images and text, they need to reliably recall and reason over past interactions. Two approaches have emerged — expanding the model's native context window and using external memory stores with retrieval — but no existing benchmark directly compared the two on questions that genuinely require visual evidence. Prior text-only benchmarks ignored images entirely, and multimodal conversational benchmarks allowed most questions to be answered from text alone, leaving the harder multimodal memory problem unevaluated.

Methodology The authors constructed MemLens using a four-stage pipeline modeled on the needle-in-a-haystack paradigm: for each question, a coherent multi-session chat history is built with evidence distributed across one or more user-assistant sessions alongside topically related distractor turns. Questions span five memory abilities — information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal — at four standardized context lengths (32K, 64K, 128K, and 256K tokens) under a cross-modal token-counting scheme. They then evaluated 27 large vision-language models and 7 memory-augmented agents across all lengths, and ran an image-ablation study to confirm the benchmark requires visual evidence.

Results Removing evidence images caused accuracy for two frontier models to collapse below 2% on the 80.4% of questions that have image-based evidence, confirming the benchmark is genuinely multimodal. Long-context models performed well at shorter lengths through direct visual grounding but degraded as conversation length grew, while memory-augmented agents were more length-stable but lost visual fidelity when compressing memories at storage time. Multi-session reasoning capped most evaluated systems below 30% accuracy, and neither approach alone came close to solving the full benchmark, pointing toward hybrid architectures that combine long-context attention with structured multimodal retrieval.

Plain-language abstract MemEye is a benchmark and evaluation framework for testing how well AI agents remember and reason over visual information across long conversations. It introduces 371 question-answer pairs spanning eight everyday life scenarios, each tagged by how fine-grained the required visual evidence is and how complex the memory reasoning must be, then tests 13 different memory systems to reveal where they break down.

Motivation Existing long-term memory benchmarks for AI agents are mostly text-centric: even when images are present, many questions can be answered from captions or dialogue context alone, without retaining the original visual evidence. There is also little coverage of questions that require tracking how a visual scene changes over time. This leaves two core failure modes undiagnosed: agents losing fine-grained visual details during memory compression, and agents failing to synthesize a coherent current state when earlier observations are superseded by new ones.

Methodology The authors built a two-dimensional evaluation framework. The X-axis captures visual evidence granularity across four levels — scene-level, region-level, instance-level, and pixel-level — reflecting how much detail must be preserved from images. The Y-axis captures memory reasoning depth across three levels: atomic retrieval of a single fact, relational association of evidence across sessions, and evolutionary synthesis where the agent must resolve conflicts and track state changes. A benchmark of 371 mirrored multiple-choice and open-ended questions was constructed across 221 sessions, 848 dialogue rounds, and 438 images from eight life-scenario tasks. Questions were validated with ablation-driven gates to check answerability, shortcut resistance, visual necessity, and reasoning structure. Thirteen memory methods were evaluated across four vision-language model backbones.

Results No current memory system comes close to saturating the benchmark. The best-performing method, SRAG(V), achieved an LLM-as-a-Judge score of 0.49 on open-ended questions and an exact-match score of 0.62 on multiple-choice questions. Performance consistently degrades as visual granularity increases from scene-level to pixel-level and as reasoning depth increases from atomic retrieval to evolutionary synthesis. A key trade-off emerged: text-based memory methods manage state changes better but lose fine-grained visual details, while image-based memory methods preserve visual evidence but struggle with temporal validity. Cross-topic scaling experiments further showed that specialized memory mechanisms become more important as conversation history grows longer and more thematically diverse.

Plain-language abstract This paper surveys how researchers evaluate AI chatbots and assistants built on large language models when those systems must hold multi-turn conversations — exchanges that unfold over many back-and-forth messages rather than a single prompt. The authors reviewed nearly 250 published studies to map out what is being evaluated and how, producing two classification frameworks that together cover the full range of current evaluation practice.

Motivation AI conversational agents powered by large language models are increasingly deployed in customer service, personal assistants, and other settings that require sustained, context-aware dialogue. Despite this growth, the field lacked a systematic overview of how such multi-turn systems are evaluated — what dimensions matter and what measurement methods are available — leaving practitioners without a consolidated reference.

Methodology Using a PRISMA-inspired systematic review process, the authors examined nearly 250 scholarly sources from a range of publication venues. From this corpus they constructed two interrelated taxonomy systems: one defining what to evaluate (task completion, response quality, user experience, memory and context retention, planning and tool integration) and one categorizing how to evaluate (annotation-based evaluations, automated metrics such as BLEU and ROUGE, hybrid human-plus-quantitative strategies, and self-judging methods that use LLMs as evaluators).

Results The survey produced a structured, dual-taxonomy framework covering both evaluation dimensions and evaluation methodologies for LLM-based multi-turn conversational agents. The framework captures traditional language-understanding metrics alongside newer techniques suited to the dynamic, interactive nature of multi-turn dialogue, offering a consolidated foundation for researchers and practitioners assessing conversational AI systems.

Plain-language abstract This paper is a systematic survey of personalized dialogue generation — the field of building conversational AI systems that tailor their responses to individual users. The authors review 22 datasets, analyze 17 research papers published at top NLP conferences between 2021 and 2023, and summarize the evaluation metrics used across this body of work.

Motivation Personalization is increasingly important for conversational agents, especially as large language models can generate fluent but generic responses. The field lacks a unified definition of personalization — it can mean giving an agent a persona, modeling a user's traits, or both — and no comprehensive survey had catalogued the datasets, methods, and evaluation practices across this rapidly growing area.

Methodology The authors conducted a keyword-based literature search of top NLP venues (ACL, NAACL, EMNLP, AAAI, and others) covering 2021 through October 2023, selecting 17 seminal works. They systematically categorized 22 datasets by language, persona representation type (descriptive sentences or key-value pairs), data source (crowdsourcing or social platforms such as Reddit and Weibo), and features such as persona grounding labels and multi-session support. They then identified five distinct problem types across the surveyed methods and compiled a summary of evaluation facets and metrics.

Results The survey identifies five distinct problem formulations within personalized dialogue generation and highlights both benchmark datasets (such as PersonaChat and ConvAI2) and newer datasets with richer features including knowledge graphs, empathy signals, and out-of-distribution personas. The authors note strong language and domain biases across datasets — most are English or Chinese, and several are domain-specific rather than truly open-domain. The paper also surveys recent progress by large language models on personalized dialogue tasks and outlines open challenges and directions for future research.

Plain-language abstract StreamMemBench is a streaming benchmark that tests whether a personal-agent memory system can turn what it observes and how users interact with it into future-oriented assistance. Built on EgoLife egocentric lifelogs, it anchors each evaluation on a hidden piece of user-specific evidence and wraps a two-step task around it: an initial task that depends on the evidence, then a follow-up task that tests whether the agent reused both the evidence and the user's feedback. Four metrics diagnose evidence retention, feedback incorporation, initial evidence use, and follow-up reuse.

Motivation A central job of personal-agent memory is to carry stored observations and prior interactions forward into later, similar tasks, but existing memory benchmarks test dialogue recall or task improvement in isolation and usually rely on scripted or synthesized dialogues whose feedback is not tied to verifiable observations. That leaves the trajectory from streaming observations to later assistance largely untested — and even commercial assistants such as ChatGPT and Gemini store information that fails to help when it is actually needed.

Methodology In the construction stage an anchor agent processes five-minute EgoLife segments in stream order and extracts a user-specific evidence anchor plus two application-oriented queries, and a review agent retains a candidate only if both queries satisfy Leak=0 (the query does not reveal the evidence), Need=1 (ignoring the anchor yields a wrong or generic answer), and Natural=1 (it reads as a plausible request). In the evaluation stage the memory system ingests the lifelog chronologically, answers the initial task, receives confirming or correcting feedback from a user simulator, commits that interaction to memory, and then answers a follow-up task grounded in the same anchor; an evaluation agent scores Fidelity, Feedback Incorporation, Initial Evidence Use, and Follow-up Reuse against atom-level checklists, and the Fidelity−IEU / Fidelity−FUR gaps localize failures.

Results Across eight systems — two retrieval baselines (RAGraw, RAGext) and six memory systems (A-Mem, Mem0, EverMemOS, MemOS, MemoryOS, MemSkill) — on two backbones, current memory systems are not yet reliable for future-oriented assistance: they often fail to use evidence from egocentric observations in the initial task and to turn interaction feedback into reusable follow-up behavior. The failures are not explained by storage alone — systems frequently retain the evidence (with some Fidelity inflated by raw-text retention) yet do not use it — which motivates evaluation that traces each piece of evidence from its first appearance in the stream through initial use, feedback incorporation, and follow-up reuse.

Plain-language abstract MemTrace is a benchmark for testing how well an AI assistant remembers facts about a user across many conversations. Instead of scoring isolated questions, it tracks each individual fact—like a user's job title—and repeatedly asks about it under different conditions: long after it was mentioned, when it has since changed, or when the question contains false information. This reveals failures that a single overall accuracy score hides, such as a system that recalls a user's current role correctly but invents a false history of how they got there.

Motivation Long-term memory benchmarks usually aggregate accuracy over question rows or interaction episodes, treating questions that probe the same underlying fact as independent items. That makes it impossible to hold a fact fixed and ask how a system behaves as conditions around it change—whether it still recalls a fact after many sessions, whether it tracks how the fact evolved, and whether it behaves safely when evidence is missing or contradicted. Two systems with similar pooled scores can fail in entirely different ways, and aggregate scoring cannot show which.

Methodology MemTrace makes the knowledge point—a single typed fact about the user—the unit of measurement, and probes each fact along three controlled dimensions: memory age (how many sessions ago it appeared), question type (current state, an earlier state, or the trajectory of change), and evidence condition (present, missing, or contradicted by a false premise). It comprises 835 typed knowledge points from 20 users, expanded into 15,422 question rows and over 200,000 scored answers, and evaluates 13 memory-system configurations across four paradigms: long-context models, retrieval-augmented systems, external-memory stores, and agentic-memory architectures. A diagnostic step classifies each failure by whether the needed evidence was unreachable or reachable but unused.

Results Performance varies systematically across all three dimensions. Long-context systems answer recent facts well but lose accuracy as facts age, especially on trajectory questions; RAG systems, including graph-based retrieval, handle current and earlier-state questions better than questions about change over time; some external-memory systems decline almost all questions about facts that were never mentioned yet rarely correct a false premise. The dominant remaining bottleneck is evidence use, not retrieval: when systems fail, the evidence was already retrievable about 10× more often than it was missing, so improving memory depends on using reachable evidence rather than storing or retrieving more.

Plain-language abstract MemSyco-Bench is a benchmark that tests a specific failure mode in AI agents with long-term memory: sycophancy, where an agent trusts a memory of something the user said or believed before, even when that memory is outdated, out of scope, or contradicted by current evidence. Existing memory benchmarks mostly test whether an agent can retrieve the right memory; this one tests whether the agent uses a retrieved memory correctly once it has it.

Motivation The authors first show the problem is real: adding a plausible-but-wrong memory snippet before an objective factual question drops accuracy across all tested models and roughly doubles the rate at which the agent adopts the incorrect claim. They then show existing benchmarks (LongMemEval, LoCoMo, STALE, PersonaMem) can't isolate this: 47.4-66.1% of their errors come from failed retrieval, versus only 5.8-13.7% from correct-retrieval-but-wrong-use, so post-retrieval reasoning failures are essentially invisible in current scores.

Methodology MemSyco-Bench defines five task categories matched to the decisions an agent should make about a retrieved memory: reject it as factual evidence, respect its scope, resolve a conflict between memory and objective evidence, track that a memory has been updated, and use a genuinely valid memory for personalization. Multiple memory systems and backbone models are evaluated across these categories.

Results In the preliminary study, injecting a misleading memory cue drops factual accuracy on DeepSeek-V4-Flash from 56.1% to 40.2% and raises its sycophancy rate from 24.3% to 52.3%, with similar (smaller) effects on the other tested models. Full benchmark results show current memory systems generally increase sycophancy and struggle to balance personalization against factual reliability.

Plain-language abstract MemOps is a benchmark that reframes long-term conversational memory as a lifecycle of explicit operations (remembering, forgetting, updating, reflecting, and their compositions) rather than as static question answering. Each memory event is a structured trace specifying its trigger, target, scope, state transition, and supporting evidence. A controllable pipeline embeds these operations into long, task-oriented conversations, producing gold operation traces and six categories of operation-level probes evaluated under both adjacent-evidence and long-context settings. Across long-context, retrieval-based, parametric, and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.

Motivation Existing long-term memory benchmarks such as LoCoMo, LongMemEval, MemBench, and others evaluate almost exclusively through downstream question answering, scoring only the correctness of a final answer. That black-box formulation conflates heterogeneous causes of failure (missing the introduction of a relevant fact, binding an operation to the wrong target, or relying on a stale value after a correction) and can credit a correct answer that rests on an inconsistent or unsafe memory state. This is most acute in dynamic long-horizon interaction, where memory functions as a lifecycle process: a user introduces a fact, corrects it, asks that part be forgotten, or implicitly signals a preference, each of which is a distinct operation with its own trigger, target, scope, state transition, and characteristic failure modes that downstream QA cannot diagnose.

Methodology Each instance is a tuple of a topic-specific user background, a set of evidence conversations, a gold operation trace, and evaluation probes. The trace is the evaluation anchor: each operation carries a type, target object, old and new value, and evidence spans quoted verbatim from user turns. Five operation types are defined, including TrajectoryOps that compose remember, update, forget, and reflect events across time so the benchmark can check intermediate states and their temporal order. A four-stage generation pipeline (background construction; evidence conversation and gold-trace generation; operation-level probe generation; long-context dialogue generation with distractor pools) produces natural conversations with explicit operation supervision, verified by local schema and span gates plus an LLM verifier. Six probe categories (operation trace, target binding, state transition, candidate disambiguation, operation application, state trajectory) score intermediate memory behavior, and representative systems from four paradigms are evaluated under adjacent-evidence and long-context settings.

Results Performance is strongest when evidence sits adjacent to the query and degrades consistently once it is dispersed into long, distractor-laden histories, for both answer accuracy and operation-level reliability. Session-level retrieval substantially outperforms turn-level retrieval, and managed-memory services that preserve longer, context-rich memory units outperform those storing short, isolated facts, indicating that surrounding context is critical for correctly executing lifecycle operations rather than merely retrieving relevant content. Parametric memory, which folds interaction history into model parameters, remains markedly unreliable across almost all diagnostic dimensions. Reconstructing an ordered memory-state trajectory across multiple composed operations is far more fragile than any single-step operation, and this weakness persists even for otherwise strong long-context models, exposing a failure mode that final-answer accuracy alone would not surface.

Plain-language abstract RECON (Reasoning over Extended Contexts with Obfuscated Narratives) is a benchmark that tests whether an agent's memory can maintain a coherent, evolving understanding over long contexts where facts do not just accumulate but interact, contradict, and cascade. It spans 24 investigative case files across criminal, medical, and financial domains, each 50k to 100k tokens, with 1,604 questions over six memory-intensive tasks, and reports that even the strongest non-Oracle system reaches only 22.4% accuracy.

Motivation Existing memory benchmarks model memory as a state machine: they check whether an agent can retrieve a scattered fact or detect that a fact changed. Real workflows demand more. When a lab result is revised on Day 9, a witness statement is contradicted on Day 5, or a flagged transaction is reversed, an agent must trace which downstream conclusions are affected, which survive on independent support, and how an alternative timeline would have unfolded. RECON targets what happens after a change, not just whether the current value is tracked.

Methodology A deterministic pipeline constructs each case from a provenance DAG, then a question generator traverses the DAG to synthesize questions per task category, with a fixed question matrix enforcing per-task and per-format quotas so the distribution does not drift between runs. The six tasks are chain reconstruction (order 5-15 evidence hops), cascade propagation (which conclusions break versus survive after an invalidation), source-conflict resolution, counterfactual reasoning, temporal-constraint satisfaction, and temporal fact retrieval as a control. Solvers - long-context, RAG, hybrid RAG, Supermemory, Mem0, Mem0-Graph, Hindsight, closed-book, and an Oracle handed the full structured ground truth - share one answering template; freeform answers are graded by two independent LLM judges from different model families and averaged, while other formats are scored deterministically with an explicit abstain option (+1 correct, -0.2 wrong, 0 abstain).

Results Even the Oracle given the ground-truth dependency graph reaches only 54.6% accuracy, and the best non-Oracle system 22.4%; RAG attains 20.6% on full-coverage retrieval hits. Decomposing failures shows roughly four in five persist even when retrieval succeeds, placing the bottleneck in reasoning rather than evidence selection. Human annotators reach 63.0% accuracy, exceeding the Oracle solver by 8.4 points and confirming reasoning, not retrieval, as the residual challenge across long-context, retrieval, and agentic-memory architectures.

Plain-language abstract A benchmark and method for a memory capability separate from recall: keeping track of which version of a fact is currently in force after it has been revised across sessions. The benchmark, StateMemBench, generates each scenario as a symbolic program of state operations so the correct answer is computed by replay and the specific way a lazy reader would get it wrong is known in advance. The method, StateMem, parses each turn into structured state units with typed dependency links, then handles supersession and staleness deterministically rather than with an LLM.

Motivation Existing memory systems and benchmarks optimize recall of relevant facts, but as agents run longer, facts, constraints and decisions get revised, and an answer must reflect the current state rather than a superseded one. The authors call the failure state drift: the relevant fact is present in the assembled context, but the agent acts on a stale or incomplete version of it. This is distinct from dialogue state tracking, which prescribes a slot-value representation and evaluates it directly over cooperative dialogues that accumulate a goal monotonically; here state is whatever a system must maintain to answer correctly, evaluation is purely behavioral, and revisions are adversarial across sessions. Some concurrent work centers state, but the authors argue none cleanly isolates state tracking from the other errors it co-occurs with.

Methodology The authors first define and label drift on existing benchmarks, assigning a failure to drift only after excluding retrieval, comprehension, schema and reasoning readings, dropping unassignable points, and cross-checking with two judge passes, a cross-family judge and two human annotators. They then build StateMemBench: each scenario is a symbolic event program of typed operations over ground, derived and declared state; the gold answer comes from deterministic replay; a family of executable lazy reader policies is run against the replay, and a scenario is admitted as a trap when policies disagree, with the disagreeing set forming its failure-mode signature (status, salience, sequence, compound, plus anti-trap controls). Programs are grounded in public data for surface vocabulary and rendered into multi-session dialogue by a strong LLM, then programmatically verified for fact placement and phrase leakage. Probes are closed-pool: an unseen pool of three to four options holds the gold answer, the targeted policy's drift answer and neutral distractors. StateMem itself runs a per-turn TurnEncoder producing state units (id, content, priority, source, deps), a deterministic update stage applying supersessions and marking dependents needs_recheck by dependency-graph traversal, and a single answer-time call over the assembled active state. A wrapper variant applies the same trace-then-resolve structure as a prompt-level transformation of any backend's answer call, evaluated against a length- and cost-matched generic-extraction control.

Results Drift leads the confirmed failure distribution on several existing benchmarks (63.5% on MemoryArena-shopping, 44.4% on LongMemEval oracle where retrieval is perfect by construction, 10 of 16 on tau-squared-bench-Z) but is not universal, falling to 19.0% on MemoryArena-travel. On StateMemBench, long-context is not the strong baseline it is on recall tasks: the best long-context model reaches 0.277 and same-backbone long-context 0.149. StateMem reaches 0.363 on DeepSeek-V4-Flash, 1.8x the best memory system and 2.4x same-backbone long-context, and 0.233 on Qwen-3.5-9B, 1.6x the best memory system, both significant by paired McNemar at p < 0.001; GraphRAG at 0.224 is statistically level on Qwen. Ablations put supersession marking as the largest single component and show dependency propagation over-fires on Set B anti-traps by 12.5 points, so removing it leaves DeepSeek slightly better. Drift-rate analysis shows the memory layer barely changes outcomes on a weak answerer, where every arm drifts at 61 to 66%, and separates on the stronger backbone, where StateMem's drift rate falls 15 points and correct answers rise by 42 while long-context, Mem0 and BM25 move by 1 to 3 points. The wrapper improves every one of six backends on both benchmarks, adding 31.7 to 66.6 points on StateMemBench with 15.0 to 31.7 attributable to state structure over the matched control, significant in all twelve cells. State tracking does not cost recall: StateMem also leads memory systems on LongMemEval (0.656 on DeepSeek) and LoCoMo (0.592), with margins concentrated on temporal-reasoning and knowledge-update question types. The authors note StateMem mirrors the policy family behind the traps, so its StateMemBench margins should be read as an upper bound.

Plain-language abstract A benchmark for the decision an agent makes before it writes to memory: should this piece of interaction-derived information be persisted durably, used only in the current context, re-checked against the world, or clarified with the user. It evaluates three models from two families under three prompt conditions, and separately tests whether the stated decision predicts which memory tool the model actually calls.

Motivation Persistent memory personalizes an agent, but an incorrect durable update silently distorts future behavior. A temporary request should not become a standing preference, a service status can go stale, one tool failure may be noise, and an underspecified correction may need a question before it is generalized. The authors argue the critical capability is not recall but commitment, and that two sources of uncertainty are usually collapsed: verification queries the world, which is authoritative for changing facts, while clarification queries the user, who is authoritative for intent and scope. Prior work covers adjacent targets (binary session-level storage gating, ADD/UPDATE/DELETE/NOOP operations, whether retrieved memory grounds tool parameters, general ambiguity clarification) without jointly distinguishing durable storage, local use, world verification and user clarification at commitment time.

Methodology Each item supplies an acquire context, a candidate update and a later reuse context, with a gold action assigned by released rules and a tie between persist and a weaker action resolved toward the weaker commitment. 140 primary scenarios are split 70/70 by sorted identifier within category, covering stable and episodic preferences, freshness-sensitive facts, one-off corrections, policy constraints, ambiguous updates and noisy failures at 20 items each, with eight lexical traps where a surface cue points to the wrong action. Two non-authors labeled the held-out and contrast items blind to author labels and to each other, with a blind third resolving ties. Claude Haiku 4.5, Claude Sonnet 4.6 and a locally served Qwen3.5-9B (Q4_K_M via Ollama, temperature 0, seed 13, thinking disabled) were each run under three conditions: a bare prompt defining the actions, a policy prompt adding five commitment rules including the tie-breaker, and a four-shot prompt with one development example per action. A separate track, MCB-Act, removes the label vocabulary and requires one structured tool call, scored by mapping the selected tool to an action. Analysis uses accuracy with bootstrap intervals, macro-F1, over-memory, under-memory and per-class recalls, with exact paired McNemar tests and Holm correction within each family of comparisons.

Results Both model families under-ask. Claude label-mode verification recall runs 0.889 to 1.000 while clarification recall runs 0.500 to 0.750; bare Qwen verifies 12 of 18 freshness items and asks on 0 of 12 clarification items. Few-shot prompting lifts Qwen accuracy from 0.557 to 0.771 (paired delta +0.214, p_H = 0.002) and clarification recall from 0 to 0.333, still missing 8 of 12. The policy prompt raises Qwen accuracy by only 0.071 (p_H = 0.539) but cuts erroneous persistence from 0.243 to 0.100 (p_H = 0.038), moving the uncertainty to verification (recall 0.667 to 0.944) rather than to the user (clarification 0 to 0.083). Haiku's policy and few-shot gains survive correction (p_H = 0.002 and 0.047); Sonnet's do not, so the benchmark measures a prompt-conditioned commitment policy rather than a fixed model trait. Label-to-tool agreement is 0.571 for each Claude model and 0.229 for Qwen; Sonnet accuracy falls from 0.814 to 0.529 (p_H < 0.001) and Qwen from 0.557 to 0.343 (p_H = 0.047), with Qwen calling use_now on 54 of 70 items and verification recall collapsing to 0.056. All emitted arguments pass the deterministic well-formedness rules, locating the bottleneck in tool choice. On the combined 140 Qwen items of the contrast extension, bare, policy and few-shot accuracy is 0.614, 0.757 and 0.843, with clarification remaining the weakest class throughout; the authors retain the extension as a controlled sensitivity check rather than a claim of naturalistic external validity, since its rule-authored templates align closely with the explicit policy rules. MCB-Act scores tool-call selection and does not execute downstream effects.

Plain-language abstract SkillEvolBench is a benchmark designed to test whether AI agents can turn their own task experience into reusable procedural skills — not just replay what they did once, but extract general instructions a future agent can follow on related tasks. It contains 180 tasks across six real-world work domains and evaluates ten large language model configurations under three agent frameworks, comparing skill-based conditions against baselines that use no skills or raw trajectory replay.

Motivation LLM agents accumulate detailed records of how they solved past tasks, but it is unclear whether that experience can be distilled into compact, reusable procedures rather than task-specific patches. Prior work either studied skill use (given curated skills) or experience reuse (replaying trajectories), leaving a gap: can agents themselves convert noisy one-off episodes into external skill artifacts that help on harder, related tasks they have not yet seen?

Methodology The benchmark organizes 180 tasks into six environments covering code modification, API orchestration, data processing, document transformation, research synthesis, and communication operations, with five task families per environment. Each family follows a six-role arc: three acquisition tasks (canonical, enriched, variant) used to build the skill library, and three frozen deployment tasks (context-shift, adversarial, composition) where no further skill updates are allowed. Agents operate in Self-Generated or Curated-Start settings; a separate Skill Author step uses compacted trajectories and structured verifier feedback to decide whether to write a new skill, revise an existing one, or leave the library unchanged. Controls include a No-Skill condition and a Raw-Trajectory condition that provides the original episode directly.

Results Across ten model configurations and three agent harnesses, current agents adapt locally but rarely form robust reusable skills. Raw-trajectory reuse frequently outperforms distilled skills, indicating that current abstraction procedures discard contextual and procedural cues that remain useful for future tasks. The always-update curated variant showed the strongest average gain (+0.78 percentage points in frozen evaluation success rate), but benefits were model-specific: GPT-5.4 improved by +6.7 percentage points and Opus 4.5 by +4.4 percentage points, while Gemini 2.5 Pro was harmed by most variants (average -3.70 percentage points). Adding more skills or larger resource libraries did not reliably help and often introduced episode-specific drift and procedural clutter without improving deployment success.

Plain-language abstract Synthius-Mem is a memory system for AI agents that stores what is known about a person rather than replaying raw conversation history. Drawing on how the human brain organizes memory into specialized subsystems, it decomposes conversations into six typed domains — biography, experiences, preferences, social circle, work, and psychometrics — and retrieves structured facts at query time. On the LoCoMo benchmark it achieves 94.37% overall accuracy and 99.55% adversarial robustness while consuming roughly five times fewer tokens than full-context replay.

Motivation Current LLM memory approaches — sliding windows, summarization, embedding-based retrieval, and flat fact extraction — each reduce token cost but introduce information loss, semantic drift, or uncontrolled hallucination about the user. No published system on the LoCoMo benchmark reported adversarial robustness, meaning the ability to refuse questions about facts the user never disclosed. The paper addresses this gap by arguing that domain-agnostic storage discards the structural advantages that neuroscience shows different memory types require.

Methodology Synthius-Mem uses a multi-stage pipeline: conversations are parsed and chunked, then six parallel domain-specific extractors pull typed facts into separate stores for biography, experiences, preferences, social circle, work, and psychometrics (the last using nine validated psychological frameworks). Each domain undergoes its own consolidation and deduplication pass. At inference time, a planner LLM selects relevant domains and a CategoryRAG component retrieves structured facts with a reported mean latency of 21.79 ms. The system was evaluated on the LoCoMo benchmark (ACL 2024), which contains 10 multi-session conversations and 1,813 questions spanning single-hop recall, multi-hop reasoning, temporal reasoning, open-domain knowledge, and adversarial false-premise questions.

Results Synthius-Mem achieves 94.37% accuracy on LoCoMo, exceeding the prior best published system MemMachine (91.69%) and human performance (87.9 F1). Core memory fact accuracy reaches 98.64%. Adversarial robustness — refusal to hallucinate facts the user never disclosed — reaches 99.55%, a metric no competing system reports. Token consumption is approximately five times lower than full-context replay at 500 messages, where full-context replay itself achieves only 85.46% in the authors' controlled evaluation.

Plain-language abstract This paper introduces LifeDialBench, a new benchmark for testing how well AI memory systems can handle continuous spoken conversations recorded by wearable devices like smart glasses. Unlike existing benchmarks that focus on chatting directly with an AI, this one evaluates memory over the kind of multi-person, everyday dialogues that an always-on microphone would capture. The paper also proposes a new evaluation protocol that respects the time-ordering of events, mimicking how such a system would work in practice.

Motivation Current AI memory benchmarks focus on one-on-one person-AI chat sessions and do not address the emerging use case of wearable devices that continuously record ambient conversations throughout the day. Existing approaches to extending memory — such as feeding ever-longer text into a model — become prohibitively expensive and still underperform. There was no public benchmark designed for the continuous, multi-person, time-stamped dialogue streams that wearable lifelogging devices produce.

Methodology The authors built LifeDialBench from two complementary data sources: EgoMem, derived from real-world egocentric video recordings (transcribed via automatic speech recognition), and LifeMem, constructed through a hierarchical life-simulation framework using a virtual community with human-in-the-loop review. Question-answer pairs cover four types — single-event, event detail, multi-event, and temporal information queries — spanning timescales from 30-second clips to week-level summaries. Crucially, the paper introduces an Online Evaluation protocol that enforces temporal causality, evaluating memory systems incrementally as data is stored rather than all at once after the fact.

Results Experiments reveal a counterintuitive finding: sophisticated memory systems with complex architectures fail to outperform a simple retrieval-augmented generation (RAG) baseline. The results show that over-designed structures and lossy compression hurt performance, and that preserving raw conversational text with high fidelity is more important than elaborate memory abstractions. Temporal retrieval is identified as a universal bottleneck across all systems tested.

Plain-language abstract This paper surveys how researchers evaluate AI chatbots and assistants built on large language models when those systems must hold multi-turn conversations — exchanges that unfold over many back-and-forth messages rather than a single prompt. The authors reviewed nearly 250 published studies to map out what is being evaluated and how, producing two classification frameworks that together cover the full range of current evaluation practice.

Motivation AI conversational agents powered by large language models are increasingly deployed in customer service, personal assistants, and other settings that require sustained, context-aware dialogue. Despite this growth, the field lacked a systematic overview of how such multi-turn systems are evaluated — what dimensions matter and what measurement methods are available — leaving practitioners without a consolidated reference.

Methodology Using a PRISMA-inspired systematic review process, the authors examined nearly 250 scholarly sources from a range of publication venues. From this corpus they constructed two interrelated taxonomy systems: one defining what to evaluate (task completion, response quality, user experience, memory and context retention, planning and tool integration) and one categorizing how to evaluate (annotation-based evaluations, automated metrics such as BLEU and ROUGE, hybrid human-plus-quantitative strategies, and self-judging methods that use LLMs as evaluators).

Results The survey produced a structured, dual-taxonomy framework covering both evaluation dimensions and evaluation methodologies for LLM-based multi-turn conversational agents. The framework captures traditional language-understanding metrics alongside newer techniques suited to the dynamic, interactive nature of multi-turn dialogue, offering a consolidated foundation for researchers and practitioners assessing conversational AI systems.

Plain-language abstract MemTrace is a framework for automatically finding and explaining errors in the memory systems used by AI language model agents. When a memory-augmented agent gives a wrong answer, MemTrace traces the sequence of memory operations — storing, updating, retrieving, deleting — to pinpoint exactly which step introduced the failure and why. The authors also release MemTraceBench, a benchmark of 160 annotated real failure cases, and show that fixing identified errors can improve end-task accuracy by up to 7.62%.

Motivation Language model agents increasingly rely on persistent memory to handle long conversations and multi-session tasks, but when these systems fail it is very hard to tell where things went wrong. Unlike simpler stateless agents where errors are usually local to one step, memory failures can originate in an early storage or update operation and only become visible much later during retrieval or answer generation. Existing memory benchmarks only measure whether a system succeeds or fails overall; they do not recover the causal chain that led to the failure, leaving a traceability gap that makes debugging slow and unreliable.

Methodology The authors instrument the source code of a memory system to record every memory update, memory read, and answer-generation step as it runs. These records are assembled into an execution graph — a directed acyclic bipartite graph whose nodes are variables (raw messages, retrieved memory units, summaries, prompts) and operations (LLM inference, retrieval, filtering, parsing), connected by directed edges that capture information flow. Given a failed case, the MemTrace attribution method retrieves relevant source messages and then iteratively traces operation subgraphs to locate the earliest minimal set of faulty operations whose correction would have prevented the failure. MemTraceBench was constructed from four representative memory systems (LongContext, RAG, Mem0, EverMemOS) and three public datasets (LoCoMo, LongMemEval, RealMem), yielding 160 human-annotated failure cases each with QA pairs, execution logs, ground-truth error labels, faulty operations, and human explanations.

Results Experiments on MemTraceBench show that diagnosing memory failures remains challenging, but MemTrace successfully recovers meaningful faulty operations and error types and generates coherent explanations suitable for debugging. The analysis reveals that memory failures are systematic, most commonly arising from operation-level issues such as information loss and retrieval misalignment. Using the fine-grained attribution signals from MemTrace to guide automatic prompt optimization produces a closed-loop correction system that improves end-task performance by up to 7.62%.

Plain-language abstract This paper introduces StratMem-Bench, a benchmark for testing how well AI-powered virtual characters use memory strategically in conversation. Rather than just checking whether a chatbot recalls facts, the benchmark asks whether it knows which memories to use, which to optionally enrich a response with, and which to ignore entirely — mirroring how humans selectively deploy information in dialogue.

Motivation Existing benchmarks for memory-augmented AI treat memory as a static fact store and evaluate only whether relevant facts are recalled — a property called factual recall. This is insufficient for human-like virtual characters, where good responses also require selectively incorporating supportive personal context and actively suppressing irrelevant information. No prior benchmark captured this distinction.

Methodology The authors built a dataset of 657 conversation instances in which a virtual character must respond given a heterogeneous memory pool partitioned into three types: required memories (must), supportive memories (nice), and irrelevant memories (irr). They evaluated state-of-the-art LLMs as virtual characters in zero-shot mode using four metrics: Strict Memory Compliance (SMC, a pass/fail rule-based check), Memory Integration Quality (MIQ, an LLM-judged 1–5 Likert quality score designed to be failure-sensitive), Proactive Enrichment Score (PES, measuring tendency to incorporate nice memories), and Conditional Irrelevance Rate (CIR, measuring how often irrelevant memories intrude).

Results All evaluated models perform well at separating required from irrelevant memories, but none achieves above 50% SMC once supportive memories are introduced into the pool. Integration quality scores drop noticeably when nice memories are used (MIQ around 3.9–4.4 versus 4.2–4.6 for must memories) and collapse severely when irrelevant memories are incorporated (MIQ 2.4–3.1). A clear proactivity–risk aversion trade-off emerges: Gemini 3 Pro reaches the highest enrichment rate (~73% PES) but also the highest irrelevance contamination rate (~47% CIR in nice-only scenarios), while GPT-5-chat stays conservative (~8% CIR) at the cost of rarely enriching responses.

Plain-language abstract AuthTrace is a benchmark for testing how well AI systems gather and organize evidence when answering questions about a single author's body of work. It uses 860 public-domain essays by five modern Chinese prose writers to create 2,099 question-answer instances, each tagged with exactly how many source documents are needed to support the answer. The benchmark lets researchers compare four different evidence-organization strategies — retrieval, memory compression, graph traversal, and thematic indexing — on the same dataset using the same metrics.

Motivation AI evidence-construction systems (retrieval, memory, graph, and structured indexing) are benchmarked in isolation, each with its own corpus and metrics, so when a system fails there is no principled way to tell whether the problem is missing evidence, noisy evidence, or poor answer synthesis. Single-author corpora are a particularly hard test case because distractors share the same style, vocabulary, and recurring themes as the correct evidence, making it difficult to distinguish direct support from mere thematic adjacency.

Methodology The benchmark is built from public-domain nonfiction prose by Lu Xun, Zhou Zuoren, Zhu Ziqing, Xiao Hong, and Yu Dafu — 860 articles totaling roughly 2 million Chinese characters — processed through a staged LLM-assisted annotation pipeline. Each instance provides a query, quoted gold evidence spans linked to source documents, atomic gold claim units, a reference answer, and an exact fan-in label (the count of distinct source documents required). Instances are grouped into Single-doc (fan-in = 1), Low multi-doc (fan-in = 2), and High multi-doc (fan-in >= 3). All systems are evaluated with a unified pack-level protocol measuring Evidence Recall, Evidence Precision, and Answer Correctness scored by an LLM judge.

Results Evidence recall was the strongest observed predictor of answer correctness across eight systems and two QA models, with a correlation of r = 0.96; most failures stem from missing evidence rather than flawed answer synthesis. Flat retrieval degrades 2–3 times faster than thematically organized evidence construction as fan-in increases. Graph retrieval performs best at single-document grounding (fan-in = 1) while thematic indexing leads on multi-document synthesis (fan-in >= 2), with a crossover at fan-in 2, providing a practical selection heuristic for practitioners.

Plain-language abstract When an AI system answers a question by drawing on a document, current tools simply label the answer as either 'supported' or 'unsupported.' This paper argues that label is too coarse: a verbatim quote, a paraphrase, a deduction, and an assumption-laden inference are all technically 'supported' but require very different levels of reader scrutiny. The authors propose building a richer taxonomy of support relations that would let interfaces show not just whether a claim is grounded, but exactly how it connects to the source.

Motivation Binary groundedness labels obscure the wide variety of ways a language model can transform source text into an answer — from direct quotation to paraphrase to inductive inference — and may lull readers into passive acceptance rather than critical engagement. Survey evidence already shows that higher confidence in AI is associated with less critical thinking. Without finer-grained provenance information, users cannot easily judge which claims deserve scrutiny and which can be trusted as verbatim.

Methodology This is a position and research-agenda paper rather than an empirical study. The authors demonstrate the insufficiency of binary labels through worked examples tested on four commercial systems, then sketch a construction path for a reader-centred taxonomy: a structured literature review drawing on Toulmin's argumentation model, Gricean pragmatics, and scholarly discourse ontologies (e.g., ScholOnto) to produce candidate support relations; iterative refinement guided by discriminability (can annotators reliably distinguish categories?) and actionability (do distinctions matter for provenance interfaces?); and validation via a human annotation study on statement-source pairs enriched from existing hallucination benchmarks, with interannotator agreement measuring construct validity.

Results The paper does not report experimental measurements; it is a proposal. Its key contribution is a framework articulating a taxonomy of support relations — including direct quotation, paraphrase, deductive support, inductive support, and support contingent on ancillary assumptions — and an evaluation roadmap involving human annotation benchmarks and LLM-as-judge protocols. The authors argue this taxonomy would enable interfaces that communicate the type of grounding behind each AI-generated claim, supporting more informed critical reading.

Plain-language abstract This paper examines whether open-source language models fine-tuned specifically to act as evaluators can replace GPT-4 when judging the quality of other AI language model outputs. The authors run a systematic series of experiments comparing several fine-tuned judge models against GPT-4 across multiple benchmarks, then propose a hybrid method that mixes the two approaches to cut costs while preserving accuracy.

Motivation Evaluating large language models is expensive and raises privacy concerns when it depends on proprietary APIs such as GPT-4. Researchers had proposed fine-tuning smaller open-source models as local substitutes, and those models reported accuracy matching GPT-4 on their own test sets — but those benchmarks closely resembled the training data. No systematic study had tested whether fine-tuned judge models actually generalize beyond their training distribution.

Methodology The authors benchmarked four representative fine-tuned judge models — JudgeLM, PandaLM, Auto-J, and Prometheus — across dimensions including generalizability to out-of-domain data, fairness (bias toward superficial response features), ability to evaluate specific quality aspects, and responsiveness to prompt engineering. They then analyzed the softmax probability distributions of the fine-tuned judges to build a confidence indicator, and used it to design CascadedEval: a hybrid pipeline that routes low-confidence samples to GPT-4 for re-evaluation while handling high-confidence samples locally.

Results Fine-tuned judge models surpassed GPT-4 on their respective in-domain test sets but fell short on every other dimension tested: generalizability, fairness, aspect-specific evaluation, and prompt-engineering responsiveness. The analysis showed these models behave as task-specific classifiers overfitted to their training data rather than as general evaluators. CascadedEval achieved accuracy on par with GPT-4 on both in-domain and out-of-domain test sets while using only 50% of the GPT-4 API calls, cutting the cost by half.

Plain-language abstract This paper is a comprehensive survey of systems where large language models (LLMs) are used as automated evaluators — a setup called LLM-as-a-Judge. It catalogs existing approaches, proposes a formal definition and taxonomy, identifies key reliability challenges, and introduces a benchmark for measuring how trustworthy these judge systems are. The survey covers strategies for making LLM judges more consistent and less biased, as well as practical applications across domains from NLP to law and science.

Motivation Evaluating AI-generated outputs at scale is difficult: expert human review is costly and inconsistent, while automated metrics like BLEU and ROUGE miss nuance and fail on open-ended tasks such as story generation. LLM-as-a-Judge emerged as a way to combine the scalability of automatic metrics with the contextual reasoning of human experts, but the field lacked formal definitions, systematic reviews, and agreed methods for assessing whether these judge systems are actually reliable.

Methodology The authors conduct a systematic literature review of the LLM-as-a-Judge paradigm, organizing existing work into a formal taxonomy that spans in-context learning approaches, model selection, post-processing strategies, and evaluation pipelines. They analyze strategies for improving judge reliability — including prompt engineering, fine-tuning, and bias mitigation — and examine metrics and datasets used to assess judge quality. They also design a novel benchmark specifically for evaluating the reliability of LLM-as-a-Judge systems, and survey practical deployments across machine learning, finance, law, and scientific domains.

Results The survey finds that LLMs such as GPT-4 can evaluate text generation tasks at a level comparable to humans, and that LLM-as-a-Judge has been successfully applied across NLP tasks including summarization, dialogue, and reasoning. The paper identifies persistent challenges — position bias, inconsistency, and adversarial vulnerability — and maps out mitigation strategies. It also documents the growing use of LLM judges in reinforcement learning from human feedback pipelines and in guiding reasoning path selection in multi-agent systems. The accompanying benchmark and taxonomy are offered as a reference infrastructure for the field.

Plain-language abstract This paper is a comprehensive survey of using large language models (LLMs) as automatic evaluators — a practice called "LLMs-as-judges." Instead of relying on human annotators or simple metrics to assess AI-generated text, researchers are increasingly asking LLMs themselves to rate or compare outputs. The survey maps out the entire landscape of this approach: why it is used, how evaluation systems are built with it, where it is applied, how to assess its quality, and what its known failure modes are.

Motivation Traditional evaluation metrics like BLEU and ROUGE often fail to capture important qualities of generated text such as fluency, logical coherence, and creativity. Human annotation is accurate but slow and expensive to scale. This created a gap: a need for evaluation methods that are flexible, interpretable, generalizable across tasks, and cheap enough to run at scale — a gap the LLMs-as-judges paradigm aims to fill.

Methodology The authors conduct a structured literature survey organized around five key perspectives: Functionality (why use LLM judges), Methodology (how to build evaluation systems using LLMs, including single-LLM approaches with prompt engineering and fine-tuning, multi-LLM systems, and human-AI collaboration), Application (domains where LLM judges are deployed), Meta-evaluation (methods for assessing judge quality), and Limitations (known biases and failure modes). The survey also covers how LLM judges are used in data annotation and synthesis workflows.

Results The survey identifies and taxonomizes a wide range of biases that afflict LLM judges, including position bias, verbosity bias, self-enhancement bias, sentiment bias, token bias, overconfidence bias, and diversity bias tied to demographic identity markers. It finds that prompt template choice alone can lead to inconsistent or biased assessments, and that LLMs can inherit implicit biases from pretraining data. The paper maps future research directions and maintains an open-source resource list cataloguing work in this rapidly growing area.

Plain-language abstract This paper is a comprehensive survey of how LLM-based agents — AI systems that go beyond single-step text generation by planning, using tools, and maintaining memory across multiple steps — are evaluated. It catalogs the benchmarks, environments, and frameworks used to assess these agents across a wide range of tasks and domains.

Motivation Standard language model benchmarks were designed for single-turn, text-to-text tasks, but LLM-based agents operate in multi-step, dynamic environments where they plan, call external tools, and adapt to feedback. No comprehensive mapping of evaluation methodologies existed for this broader class of systems, leaving researchers and practitioners without a clear guide to what benchmarks exist, what they measure, and where the gaps are.

Methodology The authors systematically analyze the evaluation landscape across four dimensions: (1) fundamental agent capabilities such as planning, multi-step reasoning, tool use, self-reflection, and memory; (2) application-specific benchmarks for web agents, software engineering agents, scientific agents, and conversational agents; (3) benchmarks for generalist agents and leaderboards; and (4) agent evaluation frameworks used throughout the development cycle, including tools such as LangSmith, Langfuse, and BrowserGym.

Results The survey identifies an emerging trend toward more realistic and continuously updated benchmarks that pose harder challenges. It also surfaces critical gaps in the field: existing evaluations largely neglect cost-efficiency, safety, robustness, and fine-grained or scalable assessment methods. The authors propose these as priority directions for future research in agent evaluation.

Plain-language abstract AlphaEval is a benchmark designed to test AI agent products under conditions that actually resemble real business use, rather than the idealized settings of typical research benchmarks. It contains 94 tasks drawn directly from seven companies that deploy AI agents in their core operations, covering six occupational domains. The paper also introduces a systematic process for turning genuine business requirements into reproducible evaluation tasks, and uses the benchmark to compare six frontier language models deployed through four commercial agent products.

Motivation Existing agent benchmarks like SWE-bench and WebArena are built from retrospectively curated tasks with clear requirements and deterministic metrics — conditions that do not match how agents are actually used in production. A survey of 27 AI product companies found that 63% have low confidence that model updates genuinely improve their products, 25.9% have no explicit evaluation criteria, and 70.4% rely on developers testing as a side activity. This gap between research benchmarks and production reality was the central problem the paper set out to address.

Methodology The authors partnered with seven companies and AI-focused organizations to collect 94 authentic production tasks spanning six O*NET occupational domains. Tasks were formalized through a four-stage pipeline: partner engagement, requirement elicitation, task formalization, and iterative validation. The evaluation framework combines multiple paradigms — LLM-as-a-Judge, reference-driven metrics, formal verification, rubric-based assessment, and automated UI testing — with individual domains composing several paradigms simultaneously. Six frontier models (including Claude Opus 4.6, GPT-5.2, and Gemini 3 Pro Preview) were evaluated across four commercial agent scaffolds (Claude Code, Codex, GitHub Copilot, and Cursor), yielding 14 model-scaffold configurations.

Results The best-performing configuration (Claude Code with Opus 4.6) achieved only 64.41 out of 100, indicating a substantial gap between frontier agent capability and production requirements. Scaffold choice proved as consequential as model choice: the same Opus 4.6 model scored 64.41 via Claude Code but only 53.45 via Codex, an 11-point difference. Domain performance varied widely, from 62.0 on Technology Research tasks to 30.0 on Human Resources tasks, showing that no single aggregate score captures production readiness. The paper also identifies six recurring production-specific failure modes through systematic error analysis.

Plain-language abstract WebForge is a fully automated system for building benchmarks that test AI agents that control web browsers. It generates complete, interactive fake websites from scratch — no human annotation required — and includes realistic web nuisances like pop-ups and cookie dialogs. Using this system, the authors built WebForge-Bench, a set of 934 tasks across 7 domains and 3 difficulty levels, and used it to evaluate multiple AI models.

Motivation Existing browser-agent benchmarks are caught in a trilemma: real-website benchmarks are realistic but become outdated as web content changes (nearly half of one benchmark's tasks expired within two years), while controlled sandbox environments are reproducible but unrealistically clean and require expensive manual curation. No prior benchmark simultaneously achieved realism, reproducibility, and scalability, and none offered fine-grained, multi-dimensional control over task difficulty.

Methodology WebForge uses a four-agent pipeline — Plan, Generate, Refine, and Validate — to produce self-contained static websites end-to-end. The Plan Agent designs tasks using a seven-dimensional difficulty vector (covering navigation depth, visual complexity, reasoning difficulty, and other axes) at three levels each. The Generation Agent builds functional HTML/CSS/JS websites populated with real data collected from the web. The Refinement Agent injects realistic noise such as pop-ups, cookie dialogs, and network delays. The Validation Agent checks that each task is actually solvable. The resulting environments require no external services and can be run by opening an HTML file.

Results On WebForge-Bench (934 tasks), difficulty stratification effectively separated model capabilities: Level 1 tasks were solved by at least one model 95% of the time, while Level 3 tasks remained unsolved by all 14 evaluated models 23.5% of the time. Cross-model experiments revealed distinct capability profiles invisible to aggregate scores — for example, the strongest model (Gemini-3-Pro, 75.9% overall) dropped 35 percentage points on Visual Complexity tasks from Level 1 to Level 3, while a mid-tier model (Gemini-3-Flash, 67.1%) dropped 42 percentage points on both Information Complexity and Reasoning tasks. The weakest model (GPT-5-Nano, 31.3%) achieved near-zero accuracy on all dimensions at Level 3.

Plain-language abstract This paper introduces NumLeak, a measurement framework for detecting when large language models have memorized public numeric benchmark data — such as financial return series, unemployment figures, and climate records — and are recalling those memorized values rather than reasoning from scratch. It tests several top-tier models (including Claude Opus, Sonnet, Haiku, and GPT-5.4) and shows that the best models reproduce historical data with near-perfect fidelity, raising serious concerns about evaluations that use those same public datasets.

Motivation Public numeric datasets like the Fama-French factor library, macroeconomic releases, and NOAA temperature records are widely mirrored online and likely appear in the pretraining corpora of large language models. If a model can retrieve historical values simply from a date and a series name, then any evaluation or financial strategy that conditions on those dates may be measuring memorized recall rather than genuine predictive skill — a form of data contamination that prior work on verbatim text extraction had not specifically addressed for continuous, date-indexed numeric series.

Methodology NumLeak combines API-boundary behavioral probes on closed production models with a controlled white-box validation on an open model. Four diagnostic metrics are used jointly: Pearson correlation with published ground truth, mean absolute error, within-25-basis-point accuracy, and sign accuracy, along with parse rate to track refusal behavior. The identification protocol applies factor-specificity contrasts, temporal stratification, fabricated-series probes, and rank/value decoupling tests. For white-box validation, the authors LoRA fine-tune Qwen-2.5-1.5B-Instruct on a synthetic date-indexed series (SMR-A) at four exposure levels (0x, 1x, 5x, 20x mentions per date-value pair) and probe using both open-ended generation and direct log-probability inspection.

Results Top-tier frontier models recall Fama-French market excess returns at 3-seed pooled Pearson r=0.97-0.99, with comparable fidelity on U.S. unemployment, CPI inflation, and NOAA temperature. Recall weakens monotonically with model capability within each provider (Opus 4.7 at r=0.99, Sonnet 4.6 at 0.97, Haiku 4.5 at 0.57). On a recent-release holdout, parse rate collapses to 21-57% but correlation stays near 0.99 on the months the model does answer, consistent with a memorization channel bounded by training data availability. A system-prompt defense blocks 99.8% of a non-adaptive single-turn suffix attack at near-zero utility cost, and a regression using a model's own date-to-market-sentiment output collapses from r=0.74 to r=0.02 once the model's recall is residualized out.

Plain-language abstract Modern AI assistants often store conversation history as multiple forms — raw dialogue turns, extracted facts, summaries, and timelines — all linked to the same original source. When benchmarks test how well these systems retrieve relevant memories, they must choose which stored form counts as a correct answer. This paper shows that this seemingly minor choice — called the scoring target — is routinely left unspecified, and that changing it can flip which system looks better without changing how either system actually behaves.

Motivation Conversational memory benchmarks are increasingly used to guide design decisions about how AI systems store and retrieve dialogue history. When a memory system transforms raw conversation turns into facts or summaries, multiple stored versions of the same evidence exist in the same index. Benchmarks implicitly choose which version to credit as correct, but this choice is almost never stated explicitly — creating a hidden variable that can change benchmark conclusions without changing the underlying retrieval system.

Methodology The authors introduce TIAP, a fixed-output audit procedure that rescores already-saved ranked retrieval outputs under three different scoring targets — Raw (only the original source turn gets credit), Source (any stored descendant linked to the same source gets credit), and Canonical (only transformed serving-form memories get credit) — without re-running retrieval. They apply this to two benchmark datasets, LoCoMo and LongMemEval-S, across four retrieval providers, and also test transfer to two external memory architectures, Mem0 and MemoryOS. A 1,902-case semantic audit using five large language models as judges assessed whether relaxed credit was semantically warranted.

Results Switching only the scoring target changed nDCG scores on 83.4% to 94.0% of shared queries across the two benchmarks and four retrievers. The same fixed ranked outputs flipped the winner ordering between memory architectures (Mem0 and MemoryOS) depending on which target was used, and reversed recommendations about parser density (how many facts to extract per turn). The semantic audit found that relaxed source-linked credit was fully supported by the retrieved content only 29.2% of the time under a five-model majority vote, with 31.2% of relaxed credits entirely unjustified. The authors conclude that benchmark conclusions about memory system design can silently reverse based solely on which stored form is credited, and recommend that all conversational-memory papers explicitly define and report their scoring target.

Plain-language abstract MemConflict is a diagnostic framework for testing whether AI conversational agents can correctly handle conflicting information in their long-term memory. It introduces a benchmark where user facts change over time, contradict each other, or apply only under certain conditions, then measures whether six representative memory systems retrieve and use the right information to answer questions.

Motivation Conversational AI agents rely on long-term memory to track user-specific facts across many sessions, but user information is not static: it evolves, can be contradicted, and may apply only in certain contexts. Existing evaluations mostly measure whether a system gives the right final answer, which obscures whether failures come from retrieving the wrong memory, ranking it too low, or failing to use a correctly retrieved memory. No prior framework systematically diagnosed these distinct failure modes under conflicting memory candidates.

Methodology The authors define three conflict types — dynamic (a later update supersedes an older state), static (a false contradiction should not overwrite a stable fact), and conditional (multiple memories are valid under different conditions) — grounded in an information-quality notion of fitness-for-use. They simulate long-horizon multi-session conversation histories from structured user profiles, inject cross-session conflicts, and add semantically similar distractors to create competition among memory candidates. A two-level evaluation protocol combines black-box scoring of final answers with white-box inspection of which memory items were retrieved and ranked, applied to six representative long-term memory systems.

Results Experiments on six systems show uneven strengths across conflict types, with no system dominating all three. Answer correctness frequently diverges from memory retrieval quality, revealing a measurable Evidence Utilization Gap. Retrieval failures account for the majority of errors across most systems and conflict types, but non-trivial utilization failures also appear — notably, LangMem under dynamic conflicts more often retrieves the relevant memory but still produces a wrong answer than it fails to retrieve it at all. Performance degrades with longer histories, more distractors, implicit queries, and larger conflict distances between conflicting memory entries. MemOS shows the strongest overall performance and efficiency, while Letta is competitive on conditional conflicts and LangMem is strongest on dynamic conflicts.

Plain-language abstract Engram is an open-source long-term memory engine for LLM agents. Instead of replaying an entire conversation history into the prompt, it stores the past as a bi-temporal knowledge graph and retrieves a small, precisely-targeted slice at answer time. On a standard 500-question memory benchmark that lean ~9.6k-token slice answers more accurately than feeding the model the full ~79k-token history, turning memory from a cost optimization into an accuracy improvement.

Motivation Stateless LLM agents forget across sessions, and the usual fix — concatenating the whole history — grows token cost and latency linearly and loses accuracy as distractors crowd the window ('lost in the middle'). Two gaps stay open: most memory systems are cheaper or faster but not more accurate than full-context, and memory benchmarks run on inconsistent harnesses where one system reports wildly different scores across sources. Engram targets both — beating full-context on accuracy, and shipping a neutral, re-runnable harness.

Methodology A dual-process design. A System-1 hot write path appends lossless episodes with no LLM (sub-50ms) and enqueues them. A System-2 async path extracts atomic (subject,predicate,object) facts, builds a bi-temporal knowledge graph (valid time vs transaction time on every fact and edge), detects conflicts, and resolves them cheap-then-escalate: exact slot match, embedding similarity, and content subsumption handle the common case with no LLM call, invalidating (never deleting) a superseded fact and recording a supersedes chain and provenance, with only ambiguous cases escalated to an LLM adjudicator. The hybrid read path retrieves through four channels (dense, BM25, graph n-hop, recency/salience), fuses them with Reciprocal Rank Fusion, applies an 'as-of' temporal filter and an abstention gate, and assembles a deduplicated, provenance-tagged, token-budgeted context of facts plus raw chunks.

Results On the full 500-question LongMemEval_S under the official category-specific judge, Engram's lean configuration scores 83.6% vs 73.2% for full-context (+10.4 points, McNemar exact p<10^-6) at ~8x fewer tokens (9.6k vs 79k), 0/500 errored. The gain is load-bearing on the read path being hybrid: facts alone lose recall, while facts plus retrieved chunks recover detail. Bi-temporal modeling pays off most on knowledge-update (87.5%) and temporal (81.1%) categories, while multi-session aggregation and preference remain headroom. The paper documents measurement-integrity pitfalls (truncation, home-grown judges, full-history leaks) and ships a neutral in-repo harness with the official judge baked in and raw per-question logs, every number reproducible by command.

Plain-language abstract GitOfThoughts stores an LLM agent's reasoning tree as a git repository — every scored thought is a commit, scores are git notes, validation outcomes are tags, and retrieval is git log over the agent's own history — which makes reasoning replayable, auditable, diffable, and mergeable across agents. The paper then asks the harder question of whether memory, in any substrate, actually improves accuracy, and runs a pre-registered comparison of five substrates (none, markdown, vector, graph, git).

Motivation Reasoning is the last unversioned software process: chains of thought expire with the context window, pruned search branches leave no record, and memory buffers cannot be diffed, merged, or audited. The authors argue this ephemerality is a structural blocker — it prevents reproducibility ('what did the agent think at step 17?'), audit (detecting train–test leakage or gold-answer memorization), memory transfer between agents, and incident review. Code, infrastructure, datasets, and experiments are all version-controlled; reasoning is the remaining outlier.

Methodology A reasoning tree shares git's structural invariants, so the paper maps it one-to-one onto git primitives (node = commit, refinement = parent edge, score = note, outcome = tag, session vs. cross-session = branch, retrieval = git log --grep / -S). A pluggable MemoryBackend routes every read/write through one interface so the same agent can swap substrate with a one-line change. To isolate retrieval from write-path noise, all five backends ingest identical answer-free lessons, then solve held-out, domain-stratified problems read-only; benchmarks are GPQA-Diamond and MATH-500, scored with paired-bootstrap CIs, across two backbones and pre-registered replications, with a similarity sweep to locate when retrieval helps.

Results H-substrate is supported: git delivers auditability, line-level diffs over reasoning text, deterministic replay by SHA, and mergeable memory at accuracy parity, costing ~15 ms/write and ~48 ms/read. H-memory is rejected: across two benchmarks, two backbones, and up to n=500, no substrate reliably improves accuracy on novel problems, and a +15 pp git trend at n=40 collapsed under its pre-registered replication. Memory pays only above a 'copyability threshold' — a near-duplicate retrieved case (cosine ≳ 0.8) lifts accuracy +12 to +13.5 pp, and a 4.5× larger model steepens that step to +22.5–28.5 pp but still extracts no transferable method; the only general accuracy lever is test-time sampling (self-consistency, +3.4 pp at n=500). The authors deliberately document a measurement bug, a retracted result, and a refuted hypothesis as the evaluation standard.

Plain-language abstract StreamMemBench is a streaming benchmark that tests whether a personal-agent memory system can turn what it observes and how users interact with it into future-oriented assistance. Built on EgoLife egocentric lifelogs, it anchors each evaluation on a hidden piece of user-specific evidence and wraps a two-step task around it: an initial task that depends on the evidence, then a follow-up task that tests whether the agent reused both the evidence and the user's feedback. Four metrics diagnose evidence retention, feedback incorporation, initial evidence use, and follow-up reuse.

Motivation A central job of personal-agent memory is to carry stored observations and prior interactions forward into later, similar tasks, but existing memory benchmarks test dialogue recall or task improvement in isolation and usually rely on scripted or synthesized dialogues whose feedback is not tied to verifiable observations. That leaves the trajectory from streaming observations to later assistance largely untested — and even commercial assistants such as ChatGPT and Gemini store information that fails to help when it is actually needed.

Methodology In the construction stage an anchor agent processes five-minute EgoLife segments in stream order and extracts a user-specific evidence anchor plus two application-oriented queries, and a review agent retains a candidate only if both queries satisfy Leak=0 (the query does not reveal the evidence), Need=1 (ignoring the anchor yields a wrong or generic answer), and Natural=1 (it reads as a plausible request). In the evaluation stage the memory system ingests the lifelog chronologically, answers the initial task, receives confirming or correcting feedback from a user simulator, commits that interaction to memory, and then answers a follow-up task grounded in the same anchor; an evaluation agent scores Fidelity, Feedback Incorporation, Initial Evidence Use, and Follow-up Reuse against atom-level checklists, and the Fidelity−IEU / Fidelity−FUR gaps localize failures.

Results Across eight systems — two retrieval baselines (RAGraw, RAGext) and six memory systems (A-Mem, Mem0, EverMemOS, MemOS, MemoryOS, MemSkill) — on two backbones, current memory systems are not yet reliable for future-oriented assistance: they often fail to use evidence from egocentric observations in the initial task and to turn interaction feedback into reusable follow-up behavior. The failures are not explained by storage alone — systems frequently retain the evidence (with some Fidelity inflated by raw-text retention) yet do not use it — which motivates evaluation that traces each piece of evidence from its first appearance in the stream through initial use, feedback incorporation, and follow-up reuse.

Plain-language abstract MemTrace is a benchmark for testing how well an AI assistant remembers facts about a user across many conversations. Instead of scoring isolated questions, it tracks each individual fact—like a user's job title—and repeatedly asks about it under different conditions: long after it was mentioned, when it has since changed, or when the question contains false information. This reveals failures that a single overall accuracy score hides, such as a system that recalls a user's current role correctly but invents a false history of how they got there.

Motivation Long-term memory benchmarks usually aggregate accuracy over question rows or interaction episodes, treating questions that probe the same underlying fact as independent items. That makes it impossible to hold a fact fixed and ask how a system behaves as conditions around it change—whether it still recalls a fact after many sessions, whether it tracks how the fact evolved, and whether it behaves safely when evidence is missing or contradicted. Two systems with similar pooled scores can fail in entirely different ways, and aggregate scoring cannot show which.

Methodology MemTrace makes the knowledge point—a single typed fact about the user—the unit of measurement, and probes each fact along three controlled dimensions: memory age (how many sessions ago it appeared), question type (current state, an earlier state, or the trajectory of change), and evidence condition (present, missing, or contradicted by a false premise). It comprises 835 typed knowledge points from 20 users, expanded into 15,422 question rows and over 200,000 scored answers, and evaluates 13 memory-system configurations across four paradigms: long-context models, retrieval-augmented systems, external-memory stores, and agentic-memory architectures. A diagnostic step classifies each failure by whether the needed evidence was unreachable or reachable but unused.

Results Performance varies systematically across all three dimensions. Long-context systems answer recent facts well but lose accuracy as facts age, especially on trajectory questions; RAG systems, including graph-based retrieval, handle current and earlier-state questions better than questions about change over time; some external-memory systems decline almost all questions about facts that were never mentioned yet rarely correct a false premise. The dominant remaining bottleneck is evidence use, not retrieval: when systems fail, the evidence was already retrievable about 10× more often than it was missing, so improving memory depends on using reachable evidence rather than storing or retrieving more.

Plain-language abstract SEAGym is a testing environment for AI agents that improve themselves over time by editing their own setup—their prompts, memory, skills, tools, and configuration—rather than by retraining the underlying model. Instead of just checking whether a self-improved agent scores higher on a final task, SEAGym measures the improvement process itself: whether each self-edit actually helps on new tasks, whether gains stick or later collapse, and what they cost.

Motivation Self-evolving agents improve mainly by changing their agent harness—the structured execution layer around a base model. Existing evaluations reduce this to isolated task scores or a single sequential learning curve, which obscures whether a given update produces reusable improvement, overfits the recent tasks, increases cost, or harms older behavior. Most agent benchmarks are built for static evaluation, resetting agent state between isolated episodes and removing exactly the state persistence that self-evolution depends on.

Methodology SEAGym uses an RL-style environment formulation in which the self-evolving agent supplies both the task policy and the harness-update rule, while the environment defines task sampling, feedback, schedules, and snapshot assessment. It converts Harbor-compatible static benchmarks into reusable task sources organized into train batches and frozen evaluation views—update-validation, held-out in-domain transfer, out-of-domain transfer, replay diagnostics, and cost records—and saves agent snapshots and metric artifacts. Explicit schedule parameters (state reset, task reuse, batch size, update timing) let single-task adaptation, online transfer, and epoch-based batch learning be studied under one protocol. Task rollout is separated from method update, so methods such as ACE, TF-GRPO, and AHE connect through thin wrappers while keeping their native update rules; experiments instantiate the environment on Terminal-Bench 2.0 and HLE.

Results The evaluation views provide complementary signals about the evolution process rather than one summary number. Frequent updates may fail to improve held-out performance, validation gains do not always transfer to in-domain or out-of-domain test views, and useful intermediate snapshots can collapse later or recover. Batch size, source diversity, and the rollout model backend all affect harness reliability, showing that self-evolution dynamics depend on the evaluation schedule and setup, not just the update method.

Plain-language abstract MemOps is a benchmark that reframes long-term conversational memory as a lifecycle of explicit operations (remembering, forgetting, updating, reflecting, and their compositions) rather than as static question answering. Each memory event is a structured trace specifying its trigger, target, scope, state transition, and supporting evidence. A controllable pipeline embeds these operations into long, task-oriented conversations, producing gold operation traces and six categories of operation-level probes evaluated under both adjacent-evidence and long-context settings. Across long-context, retrieval-based, parametric, and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.

Motivation Existing long-term memory benchmarks such as LoCoMo, LongMemEval, MemBench, and others evaluate almost exclusively through downstream question answering, scoring only the correctness of a final answer. That black-box formulation conflates heterogeneous causes of failure (missing the introduction of a relevant fact, binding an operation to the wrong target, or relying on a stale value after a correction) and can credit a correct answer that rests on an inconsistent or unsafe memory state. This is most acute in dynamic long-horizon interaction, where memory functions as a lifecycle process: a user introduces a fact, corrects it, asks that part be forgotten, or implicitly signals a preference, each of which is a distinct operation with its own trigger, target, scope, state transition, and characteristic failure modes that downstream QA cannot diagnose.

Methodology Each instance is a tuple of a topic-specific user background, a set of evidence conversations, a gold operation trace, and evaluation probes. The trace is the evaluation anchor: each operation carries a type, target object, old and new value, and evidence spans quoted verbatim from user turns. Five operation types are defined, including TrajectoryOps that compose remember, update, forget, and reflect events across time so the benchmark can check intermediate states and their temporal order. A four-stage generation pipeline (background construction; evidence conversation and gold-trace generation; operation-level probe generation; long-context dialogue generation with distractor pools) produces natural conversations with explicit operation supervision, verified by local schema and span gates plus an LLM verifier. Six probe categories (operation trace, target binding, state transition, candidate disambiguation, operation application, state trajectory) score intermediate memory behavior, and representative systems from four paradigms are evaluated under adjacent-evidence and long-context settings.

Results Performance is strongest when evidence sits adjacent to the query and degrades consistently once it is dispersed into long, distractor-laden histories, for both answer accuracy and operation-level reliability. Session-level retrieval substantially outperforms turn-level retrieval, and managed-memory services that preserve longer, context-rich memory units outperform those storing short, isolated facts, indicating that surrounding context is critical for correctly executing lifecycle operations rather than merely retrieving relevant content. Parametric memory, which folds interaction history into model parameters, remains markedly unreliable across almost all diagnostic dimensions. Reconstructing an ordered memory-state trajectory across multiple composed operations is far more fragile than any single-step operation, and this weakness persists even for otherwise strong long-context models, exposing a failure mode that final-answer accuracy alone would not surface.

Plain-language abstract A language model's memory can be worse than no memory at all. Give a model a memory that kept a wrong conclusion but dropped the work behind it and it re-emits the stale value as a confident answer; give the same model an empty memory and it abstains. The paper names this failure brittle memory, measures it with a reclaim-evaluation protocol that tests whether a correction can recover a known answer after compression, and shows a one-line fix (keep the recomputable source, drop the re-derivable conclusion) restores correctability at equal memory budget.

Motivation Memory systems carry information across sessions by compressing it, on the implicit assumption that a compression preserving the model's answer has preserved what matters. The paper shows the same compression decides whether the model can later be corrected: once the answer-determining source is gone, a correction has nothing to act on, and the resulting error compounds as deployed agents feed memory into memory.

Methodology Reclaim evaluation drifts a model into committing to a wrong answer via a planted premise, deepens the commitment over neutral turns, then issues a correction in a fresh session whose only inheritance is a memory written under one of three matched-budget policies: lossy (keep the salient conclusion, shed the source), source-first (keep the source, shed the conclusion), and lossy-padded (lossy plus neutral filler to at least source-first's length, controlling for budget). Success is exact recovery of the known answer, with no judge. Tasks are multi-step arithmetic and constraint-logic puzzles with objectively scorable answers; the pipeline runs end to end on llama-3.1-8b and grok-4.3 with a frontier replay on Claude models; headline cells are n=96, and three validators designed to fail against a deterministic fake all pass.

Results Within one conversation there is no wall, only anchoring: a directed correction holds far longer than a generic one (reclaim 0.79 to 0.50 over eight commitment turns) and pushing the error further back lifts reclaim rather than starving it. Across a session boundary the window becomes a wall: once the lossy note drops the source line items, even a directed correction dies, reclaim is 0.00 by measurement, and a lossy memory is worse than an empty one because models that abstain with nothing emit the confident wrong value with a source-less note. The wall sits in the same place from the 8B model to frontier systems. Source-first restores reclaim at equal budget (oracle 1.00; the deployable one-prompt distiller 0.49-0.88, concentrated on compact numeric sources), and the length-matched control rules out added text as the cause. Chained through a memory loop, one dropped-source error corrupts a growing span of downstream steps and stays uncorrectable however late it is caught, while source-first holds to a bounded budget horizon; the wall and the fix replicate on three deployed memory systems and on MultiWOZ, and past the budget where the source no longer fits, the fix fails silently unless the note records its own completeness.

Plain-language abstract RECON (Reasoning over Extended Contexts with Obfuscated Narratives) is a benchmark that tests whether an agent's memory can maintain a coherent, evolving understanding over long contexts where facts do not just accumulate but interact, contradict, and cascade. It spans 24 investigative case files across criminal, medical, and financial domains, each 50k to 100k tokens, with 1,604 questions over six memory-intensive tasks, and reports that even the strongest non-Oracle system reaches only 22.4% accuracy.

Motivation Existing memory benchmarks model memory as a state machine: they check whether an agent can retrieve a scattered fact or detect that a fact changed. Real workflows demand more. When a lab result is revised on Day 9, a witness statement is contradicted on Day 5, or a flagged transaction is reversed, an agent must trace which downstream conclusions are affected, which survive on independent support, and how an alternative timeline would have unfolded. RECON targets what happens after a change, not just whether the current value is tracked.

Methodology A deterministic pipeline constructs each case from a provenance DAG, then a question generator traverses the DAG to synthesize questions per task category, with a fixed question matrix enforcing per-task and per-format quotas so the distribution does not drift between runs. The six tasks are chain reconstruction (order 5-15 evidence hops), cascade propagation (which conclusions break versus survive after an invalidation), source-conflict resolution, counterfactual reasoning, temporal-constraint satisfaction, and temporal fact retrieval as a control. Solvers - long-context, RAG, hybrid RAG, Supermemory, Mem0, Mem0-Graph, Hindsight, closed-book, and an Oracle handed the full structured ground truth - share one answering template; freeform answers are graded by two independent LLM judges from different model families and averaged, while other formats are scored deterministically with an explicit abstain option (+1 correct, -0.2 wrong, 0 abstain).

Results Even the Oracle given the ground-truth dependency graph reaches only 54.6% accuracy, and the best non-Oracle system 22.4%; RAG attains 20.6% on full-coverage retrieval hits. Decomposing failures shows roughly four in five persist even when retrieval succeeds, placing the bottleneck in reasoning rather than evidence selection. Human annotators reach 63.0% accuracy, exceeding the Oracle solver by 8.4 points and confirming reasoning, not retrieval, as the residual challenge across long-context, retrieval, and agentic-memory architectures.

Plain-language abstract Procedural skills for LLM agents are normally judged by average improvement in task success, which hides the tasks a skill library breaks. Running the same agent with and without skills across 5,832 paired task-condition runs on two office-automation benchmarks and three model-harness stacks, the paper decomposes each pass-rate change into gains and regressions. Regressions offset 59% of gross gains, and the best libraries lead mainly by regressing less rather than gaining more. Three regression mechanisms are identified from traces - skill-description osmosis, grounding displacement and verification displacement - and the residual failures cluster at grounding and verification, the stages existing skills least often address.

Motivation Systems that generate skills automatically from execution traces are evaluated by average gain, so two libraries with identical averages can differ sharply in how many working tasks they destroy. Methods that detect and drop harmful skills identify which skill to remove but not why it regressed a task, so a mostly useful skill may be discarded wholesale. Those methods also act only when a skill is retrieved or invoked, while a skill's name and description remain in the system prompt at every step and can change behavior without ever being called - a channel selection-based approaches cannot observe.

Methodology Two benchmarks over real artifacts: OfficeQA-Pro (question answering over U.S. Treasury financial documents, where the hard part is finding the right table, vintage and definition rather than the arithmetic) and SpreadsheetBench (workbook manipulation from real Excel-forum problems, graded on exact cell values). Three stacks pair a harness with its usual model: OpenCode / minimax-m2.7, Codex / gpt-5.4-mini, Claude Code / sonnet-4.6. Each stack and benchmark runs under four conditions - none, plus three libraries built by different meta-skill creators from the same analyst-extracted failure signals, so only the library text changes within a stack. Every task falls into gain, regression, residual failure or retained against the no-skill baseline; net effect is gains minus regressions, tested with the exact two-sided McNemar test on discordant pairs with Newcombe 95% intervals. Skill invocation is read from the trajectory so presence-only influence can be separated from body engagement. Each condition is run once per task, so run-to-run variance is not estimated.

Results Regression counts range from 2 to 41 across the eighteen library conditions and never reach zero; 324 regression transitions cancel 59% of 553 gains, with the pattern holding on both benchmarks (66% of gains cancelled on OfficeQA-Pro, 56% on SpreadsheetBench). Even clearly positive cells are close-run: on SpreadsheetBench the openai library broke 41 while gaining 45 on one stack and broke 27 while gaining 30 on another. Five of eighteen conditions reach nominal p < .05, but only three survive Bonferroni correction, all on Claude Code / sonnet-4.6 with SpreadsheetBench - a real but narrow effect. Grounding displacement accounts for 72.8% of the 81 coded OfficeQA-Pro regressions and osmosis for 17.3%; on SpreadsheetBench osmosis accounts for 70 of 243 regressions against 46 with the body engaged and 32 attributable to a grader artifact. Body engagement is set by the harness rather than the skill: the same library is invoked on 73-91% of OfficeQA-Pro tasks but 5-14% of SpreadsheetBench tasks on one stack, and on 99-100% versus 0-1% on another. Correcting the known grader artifact required re-grading 226 treatment task-conditions with a full spreadsheet engine, since the default engine cannot recalculate Excel structured references and scores correct formulas as failures.

Plain-language abstract A re-evaluation of memory-based self-improving web agents (Agent Workflow Memory, ReasoningBank) across WebArena, VisualWebArena, and SCUBA, adding multiple runs and randomized task ordering to standard single-run, fixed-order evaluation. Both additions expose that reported improvements are considerably less stable than prior single-run reports suggested.

Motivation Self-improving agents that write and reuse a textual memory bank across a stream of tasks are promoted as a path to reliable long-horizon automation, but their reliability under realistic conditions — repeated runs, unordered task streams — had not been stress-tested. In enterprise deployment, tolerance for early failure is low and mistakes compound silently.

Methodology The authors re-run Agent Workflow Memory and ReasoningBank with a stronger base model and harness, then extend evaluation along two axes: multiple runs of the same setting to quantify variance, and randomly shuffled task orders to test sensitivity to the default (often implicitly easy-to-hard) task curriculum. They then manually inspect agent-written memories to identify failure drivers, and test whether adding rubrics and environment feedback to memory construction narrows the gap.

Results Variance across runs increases in 71% of settings when a self-improving loop is stacked on a baseline agent; on WebArena's GitLab subset, the best/worst run gap widens from 4.4% (no memory) to 7.8% (ReasoningBank). Under the default task order, ReasoningBank gains 1.5% on average; under random shuffling, it degrades by 4.5%. Manual inspection attributes much of this to task and environment underspecification, which causes agents to write plausible but inapplicable memories; adding rubrics and environment feedback partially, but not fully, closes the degradation.

Plain-language abstract A benchmark and method for a memory capability separate from recall: keeping track of which version of a fact is currently in force after it has been revised across sessions. The benchmark, StateMemBench, generates each scenario as a symbolic program of state operations so the correct answer is computed by replay and the specific way a lazy reader would get it wrong is known in advance. The method, StateMem, parses each turn into structured state units with typed dependency links, then handles supersession and staleness deterministically rather than with an LLM.

Motivation Existing memory systems and benchmarks optimize recall of relevant facts, but as agents run longer, facts, constraints and decisions get revised, and an answer must reflect the current state rather than a superseded one. The authors call the failure state drift: the relevant fact is present in the assembled context, but the agent acts on a stale or incomplete version of it. This is distinct from dialogue state tracking, which prescribes a slot-value representation and evaluates it directly over cooperative dialogues that accumulate a goal monotonically; here state is whatever a system must maintain to answer correctly, evaluation is purely behavioral, and revisions are adversarial across sessions. Some concurrent work centers state, but the authors argue none cleanly isolates state tracking from the other errors it co-occurs with.

Methodology The authors first define and label drift on existing benchmarks, assigning a failure to drift only after excluding retrieval, comprehension, schema and reasoning readings, dropping unassignable points, and cross-checking with two judge passes, a cross-family judge and two human annotators. They then build StateMemBench: each scenario is a symbolic event program of typed operations over ground, derived and declared state; the gold answer comes from deterministic replay; a family of executable lazy reader policies is run against the replay, and a scenario is admitted as a trap when policies disagree, with the disagreeing set forming its failure-mode signature (status, salience, sequence, compound, plus anti-trap controls). Programs are grounded in public data for surface vocabulary and rendered into multi-session dialogue by a strong LLM, then programmatically verified for fact placement and phrase leakage. Probes are closed-pool: an unseen pool of three to four options holds the gold answer, the targeted policy's drift answer and neutral distractors. StateMem itself runs a per-turn TurnEncoder producing state units (id, content, priority, source, deps), a deterministic update stage applying supersessions and marking dependents needs_recheck by dependency-graph traversal, and a single answer-time call over the assembled active state. A wrapper variant applies the same trace-then-resolve structure as a prompt-level transformation of any backend's answer call, evaluated against a length- and cost-matched generic-extraction control.

Results Drift leads the confirmed failure distribution on several existing benchmarks (63.5% on MemoryArena-shopping, 44.4% on LongMemEval oracle where retrieval is perfect by construction, 10 of 16 on tau-squared-bench-Z) but is not universal, falling to 19.0% on MemoryArena-travel. On StateMemBench, long-context is not the strong baseline it is on recall tasks: the best long-context model reaches 0.277 and same-backbone long-context 0.149. StateMem reaches 0.363 on DeepSeek-V4-Flash, 1.8x the best memory system and 2.4x same-backbone long-context, and 0.233 on Qwen-3.5-9B, 1.6x the best memory system, both significant by paired McNemar at p < 0.001; GraphRAG at 0.224 is statistically level on Qwen. Ablations put supersession marking as the largest single component and show dependency propagation over-fires on Set B anti-traps by 12.5 points, so removing it leaves DeepSeek slightly better. Drift-rate analysis shows the memory layer barely changes outcomes on a weak answerer, where every arm drifts at 61 to 66%, and separates on the stronger backbone, where StateMem's drift rate falls 15 points and correct answers rise by 42 while long-context, Mem0 and BM25 move by 1 to 3 points. The wrapper improves every one of six backends on both benchmarks, adding 31.7 to 66.6 points on StateMemBench with 15.0 to 31.7 attributable to state structure over the matched control, significant in all twelve cells. State tracking does not cost recall: StateMem also leads memory systems on LongMemEval (0.656 on DeepSeek) and LoCoMo (0.592), with margins concentrated on temporal-reasoning and knowledge-update question types. The authors note StateMem mirrors the policy family behind the traps, so its StateMemBench margins should be read as an upper bound.

Plain-language abstract A benchmark for the decision an agent makes before it writes to memory: should this piece of interaction-derived information be persisted durably, used only in the current context, re-checked against the world, or clarified with the user. It evaluates three models from two families under three prompt conditions, and separately tests whether the stated decision predicts which memory tool the model actually calls.

Motivation Persistent memory personalizes an agent, but an incorrect durable update silently distorts future behavior. A temporary request should not become a standing preference, a service status can go stale, one tool failure may be noise, and an underspecified correction may need a question before it is generalized. The authors argue the critical capability is not recall but commitment, and that two sources of uncertainty are usually collapsed: verification queries the world, which is authoritative for changing facts, while clarification queries the user, who is authoritative for intent and scope. Prior work covers adjacent targets (binary session-level storage gating, ADD/UPDATE/DELETE/NOOP operations, whether retrieved memory grounds tool parameters, general ambiguity clarification) without jointly distinguishing durable storage, local use, world verification and user clarification at commitment time.

Methodology Each item supplies an acquire context, a candidate update and a later reuse context, with a gold action assigned by released rules and a tie between persist and a weaker action resolved toward the weaker commitment. 140 primary scenarios are split 70/70 by sorted identifier within category, covering stable and episodic preferences, freshness-sensitive facts, one-off corrections, policy constraints, ambiguous updates and noisy failures at 20 items each, with eight lexical traps where a surface cue points to the wrong action. Two non-authors labeled the held-out and contrast items blind to author labels and to each other, with a blind third resolving ties. Claude Haiku 4.5, Claude Sonnet 4.6 and a locally served Qwen3.5-9B (Q4_K_M via Ollama, temperature 0, seed 13, thinking disabled) were each run under three conditions: a bare prompt defining the actions, a policy prompt adding five commitment rules including the tie-breaker, and a four-shot prompt with one development example per action. A separate track, MCB-Act, removes the label vocabulary and requires one structured tool call, scored by mapping the selected tool to an action. Analysis uses accuracy with bootstrap intervals, macro-F1, over-memory, under-memory and per-class recalls, with exact paired McNemar tests and Holm correction within each family of comparisons.

Results Both model families under-ask. Claude label-mode verification recall runs 0.889 to 1.000 while clarification recall runs 0.500 to 0.750; bare Qwen verifies 12 of 18 freshness items and asks on 0 of 12 clarification items. Few-shot prompting lifts Qwen accuracy from 0.557 to 0.771 (paired delta +0.214, p_H = 0.002) and clarification recall from 0 to 0.333, still missing 8 of 12. The policy prompt raises Qwen accuracy by only 0.071 (p_H = 0.539) but cuts erroneous persistence from 0.243 to 0.100 (p_H = 0.038), moving the uncertainty to verification (recall 0.667 to 0.944) rather than to the user (clarification 0 to 0.083). Haiku's policy and few-shot gains survive correction (p_H = 0.002 and 0.047); Sonnet's do not, so the benchmark measures a prompt-conditioned commitment policy rather than a fixed model trait. Label-to-tool agreement is 0.571 for each Claude model and 0.229 for Qwen; Sonnet accuracy falls from 0.814 to 0.529 (p_H < 0.001) and Qwen from 0.557 to 0.343 (p_H = 0.047), with Qwen calling use_now on 54 of 70 items and verification recall collapsing to 0.056. All emitted arguments pass the deterministic well-formedness rules, locating the bottleneck in tool choice. On the combined 140 Qwen items of the contrast extension, bare, policy and few-shot accuracy is 0.614, 0.757 and 0.843, with clarification remaining the weakest class throughout; the authors retain the extension as a controlled sensitivity check rather than a claim of naturalistic external validity, since its rule-authored templates align closely with the explicit policy rules. MCB-Act scores tool-call selection and does not execute downstream effects.

Plain-language abstract REALTALK is a dataset of real conversations collected over 21 days from messaging apps, where pairs of people who initially met through those apps exchanged daily messages. The dataset contains 10 unique conversations totaling more than 16,000 words each, and is used to evaluate whether AI language models can hold long-term, emotionally aware conversations the way humans do.

Motivation Most research on long-term dialogue systems uses synthetic conversations generated by AI models rather than real human exchanges, leaving it unclear whether those simulated dialogues capture the emotional nuance and persona consistency of genuine human interaction. REALTALK was created to fill that gap by providing an authentic benchmark that can be directly compared against LLM-generated conversation datasets.

Methodology Ten participants each engaged in two separate 21-day conversation threads with different partners via messaging apps, producing conversations that span approximately 21 daily sessions per pair. The dataset was analyzed for emotional intelligence attributes and persona consistency, and compared against LLM-generated conversations. Two benchmark tasks were then defined: persona simulation, where a model must continue a conversation on behalf of a specific user given prior dialogue context, and memory probing, where a model must answer questions requiring recall of information from earlier in a long conversation.

Results Analysis showed that real-world conversations contain more diverse emotional expressions and greater variation in persona stability than synthetic LLM-generated dialogues. Benchmark experiments found that models struggle to simulate a specific user's conversational style from dialogue history alone, but fine-tuning on that user's own chat history improves persona emulation. Existing models also faced significant challenges in recalling and leveraging long-term context from real-world conversations.

Plain-language abstract This paper introduces AgenticAI-DialogGen, a software framework that automatically generates realistic, topic-organized conversations for training and testing AI language models on memory tasks. It also releases a new dataset called TopicGuidedChat (TGC) built from these generated conversations, which encodes both short-term and long-term memory cues for each pair of simulated speakers.

Motivation Existing conversational datasets lack the structure needed to train language models to reliably remember and reference information across a conversation — either they test short question-answering without capturing speaker personality, or they capture free-form dialogue without organizing it around coherent topics or memory. Creating such datasets manually is expensive and hard to do consistently, creating a gap that prevents progress on memory-aware conversational AI.

Methodology The framework takes an unstructured conversational dataset as input — primarily the Multi-Session Chat (MSC) dataset, restricted to 1,001 speaker pairs with four sessions each — and runs it through a pipeline of coordinated LLM-based modules and agents. These components preprocess conversations, extract factual knowledge triples, group them into topics, build per-speaker knowledge graphs, generate speaker personas, simulate multi-turn dialogues using LangGraph-based agents, validate and refine the output for topical adherence and quality, and finally produce memory-grounded question-answer pairs. The resulting TGC dataset stores long-term memory as structured knowledge graphs and short-term memory as simulated conversational turns.

Results Human and automatic evaluations show that AgenticAI-DialogGen improves discourse quality and topic coherence compared to baselines. Lightweight language models fine-tuned on the TGC dataset outperformed larger zero-shot models on memory-grounded tasks, demonstrating that the dataset's structured memory design provides practical utility for training memory-aware conversational systems.

Plain-language abstract This paper surveys how researchers evaluate AI chatbots and assistants built on large language models when those systems must hold multi-turn conversations — exchanges that unfold over many back-and-forth messages rather than a single prompt. The authors reviewed nearly 250 published studies to map out what is being evaluated and how, producing two classification frameworks that together cover the full range of current evaluation practice.

Motivation AI conversational agents powered by large language models are increasingly deployed in customer service, personal assistants, and other settings that require sustained, context-aware dialogue. Despite this growth, the field lacked a systematic overview of how such multi-turn systems are evaluated — what dimensions matter and what measurement methods are available — leaving practitioners without a consolidated reference.

Methodology Using a PRISMA-inspired systematic review process, the authors examined nearly 250 scholarly sources from a range of publication venues. From this corpus they constructed two interrelated taxonomy systems: one defining what to evaluate (task completion, response quality, user experience, memory and context retention, planning and tool integration) and one categorizing how to evaluate (annotation-based evaluations, automated metrics such as BLEU and ROUGE, hybrid human-plus-quantitative strategies, and self-judging methods that use LLMs as evaluators).

Results The survey produced a structured, dual-taxonomy framework covering both evaluation dimensions and evaluation methodologies for LLM-based multi-turn conversational agents. The framework captures traditional language-understanding metrics alongside newer techniques suited to the dynamic, interactive nature of multi-turn dialogue, offering a consolidated foundation for researchers and practitioners assessing conversational AI systems.

Plain-language abstract This paper is a systematic survey of personalized dialogue generation — the field of building conversational AI systems that tailor their responses to individual users. The authors review 22 datasets, analyze 17 research papers published at top NLP conferences between 2021 and 2023, and summarize the evaluation metrics used across this body of work.

Motivation Personalization is increasingly important for conversational agents, especially as large language models can generate fluent but generic responses. The field lacks a unified definition of personalization — it can mean giving an agent a persona, modeling a user's traits, or both — and no comprehensive survey had catalogued the datasets, methods, and evaluation practices across this rapidly growing area.

Methodology The authors conducted a keyword-based literature search of top NLP venues (ACL, NAACL, EMNLP, AAAI, and others) covering 2021 through October 2023, selecting 17 seminal works. They systematically categorized 22 datasets by language, persona representation type (descriptive sentences or key-value pairs), data source (crowdsourcing or social platforms such as Reddit and Weibo), and features such as persona grounding labels and multi-session support. They then identified five distinct problem types across the surveyed methods and compiled a summary of evaluation facets and metrics.

Results The survey identifies five distinct problem formulations within personalized dialogue generation and highlights both benchmark datasets (such as PersonaChat and ConvAI2) and newer datasets with richer features including knowledge graphs, empathy signals, and out-of-distribution personas. The authors note strong language and domain biases across datasets — most are English or Chinese, and several are domain-specific rather than truly open-domain. The paper also surveys recent progress by large language models on personalized dialogue tasks and outlines open challenges and directions for future research.

Plain-language abstract This paper introduces OmniBehavior, the first benchmark for testing how well large language models can simulate real human behavior across multiple online scenarios over extended time periods. The benchmark is built from actual user interaction logs on Kuaishou, a major video platform, covering 200 users over three months and spanning actions like browsing, purchasing, searching, and conversing across five distinct contexts. The authors use it to evaluate state-of-the-art LLMs and characterize where and why they fail as user simulators.

Motivation Existing benchmarks for LLM-based user simulation are limited to single scenarios, narrow action spaces, or synthetic data. Real human behavior crosses many contexts over long periods — a purchase decision may stem from a video watched days earlier, and a comment in a live stream may reflect a post-purchase experience. These fragmented benchmarks cause systematic misinterpretation of how capable LLMs actually are at modeling authentic human behavior.

Methodology The authors collected real interaction logs from Kuaishou across five scenarios (video browsing, live streaming, advertising, e-commerce, and customer service). They sampled 200 representative users, aggregated their complete interaction traces with timestamps over a three-month period, applied multi-level cleaning and anonymization, and constructed a benchmark with 22 distinct action types and trace lengths from 50 to over 100,000 actions (averaging around 32,000 tokens). They then evaluated both closed-source models (including Claude-4.5-Opus, GPT-5.2, Gemini-3-Flash) and open-source models (including Qwen3-235B, DeepSeek-V3) on a user-conditioned prediction task requiring the model to predict all user behaviors in a given scenario given the user's profile and history.

Results Even the best-performing model, Claude-4.5-Opus, achieved an overall score of only 44.55, and F1 scores on binary behavior prediction tasks (like, share) did not exceed 40% for most models. Extending context windows beyond 32K tokens did not consistently improve performance. Analysis showed that integrating additional scenarios expands interest coverage by approximately 20–30%, and over 80% of conversion paths span multiple scenarios and days. LLMs exhibited three structural biases: hyper-activity (overestimating action probabilities), persona homogenization (simulated users cluster together while real users are distinct), and a Utopian bias (LLMs produce unrealistically positive and polite behavior, failing to simulate dissatisfied or adversarial users).

Plain-language abstract This paper is a survey of research on using large language models (LLMs) to simulate how users behave in conversations. The authors organize a large body of recent work under a new taxonomy, examine the techniques used to build these simulators, and identify open challenges in the field.

Motivation User simulation has long been important for building and evaluating conversational AI systems, but the arrival of LLMs has rapidly expanded what is possible, creating a fragmented literature with no unified framework. A dedicated survey covering conversational user simulation specifically — as distinct from earlier statistical or task-specific approaches — was absent, leaving researchers without a systematic map of the space.

Methodology The authors conducted a systematic literature survey, collecting and categorizing recent work on LLM-based conversational user simulation. They developed a novel taxonomy organized around three questions: who is being simulated (ranging from general users to persona-level, role-playing, and individual users), what interaction pattern is being simulated (Human–AI, Human–Human, AI–AI, and many-human–AI), and how the simulation is technically implemented. Core techniques and evaluation methodologies across these categories are analyzed and compared.

Results The survey maps the field into a unified framework and identifies key research trends across domains including search, recommendation, task-oriented dialogue, education, and human-computer interaction. The authors find that LLM-generated conversational data can directly improve downstream system performance, and that evaluation remains a major open challenge — human judgment is the gold standard but costly, while LLM-as-judge approaches lack reliability for long or complex conversations. Open challenges including limited user feedback signals, persona fidelity, and scalable evaluation are identified and organized.

Plain-language abstract This paper surveys how the quality, diversity, and complexity of synthetic training data — data generated by large language models — affect the performance of the models trained on that data. The authors introduce a unified framework called QDC (Quality, Diversity, Complexity) to systematically compare and understand the many synthetic data generation methods that have emerged, and to explain why some methods produce better models than others.

Motivation Synthetic data generation with large language models has become common practice, but existing methods are ad hoc and rarely compared directly. Most approaches focus narrowly on maximizing data quality and quantity while discarding the majority of generated samples, without understanding which intrinsic data characteristics actually drive downstream model generalization. This paper addresses the lack of a principled framework for evaluating and designing synthetic data generation algorithms.

Methodology The authors conduct a structured literature survey organized around three research questions: how to define and measure quality, diversity, and complexity in datasets; how each characteristic affects model generalization; and how existing synthetic data generation algorithms promote each characteristic. They taxonomize synthetic data generation methods by the components they use and their effects on the QDC composition of the resulting data, extending the analysis to implications for reinforcement learning and self-improvement algorithms.

Results The survey finds that data quality is essential for in-distribution generalization, data diversity is essential for out-of-distribution generalization, and complexity is beneficial for both. The authors identify Quality-Diversity trade-offs in training data that directly affect downstream model performance. They also observe that most current models are evaluated and optimized only for output quality, which limits output diversity and constrains the potential for iterative self-improvement — arguing that balancing all three characteristics is essential for future self-improving AI systems.

Plain-language abstract EngramaBench is a benchmark for testing how well AI assistants remember and reason across information accumulated over many conversations. The paper introduces the benchmark, presents a graph-structured memory system called Engrama, and compares it against two baselines: simply feeding all prior conversation history into the prompt, and a vector-retrieval memory system called Mem0.

Motivation AI assistants are increasingly used as long-term collaborators, but existing benchmarks do not adequately test whether memory systems can integrate information across distinct areas of a user's life, reason about how things change over time, or correctly refuse to answer fabricated questions. There was also no clear answer to when structured memory architectures outperform brute-force context inclusion.

Methodology The benchmark is built around five synthetic personas, each with 20 timestamped multi-session conversations and 30 queries covering five task types: factual recall within a single topic area, cross-domain integration, temporal reasoning, adversarial abstention (refusing fabricated questions), and emergent synthesis. All three evaluated systems—GPT-4o full-context, Engrama, and Mem0—use the same answering model (GPT-4o) so that score differences reflect only the memory architecture. Engrama organizes memory into a graph of entities, semantic spaces, temporal traces, and cross-space links; Mem0 uses flat vector retrieval over extracted memories; full-context prompting concatenates all prior conversations directly into the prompt.

Results GPT-4o full-context achieved the highest overall composite score (0.6186), followed by Engrama (0.5367) and Mem0 (0.4809). However, Engrama was the only system to outperform full-context prompting on cross-space reasoning queries (0.6532 vs. 0.6291, n=30), which are the queries most diagnostic of structured long-term memory. Engrama reached about 87% of GPT-4o's composite score at roughly 20% of the query-time cost ($0.67 vs. $3.33 for 150 queries). Ablations showed that the components giving Engrama its cross-space advantage actually reduce global composite score, revealing a design tension between specialization for cross-domain reasoning and overall optimization.

Plain-language abstract UniToolCall is a unified framework that standardizes how AI language model agents learn to use tools — that is, how they translate plain-language instructions into executable function calls against external APIs and systems. It provides a consistent way to represent tool interactions, a large curated training dataset, and a common evaluation benchmark, so that researchers can train and compare tool-using agents on equal footing.

Motivation Existing research on tool use for language model agents suffers from fragmentation: different projects use incompatible representations for how agents call tools, datasets are constructed inconsistently and ignore the structural variety of real tool-use trajectories, and evaluation benchmarks cannot be directly compared. This lack of standardization makes it hard to measure progress or build on prior work.

Methodology The framework assembles a tool pool of over 22,000 tools and builds a hybrid training corpus of more than 390,000 instances by combining 10 standardized public datasets with synthetically generated trajectories that are structurally controlled to cover single-hop and multi-hop, single-turn and multi-turn, and both serial and parallel execution patterns. An Anchor Linkage mechanism is introduced to enforce cross-turn dependencies for coherent multi-turn reasoning. For evaluation, seven public benchmarks are converted into a unified Query-Action-Observation-Answer (QAOA) representation with fine-grained scoring at the function-call, turn, and conversation levels.

Results Fine-tuning Qwen3-8B on the UniToolCall dataset substantially improves tool-use performance. Under the distractor-heavy Hybrid-20 evaluation setting, UniToolCall achieves 93.0% single-turn Strict Precision, outperforming leading commercial models including GPT, Gemini, and Claude.

Plain-language abstract Graph2Counsel is a framework that automatically generates synthetic mental health counseling dialogues by grounding them in Client Psychological Graphs (CPGs) — structured representations of how a patient's thoughts, emotions, and behaviors interact. The system produces 760 multi-turn counseling sessions from 76 graphs derived from real therapy transcripts, and shows that models fine-tuned on this data outperform those trained on comparable existing datasets.

Motivation Large language models need high-quality counseling dialogue data to be safely adapted for mental health support, but real session transcripts are scarce due to strict privacy and confidentiality constraints. Existing synthetic datasets rely on flat text descriptions — demographics, symptom lists, or questionnaire scores — that miss the functional dependencies among a client's cognitive, emotional, and behavioral states, leading to psychologically inconsistent synthetic interactions.

Methodology The framework extracts CPGs and counselor strategies from anonymized real therapy transcripts using a prompt-based LLM pipeline. CPGs are represented as directed edge lists encoding excitatory and inhibitory relationships among psychological processes. These graphs, combined with CPG-derived client profiles and real counselor strategies, are fed to GPT-4o under four prompting conditions: a baseline, Guided Counseling (GC), GC with Chain-of-Thought reasoning, and a two-stage GC with Multi-Agent critique-and-refinement. Generated sessions are evaluated by LLM-as-a-judge using the Cognitive Therapy Rating Scale and Working Alliance Inventory, by four licensed clinician experts, and through downstream fine-tuning of Llama3-8B-Instruct via QLoRA on two counseling benchmarks.

Results In expert evaluation across five dimensions — specificity, counselor competence, authenticity, conversational flow, and safety — Graph2Counsel ranked first on all metrics compared to three prior synthetic dataset baselines (CACTUS, MAGneT, SQPsychConv), with substantial inter-annotator agreement (Krippendorff's alpha = 0.70). Generated client profiles were rated CPG-aligned in 90% of cases and realistic in 97% of cases. Fine-tuning Llama3-8B on Graph2Counsel data (Llama3-G2C) achieved the highest Few-Shot Chain-of-Thought accuracy on CounselingBench (0.631) among all fine-tuned models, and also improved performance on CounselBench.

A configurable simulation framework that injects realistic patient 'noise' (incomplete, inconsistent, distracting user behavior) into evaluation, exposing that strong static-benchmark scores collapse under realistic interaction.

Why it matters A concrete instance of controllable distractor/noise injection in a user simulator for agent eval, plus evidence of the static-benchmark-vs-realistic-interaction gap — the methodological pattern needed for stress-testing memory systems with adversarial/noisy synthetic users.

Plain-language abstract This paper is a comprehensive survey of how the concept of 'persona' is used in large language models (LLMs). It organizes a fast-growing but scattered body of research into two distinct lines: role-playing, where the LLM itself takes on an assigned persona (such as a doctor, engineer, or judge), and personalization, where the LLM adapts its responses based on a user's persona. The authors also cover methods for evaluating LLM personality.

Motivation Research on using personas to tailor LLM behavior had grown rapidly but remained disorganized, lacking a unified framework or systematic taxonomy. The authors identified this gap and set out to produce the first survey that brings role-playing and personalization together under a single conceptual lens.

Methodology The authors conducted a structured literature review, categorizing existing studies into two main streams: LLM Role-Playing (covering environments such as software development, games, medical applications, and LLM-as-evaluator) and LLM Personalization (covering applications such as recommendation, search, education, healthcare, and dialogue). They further analyzed role-playing schemas (single-agent vs. multi-agent) and emergent social behaviors, and reviewed approaches to LLM personality evaluation using frameworks such as the Big Five and MBTI. A continuously maintained paper collection was released alongside the survey.

Results The survey demonstrates that role-playing enables LLMs to perform better in structured, task-oriented environments—including software development pipelines, medical consultation, and automated evaluation—while personalization approaches allow LLMs to tailor responses across domains such as recommendation and healthcare. The paper establishes a systematic taxonomy covering both lines of research and highlights emergent social behaviors (such as conformity and adversarial debate) that arise under multi-agent role-playing schemas. It had received 44 citations as of the time of indexing.

Plain-language abstract Zep is a memory layer service for AI agents built on top of a temporally-aware knowledge graph engine called Graphiti. Unlike standard retrieval systems that work with static document collections, Zep continuously ingests conversational and structured business data, maintains a timeline of facts and relationships, and retrieves relevant context for LLM agents at query time.

Motivation Large language models powering chat agents are constrained by their context windows and static training knowledge, and existing retrieval-augmented generation (RAG) approaches work only with fixed document corpora. Enterprise applications require agents that can integrate and reason over continuously evolving data — including ongoing conversations and business records — but no existing memory system addressed this dynamic, temporally-aware requirement at production scale.

Methodology Zep represents memory as a hierarchical knowledge graph with three tiers: an episodic subgraph storing raw messages, a semantic entity subgraph of extracted entities and facts, and a community subgraph of clustered entity groups. New data is ingested by extracting entities and facts via LLM prompts, resolving duplicates, and invalidating outdated edges using a bi-temporal model. Retrieval combines cosine similarity search, BM25 full-text search, and breadth-first graph traversal, followed by reranking. The system was evaluated on the Deep Memory Retrieval (DMR) benchmark (500 multi-session conversations) and the LongMemEval benchmark (conversations averaging 115,000 tokens), using GPT-4o-mini and GPT-4o for generation and BGE-m3 for embeddings and reranking.

Results On the DMR benchmark, Zep achieved 94.8% accuracy with GPT-4-turbo and 98.2% with GPT-4o-mini, compared to MemGPT's 93.4% with GPT-4-turbo. On the more demanding LongMemEval benchmark, Zep improved accuracy by 15.2% with GPT-4o-mini and 18.5% with GPT-4o over full-context baselines, while simultaneously reducing response latency by approximately 90% and compressing average context from 115,000 tokens to 1,600 tokens. The largest accuracy gains appeared in complex question types such as single-session-preference, multi-session, and temporal-reasoning tasks.

Plain-language abstract HyperMem is a memory system for conversational AI agents that need to remember information across long, extended conversations. Rather than storing conversation history as flat text chunks or simple knowledge graphs, it organizes memory into a three-level hierarchy of topics, episodes, and facts, and uses hyperedges to group related memories that may be spread across many exchanges. When answering a question, it searches this structure from coarse to fine, moving from relevant topics down to specific facts.

Motivation Conversational agents lose access to earlier parts of a conversation as it grows beyond their fixed context window, requiring an external memory system. Existing approaches like chunked retrieval and graph-based memory use pairwise relationships, which cannot capture higher-order associations — cases where three or more pieces of information are jointly relevant but scattered across different time points in the dialogue. This fragmentation causes incomplete or incorrect retrieval when questions require multi-hop reasoning across a long history.

Methodology HyperMem structures memory into topics, episodes, and facts. Dialogue streams are segmented into episodes using LLM-driven boundary detection; related episodes are then linked under shared topic nodes via hyperedges; and fine-grained facts are extracted from each episode. A hybrid lexical-semantic index is built using BM25 and dense embeddings propagated through the hypergraph. At query time, retrieval proceeds coarse-to-fine: first identifying relevant topics via combined reciprocal rank fusion and reranking, then expanding to associated episodes, then selecting the most pertinent facts to assemble the response context. The system was evaluated on the LoCoMo long-term conversation benchmark.

Results HyperMem achieves 92.73% LLM-as-a-judge accuracy on the LoCoMo benchmark, reported as state-of-the-art. In an efficiency comparison, the Episode + Fact retrieval configuration reaches this accuracy using roughly 7.5 times the tokens of a Mem0 baseline, while a Fact Only configuration reaches 89.48% at just 2.5 times the tokens — both substantially outperforming graph-based RAG methods such as GraphRAG (67.60% accuracy at 35.3 times token usage) and HyperGraphRAG (86.49% at 26.3 times). Ablation results show that removing topic-level retrieval causes the largest single accuracy drop (3.76 percentage points), and that episode context consistently adds 3–4% over fact-only retrieval.

Plain-language abstract This paper introduces MemTree, a memory management algorithm for large language models (LLMs) that organizes information in a dynamic tree structure, similar to how human cognitive schemas work. Rather than storing each piece of information as a separate, unconnected entry, MemTree groups and links related memories hierarchically so that LLMs can reason more effectively over long conversations and documents.

Motivation Large language models struggle with long-term memory: even as context windows have expanded to millions of tokens, models rely on key-value caches processed through a fixed number of layers, which cannot effectively aggregate extensive historical data. Existing external memory methods store past experiences as isolated entries in flat lookup tables, losing the interconnected, hierarchical structure that makes human memory efficient for retrieval and reasoning.

Methodology MemTree represents memory as a dynamic tree where each node stores aggregated textual content and semantic embeddings at varying levels of abstraction. When new information arrives, the algorithm traverses the tree from the root, computing semantic similarity between the new content and existing nodes; if similarity exceeds a threshold the information is routed to that branch, otherwise a new leaf node is created. This insertion process runs in O(log N) time where N is the number of conversational interactions. The system was evaluated on benchmarks for multi-turn dialogue understanding and document question answering, including the MultiHop dataset.

Results MemTree significantly enhances performance on benchmarks requiring structured memory management, including multi-turn dialogue understanding and document question answering. The system outperforms methods that rely on flat memory lookup tables, and on tasks involving reasoning over event sequences across multiple documents, MemTree exceeds all offline comparison methods. The work was published as a conference paper at ICLR 2025.

Plain-language abstract This paper describes HLTM (Hierarchical Long-Term Semantic Memory), a memory system built for LinkedIn's AI recruiting agent (Hiring Assistant) that lets the agent remember and reason over a recruiter's past projects and preferences across many conversations. Instead of stuffing all past activity into a single context window, HLTM organizes information in a tree structure and retrieves only what is needed at low latency, while respecting data-privacy rules. The system is deployed in production at LinkedIn.

Motivation LLM-based agents used in enterprise products need long-term, personalized memory to be useful, but existing memory systems struggle with five practical challenges at industrial scale: ingesting millions of documents efficiently, retrieving answers within tight response-time budgets, enforcing strict data-privacy isolation between tenants, adapting to new query patterns without hand-crafted rules, and providing transparent provenance so the system can be audited and debugged. No prior system addressed all five challenges together for a live enterprise deployment.

Methodology HLTM organizes memory as a schema-aligned tree whose topology mirrors the enterprise data model — for LinkedIn's hiring domain, leaf nodes represent individual hiring projects, intermediate nodes represent recruiter seats, and root nodes represent organizational accounts. Each tree node stores three complementary memory views extracted offline by LLM agents: key-value facet pairs (for structured filtering), answerable question-answer pairs (for fast serving), and natural-language summaries (for unstructured queries). Retrieval at serving time combines hybrid embedding-based search with an in-context-learning answering step, restricted to the subtree that matches the query's access-control scope. A lightweight adaptation loop mines historical query patterns to continuously refine which facts are extracted and stored.

Results On a benchmark derived from LinkedIn's Hiring Assistant dataset, HLTM achieved the best answer correctness (0.798) at a query latency of roughly 3.4 seconds, outperforming all nine baselines including HippoRAG, RAPTOR, GraphRAG, and conventional RAG. On summary-style queries it improved correctness over the strongest baselines by more than 15%; on retrieval-style queries it exceeded the best baseline in F1 by more than 10%. An ablation study showed that removing tree aggregation caused roughly 12-point and 18-point drops in correctness and retrieval F1 respectively, and that removing the adaptation mechanism degraded correctness by about 10 points. HLTM also reduced query-time token usage by at least 50% relative to graph-based baselines.

Plain-language abstract This paper introduces GAM, a memory system for AI assistants that need to remember things across long conversations. The key idea is to separate the act of recording new information from the act of integrating it into long-term knowledge, which helps the AI stay coherent over time without getting confused by irrelevant details.

Motivation AI language model agents struggle to maintain coherent long-term interactions because existing memory designs force a trade-off: stream-based systems update easily but are disrupted by noise, while structured memory systems retain knowledge well but cannot adapt as conversations evolve. There was no approach that achieved both rapid context awareness and stable long-term retention simultaneously.

Methodology GAM organizes memory into two hierarchical layers: an event progression graph that captures the current dialogue in fine-grained detail, and a topic associative network that holds stable long-term knowledge. New dialogue is held in the event graph and only merged into the topic network when a semantic shift is detected, limiting interference. A graph-guided, multi-factor retrieval strategy is used to pull relevant context at query time. The framework was evaluated on two long-context dialogue benchmarks, LoCoMo and LongDialQA.

Results GAM consistently outperformed state-of-the-art baselines on both LoCoMo and LongDialQA benchmarks in reasoning accuracy and efficiency. The approach demonstrated that explicitly decoupling memory encoding from consolidation reduces interference from transient noise while preserving long-term consistency across extended agent-user dialogues.

Plain-language abstract This paper introduces SCM (Sleep-Consolidated Memory), a memory architecture for large language models that mimics how human brains store and forget information. Rather than growing an ever-larger database of raw conversation text, SCM encodes conversations into structured concepts, selectively strengthens important ones during simulated sleep cycles, and actively prunes low-value memories — enabling persistent, organized memory that stays manageable over time.

Motivation Current LLM memory approaches all have critical gaps: context windows are bounded and degrade with long input, vector databases grow without limit and never forget, and tiered storage systems like MemGPT lack any offline consolidation or biological forgetting mechanisms. None replicate how human memory actually works — a dynamic system that consolidates during sleep, prioritizes by importance, and actively prunes weak associations. SCM was designed to fill this gap by bringing neuroscience-inspired memory management to conversational AI.

Methodology SCM is built from five modules: a MeaningEncoder (using a local Llama 3.2 2B quantized model) that converts text into typed semantic concepts with 384-dimensional embeddings; a ValueTagger that assigns four-dimensional importance scores (novelty, emotional valence, task relevance, repetition frequency); a WorkingMemory buffer capped at seven items; a LongTermMemory stored as a NetworkX semantic graph backed by SQLite; and a SleepCycle module that runs NREM consolidation (Hebbian strengthening plus synaptic downscaling), REM dreaming (novel association generation), and value-based forgetting. The prototype was evaluated on a standardized suite of eight benchmark tests covering memory retention, consolidation, forgetting, graph traversal, latency scaling, and multi-session persistence.

Results Across all eight benchmark tests the prototype achieved perfect scores (1.00). It maintained 100% recall accuracy over ten-turn conversations while reducing memory noise by 90.9% through adaptive forgetting — pruning 45 of 50 noise concepts while preserving all 5 important ones. Memory search latency remained below one millisecond with hundreds of stored concepts. An ablation study showed that disabling the ForgettingModule caused the largest memory bloat (72 stored concepts vs. 24 in full SCM) and that disabling the ValueTagger dropped recall to 81.8%, confirming that multi-dimensional importance tagging is the most critical component for selective retention.

Plain-language abstract This paper proposes 'cooperative paging', a method that helps AI chatbots handle very long conversations without losing track of earlier content. When a conversation grows too long to fit in the model's memory, old sections are replaced with short keyword summaries (about 8 tokens each), and the model is given a tool it can call to retrieve the full text of any summarized section when it needs it. The paper evaluates this approach on a standard benchmark of long real-world conversations and also runs a systematic study of design choices like page size and eviction strategy.

Motivation Large language models have a fixed context window, but multi-turn conversations can grow indefinitely. When old content must be discarded to make room, existing methods either force the model to guess what it is missing (which models do poorly), only work for narrow cases like file reads, or compress content irreversibly losing detail. There was no systematic study of how page boundary detection and eviction policy affect retrieval quality in this setting.

Methodology The authors built a cooperative paging system in which evicted conversation segments are replaced with minimal keyword bookmarks (e.g., '[p3: allergy, peanut, budget]') and the model is given a recall() tool to fetch full content on demand. They evaluated it on the LoCoMo benchmark (10 real multi-session conversations with 300+ turns each) across four models from three provider families (GPT-4o-mini, DeepSeek-v3.2, Claude Haiku, GLM5), comparing against five baselines including truncation, BM25 retrieval, and a search-tool baseline. They also ran a turn-by-turn paging simulator with 3,176 probes on synthetic data and 1,600 probes on LoCoMo to ablate five page-boundary strategies and four eviction policies (FIFO, LRU, LFU, Belady oracle), and tested six bookmark generation strategies.

Results Cooperative paging achieved the highest answer quality among six methods on LoCoMo across all four tested models, with statistical significance (p=0.017 vs. BM25, paired bootstrap). The ablation showed page granularity dominates eviction policy: coarse fixed-size pages (fixed-20) reached 96.7% recall accuracy while content-aware topic-shift paging collapsed to 56.7%. The model triggers recall() correctly 96% of the time, but selects the right page only 57% of the time when bookmarks are insufficiently distinctive, shifting the bottleneck from 'when to recall' to 'which page to recall'. The best bookmark strategy (llm-batch, single-call cross-page non-overlap keywords) improved end-to-end accuracy by 8.7 points on LoCoMo and 50 points on open-domain questions over the heuristic baseline; keyword specificity alone accounted for a 25 percentage-point accuracy difference in controlled probes.

Plain-language abstract Memanto is a memory system for AI agents that need to remember information across many conversations and tasks. Instead of the complex graph-based memory architectures most agent frameworks use, Memanto organizes memory into thirteen typed categories and retrieves relevant memories using a single, fast semantic search query — achieving top benchmark scores while being simpler and cheaper to run.

Motivation As AI agents move from answering single questions to carrying out long, multi-step tasks across many sessions, they need reliable persistent memory. Existing production memory systems — such as Mem0, Zep, and A-MEM — combine knowledge graphs with vector databases, which imposes heavy computational costs: every memory write triggers multi-second pipelines involving LLM-driven entity extraction and graph synchronization. The paper argues this 'Memory Tax' is unnecessary and that simpler architectures can match or beat graph-based systems in accuracy.

Methodology Memanto is built on Moorcheh's Information-Theoretic Search engine, a no-indexing semantic database that provides deterministic (exact-match rather than approximate nearest-neighbor) retrieval with sub-90-millisecond latency and zero ingestion delay. The memory layer uses a typed schema of thirteen predefined semantic categories, an automated conflict-resolution mechanism for contradictory memories, and temporal versioning. The system was evaluated on two established long-term memory benchmarks — LongMemEval and LoCoMo — using a five-stage progressive ablation study that isolated the contribution of retrieval-limit tuning, similarity-threshold calibration, prompt design, inference model selection, and the typed schema.

Results Memanto achieved accuracy scores of 89.8% on LongMemEval and 87.1% on LoCoMo, establishing state-of-the-art results among both vector-based and hybrid graph-plus-vector systems. These results were obtained using only a single retrieval query per lookup, with no ingestion cost and no graph infrastructure — outperforming all evaluated hybrid architectures while requiring substantially lower operational complexity.

Plain-language abstract This paper argues that AI agent memory systems make a fundamental design mistake: they treat factual knowledge and personal experience as the same kind of thing, applying the same forgetting rules to both. The author proposes splitting the cognitive substrate of AI agents into four distinct layers — Knowledge, Memory, Wisdom, and Intelligence — each with its own rules for how information persists and gets updated. Companion implementations in Python and Rust are provided to show the separation is practically feasible.

Motivation The two most influential cognitive architecture frameworks for AI agents, CoALA and JEPA, both lack an explicit Knowledge layer with distinct persistence semantics. Existing systems such as Mem0, NornicDB, and Signet apply cognitive decay (time-based forgetting) equally to factual claims and episodic experiences, which the paper identifies as a category error: a scientific finding does not become less true simply because time has passed, yet current architectures treat it as if it does.

Methodology The paper is a conceptual and analytical work. The author surveys persistence semantics across a range of existing agent memory systems, identifies eight convergence points in the literature and practitioner community pointing to the same architectural gap, and analyzes two dominant frameworks (CoALA and JEPA) in detail. From this analysis, a four-layer decomposition is proposed — Knowledge (indefinite, supersession-based), Memory (Ebbinghaus decay), Wisdom (evidence-gated revision), and Intelligence (ephemeral inference-time) — with each layer defined by its persistence semantics, update mechanism, and ownership scope. The terminology is borrowed from cognitive science as analogy, but the layers are framed as engineering constructs.

Results The paper finds that no current framework or system provides the full four-layer decomposition, and that applying decay to factual knowledge produces demonstrably wrong behavior (e.g., the BEAM benchmark shows near-zero contradiction-resolution scores). A survey of April 2026 community work identified multiple independent systems — including rohitg00's LLM Wiki v2, Hermes Agent, Semantica, and MinnsDB — each converging on one or two components of the four-layer thesis without coordinating, suggesting the architectural pattern is emerging across the community. The correct operation for updating factual claims is supersession (recording the relationship between old and new claims), not forgetting.

Plain-language abstract ZenBrain is a memory architecture for AI agents that draws on 130 years of cognitive neuroscience research to give language-model agents a structured, biologically grounded way to store, consolidate, and forget information across many sessions. It organizes memory into seven distinct layers — working, short-term, episodic, semantic, procedural, core, and cross-context — and coordinates them with fifteen neuroscience-inspired algorithms. The system is open-source and ships as composable npm packages with over 11,500 automated tests.

Motivation Current AI agent memory systems borrow metaphors from computer science — virtual memory paging, flat key-value stores, or structured note-taking — but none incorporate the well-validated principles of memory consolidation, forgetting, and reconsolidation studied in cognitive neuroscience for over a century. Without principled decay and consolidation, agents suffer 'conversational amnesia' and cannot maintain consistent personality or learning across sessions. A 2026 survey explicitly identified deeper neuroscience integration as a key open challenge in the field.

Methodology ZenBrain implements seven memory layers orchestrated by nine foundational algorithms plus six new Predictive Memory Architecture (PMA) components, including a four-channel NeuromodulatorEngine modeling dopamine, norepinephrine, serotonin, and acetylcholine dynamics; a prediction-error-gated ReconsolidationEngine; TripleCopyMemory with divergent decay dynamics; and a four-dimensional PriorityMap with an amygdala fast-path. The system was evaluated across ten experiments covering memory lifecycle management, retrieval benchmarks, and system-level ablation studies using three established benchmarks — LoCoMo, MemoryAgentBench, and MemoryArena — under a 15-algorithm ablation protocol with Wilcoxon tests over 10 seeds.

Results Under challenging conditions (decay=0.20, 50 days), 7 of 15 algorithms became individually significant with quality drops ranging from -25.5% to -93.1%; under stress conditions (decay=0.25, 60 days), 9 became critical (up to -93.7% degradation). The Simulation-Selection sleep loop achieved a 37% stability improvement (p < 0.005) with 47.4% storage reduction. TripleCopyMemory retained 0.912 mean memory strength at 30 days versus near-zero Ebbinghaus baselines, and the PriorityMap achieved NDCG@10 = 0.997 versus 0.680 for chronological ordering. On the LongMemEval-500 benchmark, ZenBrain held the highest mean rank across all 12 judge-system quality cells, with a three-judge mean of 0.545 against competitors letta (0.485), a-mem (0.414), and mem0 (0.394), reaching 91.3% of long-context-oracle accuracy at 1/106th the per-query token budget.

Plain-language abstract GRAVITY is a plug-and-play memory module for AI conversational agents that improves how retrieved memories are presented to a language model at response time. Rather than dumping retrieved text fragments into a prompt, GRAVITY reorganizes that information into three structured forms — entity profiles, timestamped event tuples, and cross-session topic summaries — and injects them as explicit context. It works alongside any existing memory system without modifying it.

Motivation Long-horizon conversational agents increasingly rely on sophisticated retrieval systems, but retrieved memory fragments are fed to language models as flat, unstructured text. This forces the model to silently reconstruct which entities relate to each other, how events are ordered, and what themes connect multiple sessions — a reconstruction that fails on multi-hop and temporal questions. Even when retrieval is perfect and all relevant evidence is present, models achieve only about 80.9% accuracy, and this drops to 75.6% when the evidence is mixed with distractors, confirming the problem lies in how context is structured, not whether the right memories were fetched.

Methodology GRAVITY has two phases. During an offline build phase, raw conversation utterances are processed by batched LLM calls to extract three anchor types: entity anchors (dynamic profiles capturing attributes, relationships, and state changes), event anchors (structured subject-verb-object-time-outcome tuples linked into chronological causal traces), and topic anchors (cross-session thematic summaries). At inference time, given a user query, GRAVITY uses embedding-based reranking to select top-K anchors from each type and injects them into the host system's generation prompt as structured context alongside expanded retrieval queries. Integration requires only prompt augmentation — no changes to the host model or memory architecture. The system was evaluated on five diverse memory systems (including A-Mem, Mem0, ZEP, LiCoMemory, and LightMem) using the LongMemEval and LoCoMo benchmarks.

Results Across five host memory systems, GRAVITY improves LLM-judge accuracy by an average of 9.2% on LongMemEval-Micro, 10.1% on LongMemEval-Macro, and 7.5% on LoCoMo. Per-system gains range from 3.8% to 13.1%, with weaker baselines benefiting most (the weakest host improves by 12.2%) while the strongest still gains 3.8–5.7%. A controlled ablation comparing GRAVITY against an unstructured free-form summary injected in the same prompt position shows +5.7% versus +1.3% on LoCoMo, isolating a 4.4-point advantage attributable specifically to the explicit entity-event-topic structure rather than additional LLM-generated text volume. In the scattered-evidence setting, adding anchors recovers 2.9 percentage points without introducing any new evidence.

Plain-language abstract MemCog is a memory system for conversational AI agents that replaces simple keyword lookups with a structured, multi-step reasoning process. Rather than fetching a flat list of stored facts in response to a user query, the system lets the agent navigate through a hierarchically organized memory store—following associative links across topics—and proactively surfaces relevant context even when the user has not explicitly asked for it.

Motivation Existing agent memory systems treat memory as a passive tool: one query goes in and a flat list of text passages comes out, with no further reasoning over results. This design has three compounding weaknesses—memory is only activated when explicitly queried, the retrieval and reasoning steps are disconnected, and retrieved fragments carry no structural cues to guide further exploration. The paper addresses these limitations by rethinking memory access as an integral part of the agent's reasoning loop.

Methodology MemCog has three core components: a Navigable Memory Store that organizes user knowledge in a three-level hierarchy (dimensions, pages, and sections) with cross-dimensional associative links; a Cross-Dimensional Navigation Interface that exposes multi-step browsing actions (Browse, List, Read, Follow Links) so the agent can iteratively explore the store within a ReAct loop; and a Proactive Reasoning Protocol injected via the system prompt that instructs the agent to spontaneously initiate memory exploration when conversational context suggests relevant stored associations. The authors also construct ProactiveMemBench, a new benchmark of 500 test instances across five topic domains built through a six-step LLM-driven pipeline and validated by human annotators at a 98.4% acceptance rate.

Results MemCog achieves state-of-the-art scores on two established passive question-answering benchmarks: 92.98 on LoCoMo and 95.8 on LongMemEval, outperforming prior systems such as HyperMem and A-Mem. On the new ProactiveMemBench, removing the Proactive Reasoning Protocol alone drops Recall@5 from 59.51 to 44.90, confirming that proactive behavioral guidance is the primary driver of spontaneous memory exploration. Ablation studies show that both the navigable memory structure and the proactive protocol are individually necessary and mutually reinforcing, with neither component producing strong proactive performance on its own.

Plain-language abstract APEX-MEM is a memory framework for AI conversational agents that enables them to accurately remember and reason about information across long, multi-session conversations. It stores conversational knowledge in a structured property graph using an append-only event log, and retrieves relevant facts at query time using a set of complementary tools that combine entity lookup, graph traversal, and hybrid text search.

Motivation Large language models struggle to maintain coherent memory across extended conversations because simply extending context windows introduces noise and hallucinations, while retrieval-based approaches using unstructured text cannot track how facts evolve over time or resolve contradictions between older and newer information. Existing structured memory approaches that use entity-centric graphs are limited in their ability to represent nuanced attributes and temporal changes, and systems that overwrite facts risk losing context needed for temporal reasoning.

Methodology The authors designed APEX-MEM around three key components: a hybrid entity-event ontology that treats conversational events as first-class citizens alongside entities, an append-only event store that anchors facts to temporally grounded events rather than overwriting them, and a multi-tool retrieval framework comprising an entity lookup tool, a SQL-based graph traversal tool for temporal reasoning, and a hybrid semantic-plus-lexical search tool. The system was evaluated on three benchmarks: LOCOMO (long-term multi-session dialogue), LongMemEval (long-context factual reasoning), and SealQA-Hard (search-augmented fact-seeking questions), using LLM-as-a-Judge scoring for answer quality and factuality.

Results APEX-MEM achieved 88.88% accuracy on the LOCOMO question answering benchmark and 86.2% on LongMemEval, outperforming prior state-of-the-art session-aware memory approaches including Mem0, AMEM, Zep, MemGPT, and MemInsight across single-hop, multi-hop, temporal, open-domain, and adversarial question categories. The results demonstrate that structured property graphs with append-only temporal event storage enable more accurate and temporally coherent long-term conversational reasoning than unstructured retrieval or purely entity-centric graph methods.

Plain-language abstract Thought-Retriever is a new method for giving AI language model agents a practical long-term memory. Instead of storing and retrieving raw text chunks from a knowledge base, the system stores the intermediate reasoning and responses the model produced while answering past questions — called "thoughts" — and retrieves those when new, related questions arrive. The authors also introduce AcademicEval, a new benchmark that tests whether a model can correctly answer questions drawn from real academic papers requiring very long context.

Motivation Large language models can only work with a limited amount of text at one time, so standard retrieval systems that pull raw text chunks from large knowledge bases often miss relevant information or return too much irrelevant content. Hierarchical retrieval methods that summarize documents independently of user queries improve recall but hurt precision. There was no good way for an LLM agent to accumulate and reuse knowledge across many interactions without hitting these context-length bottlenecks.

Methodology The paper proposes Thought-Retriever, a model-agnostic algorithm in which an LLM collects its own intermediate responses (thoughts) generated while answering prior user queries, filters out meaningless or redundant ones, organizes them in a thought memory store, and retrieves the most relevant thoughts when handling new queries. This creates a self-evolving long-term memory that grows richer as the agent handles more queries. The method is evaluated on the new AcademicEval benchmark — built from real academic papers requiring faithful use of ultra-long context — as well as two additional public datasets, with all retriever-based methods capped at a maximum context length of 2,000 tokens.

Results Thought-Retriever outperforms state-of-the-art retrieval-augmented baselines across all evaluated datasets, achieving an average improvement of at least 7.6% in F1 score and 16% in win rate. The experiments also demonstrate that the system genuinely self-evolves — performance improves as the agent solves more queries — and that it learns to use higher-level, more abstract thoughts to answer more abstract questions.

Plain-language abstract This paper asks whether modern agentic search systems—where a language model dynamically issues multiple retrieval rounds during reasoning—can replace the expensive graph-structured retrieval pipelines known as GraphRAG. To find out, the authors build RAGSearch, a benchmark that tests both standard dense retrieval and five representative GraphRAG methods as retrieval backends under both training-free and reinforcement-learning-based agentic inference, across six question-answering datasets.

Motivation Most comparisons of dense RAG and GraphRAG use inconsistent evaluation protocols, partial test sets, and uncontrolled computational budgets, making it hard to know when the high preprocessing cost of graph construction is truly justified. At the same time, agentic search—where a model iteratively refines queries across multiple retrieval turns—has shown strong gains over one-shot retrieval, raising the question of whether it can substitute for explicit graph structure rather than simply complement it.

Methodology The authors introduce RAGSearch, a unified benchmark that treats dense RAG and five GraphRAG systems (HypergraphRAG, HippoRAG2, LinearRAG, RAPTOR, and GraphRAG) as interchangeable retrieval infrastructures. They evaluate two training-free agentic workflows (Search-o1 and GraphSearch) and two reinforcement-learning-trained agents (Search-R1 and Graph-R1, using GRPO) over six QA benchmarks spanning single-hop and multi-hop tasks (NQ, PopQA, TriviaQA, HotpotQA, Musique, 2WikiMultiHopQA), with matched retrieval budgets, standardized LLM backbones (Qwen2.5 at 3B, 7B, and 32B scales), and full test-set evaluation. Beyond accuracy they measure offline preprocessing cost, online inference efficiency, and output stability.

Results Agentic search substantially narrows the performance gap between dense RAG and GraphRAG, particularly in RL-based settings, but does not eliminate it. GraphRAG consistently achieves stronger and more stable performance on complex multi-hop reasoning tasks, with lower variance in answer quality across retrieval turns. Scaling the LLM backbone reduces the GraphRAG advantage: in RL-based systems, moving from 3B to 7B parameters cuts the average GraphRAG–Dense gap from 14.70 to 9.75 points. GRPO is the most effective RL training paradigm across both retrieval backends. The findings indicate that agentic search redistributes where structure emerges—shifting some from offline graph construction to online interaction—but explicit graph-based retrieval remains valuable when its offline cost can be amortized.

Plain-language abstract Mem0 is a memory architecture for AI agents that lets them remember and reuse information across separate conversations, instead of forgetting everything once a session ends. The system dynamically extracts key facts from ongoing dialogues, stores them compactly, and retrieves only the most relevant ones when answering a new question. A graph-enhanced variant also tracks relationships between entities for more complex multi-hop reasoning.

Motivation Large language models reset their knowledge at the end of every context window, so they cannot maintain consistent user preferences or prior context across multiple sessions. Longer context windows only delay this problem: even 200K-token windows eventually fill up, attention degrades over distant tokens, and forcing the model to process an entire conversation history for every query is slow and expensive. There was no production-ready system that could selectively persist, consolidate, and retrieve salient facts at scale.

Methodology The authors built Mem0 around two phases: an extraction phase that processes each new message pair alongside a rolling conversation summary and recent message window to identify salient facts, and an update phase that compares newly extracted memories against similar stored ones and applies add, update, or delete operations via a tool-call mechanism. A graph-enhanced variant stores memories as directed labeled graphs with entities as nodes and relationships as edges. Both approaches were evaluated on the LOCOMO long-term conversation benchmark against six baseline categories including RAG with varying chunk sizes and k-values, a full-context baseline (26,000 tokens per query), open-source memory tools, and proprietary systems.

Results Mem0 outperformed all memory-augmented baselines across single-hop, temporal, multi-hop, and open-domain question categories on LOCOMO. Mem0 scored approximately 67% on the LLM-as-a-Judge metric, a 26% relative improvement over the OpenAI baseline and about a 10% relative gain over the best RAG configuration. The graph-enhanced variant reached 68.44%, roughly 2% higher than base Mem0. Compared to the full-context approach (which scored ~73% but at high cost), Mem0 achieved 91% lower p95 latency (1.44 s vs. 17.1 s) and used more than 90% fewer tokens, storing each conversation in approximately 7,000 tokens versus the 26,000-token full-context baseline.

Position paper directly on the review's thesis.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

KG world model + episodic memory architecture.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Plain-language abstract Engram is an open-source long-term memory engine for LLM agents. Instead of replaying an entire conversation history into the prompt, it stores the past as a bi-temporal knowledge graph and retrieves a small, precisely-targeted slice at answer time. On a standard 500-question memory benchmark that lean ~9.6k-token slice answers more accurately than feeding the model the full ~79k-token history, turning memory from a cost optimization into an accuracy improvement.

Motivation Stateless LLM agents forget across sessions, and the usual fix — concatenating the whole history — grows token cost and latency linearly and loses accuracy as distractors crowd the window ('lost in the middle'). Two gaps stay open: most memory systems are cheaper or faster but not more accurate than full-context, and memory benchmarks run on inconsistent harnesses where one system reports wildly different scores across sources. Engram targets both — beating full-context on accuracy, and shipping a neutral, re-runnable harness.

Methodology A dual-process design. A System-1 hot write path appends lossless episodes with no LLM (sub-50ms) and enqueues them. A System-2 async path extracts atomic (subject,predicate,object) facts, builds a bi-temporal knowledge graph (valid time vs transaction time on every fact and edge), detects conflicts, and resolves them cheap-then-escalate: exact slot match, embedding similarity, and content subsumption handle the common case with no LLM call, invalidating (never deleting) a superseded fact and recording a supersedes chain and provenance, with only ambiguous cases escalated to an LLM adjudicator. The hybrid read path retrieves through four channels (dense, BM25, graph n-hop, recency/salience), fuses them with Reciprocal Rank Fusion, applies an 'as-of' temporal filter and an abstention gate, and assembles a deduplicated, provenance-tagged, token-budgeted context of facts plus raw chunks.

Results On the full 500-question LongMemEval_S under the official category-specific judge, Engram's lean configuration scores 83.6% vs 73.2% for full-context (+10.4 points, McNemar exact p<10^-6) at ~8x fewer tokens (9.6k vs 79k), 0/500 errored. The gain is load-bearing on the read path being hybrid: facts alone lose recall, while facts plus retrieved chunks recover detail. Bi-temporal modeling pays off most on knowledge-update (87.5%) and temporal (81.1%) categories, while multi-session aggregation and preference remain headroom. The paper documents measurement-integrity pitfalls (truncation, home-grown judges, full-history leaks) and ships a neutral in-repo harness with the official judge baked in and raw per-question logs, every number reproducible by command.

Plain-language abstract Infini Memory is a long-term memory architecture for LLM agents that stores memory as a library of plain-text 'topic documents' rather than vectors or a knowledge graph. Each document gathers related evidence under a subject and is maintained over time by splitting, merging, and rewriting. At answer time the agent reads memory through iterative tool calls, expanding context around matches instead of taking a single retrieval shot.

Motivation External memory systems that store observations as isolated records, summaries, or indexed fragments hit four recurring failure modes: fragmentation (evidence about one subject scattered across records), conflict (old and new versions of a fact coexisting), compression loss (summaries dropping temporal and source cues), and insufficient retrieval (single-shot top-k returning fragments without enough local context for multi-hop reasoning). Infini Memory reframes persistent memory as a lifecycle maintenance problem — write, maintain, read — and aims for an inspectable, editable state without a mandatory vector or graph backend.

Methodology Memory is a library of topic documents, each a maintenance scope with a summary, body, and entry-level metadata signatures (<seq,time,source>) that preserve order and provenance as content is rewritten. Writes are decoupled from structure: new candidates append to a buffer document, then periodic consolidation rewrites, splits, updates, and merges them into coherent topic documents. Retrieval can run over plaintext via lexical indexing rather than embeddings. At inference an agentic read procedure lets the LLM iteratively choose memory tools, inspect intermediate results, expand local context, and assemble evidence before answering.

Results On MemoryAgentBench the agentic-retrieval variant scores 64.7% overall and 81.2% on Accurate Retrieval, with gains on Factual Recall, Test-Time Learning, and Selective Forgetting. Ablations on LongMemEval_S isolate two complementary sources: holding the hybrid reader fixed, removing structural split-and-merge maintenance drops accuracy 76.0%->69.3% (-6.7, concentrated on knowledge-update and multi-session questions), while upgrading the reader from hybrid to agentic adds 3.3 points (76.0->79.3) — so maintenance matters more than the retrieval upgrade and neither is sufficient alone. A split-threshold sweep shows over-fragmentation is recoverable, but oversized documents that mix subtopics are costly.

Plain-language abstract GitOfThoughts stores an LLM agent's reasoning tree as a git repository — every scored thought is a commit, scores are git notes, validation outcomes are tags, and retrieval is git log over the agent's own history — which makes reasoning replayable, auditable, diffable, and mergeable across agents. The paper then asks the harder question of whether memory, in any substrate, actually improves accuracy, and runs a pre-registered comparison of five substrates (none, markdown, vector, graph, git).

Motivation Reasoning is the last unversioned software process: chains of thought expire with the context window, pruned search branches leave no record, and memory buffers cannot be diffed, merged, or audited. The authors argue this ephemerality is a structural blocker — it prevents reproducibility ('what did the agent think at step 17?'), audit (detecting train–test leakage or gold-answer memorization), memory transfer between agents, and incident review. Code, infrastructure, datasets, and experiments are all version-controlled; reasoning is the remaining outlier.

Methodology A reasoning tree shares git's structural invariants, so the paper maps it one-to-one onto git primitives (node = commit, refinement = parent edge, score = note, outcome = tag, session vs. cross-session = branch, retrieval = git log --grep / -S). A pluggable MemoryBackend routes every read/write through one interface so the same agent can swap substrate with a one-line change. To isolate retrieval from write-path noise, all five backends ingest identical answer-free lessons, then solve held-out, domain-stratified problems read-only; benchmarks are GPQA-Diamond and MATH-500, scored with paired-bootstrap CIs, across two backbones and pre-registered replications, with a similarity sweep to locate when retrieval helps.

Results H-substrate is supported: git delivers auditability, line-level diffs over reasoning text, deterministic replay by SHA, and mergeable memory at accuracy parity, costing ~15 ms/write and ~48 ms/read. H-memory is rejected: across two benchmarks, two backbones, and up to n=500, no substrate reliably improves accuracy on novel problems, and a +15 pp git trend at n=40 collapsed under its pre-registered replication. Memory pays only above a 'copyability threshold' — a near-duplicate retrieved case (cosine ≳ 0.8) lifts accuracy +12 to +13.5 pp, and a 4.5× larger model steepens that step to +22.5–28.5 pp but still extracts no transferable method; the only general accuracy lever is test-time sampling (self-consistency, +3.4 pp at n=500). The authors deliberately document a measurement bug, a retracted result, and a refuted hypothesis as the evaluation standard.

Plain-language abstract T-Mem is a long-term conversational-memory architecture for LLM agents that aims to make every stored memory reachable two ways: by surface similarity (descriptive recall) and by latent semantic/causal link (associative recall). It precomputes, at write time, four families of retrieval cues ('triggers'), one per quadrant of a granularity (item vs scene) x orientation (descriptive vs associative) design space, and retrieves via a topic -> scene -> item cascade fused with RRF over lexical and dense indices. It reaches state-of-the-art on LoCoMo (80.26%) and LoCoMo-Plus (74.81%).

Motivation Existing LLM memory systems (flat RAG, graph/temporal-KG, hierarchical, OS-style) all retrieve by projecting query and memory into one similarity space and taking top-K, so they are reachability-bounded by lexical/dense similarity. In long-running dialogue users rarely re-raise old topics with the same wording; they revisit them through indirect situational cues, so the target's surface form has drifted and can never be reached from the same neighbourhood. This associative half of the query-memory relation is a structural blind spot.

Methodology M is a typed tuple of scenes, items, topic labels, four trigger families, and per-speaker Persona, built by a load-bearing four-stage offline pipeline: event-closure scene segmentation, incremental data-grown topic labelling, dual-granularity item extraction (atomic + connected, one LLM call per topic), and trigger instantiation (Entity+Bridge jointly per item; Scene+Horizon per scene). Memory-construction LLM is GPT-4.1-mini, dense encoder bge-m3. Retrieval is a top-down topic -> scene -> item cascade scored by RRF over BM25 + per-type dense rankings; multi-view trigger indices surface host nodes via nan-aware max cosine, and associative triggers bypass the topic prefilter so cues outside the similarity neighbourhood still hit. Triggers stay off the QA evidence path; Persona is ambient context appended after retrieval.

Results On LoCoMo, T-Mem reaches 80.26% LLM-as-judge accuracy (51.96 token-F1), 3.25 pp above the strongest baseline HyperMem and the maximum on five of six columns. On LoCoMo-Plus it scores 74.81%, narrowing the LoCoMo-to-LoCoMo-Plus drop to 5.45 pp, about 5x tighter than HyperMem (28.38 pp) and near an order of magnitude tighter than the Mem0/SeCom/A-Mem cluster (~49 pp). Ablations confirm the scene-level associative triggers drive the associative gain: removing Scene+Horizon collapses LoCoMo-Plus by 22.19 pp (Horizon alone -12.47 pp) while moving LoCoMo by under 0.4 pp. T-Mem also reaches higher accuracy than HyperMem at a lower input-token budget.

Plain-language abstract Long-running language agents accumulate more history than fits in working context, and the usual fix is retrieval — store past events outside the model and fetch a relevant subset at query time. This paper argues retrieval only answers what can be fetched (memory access) and not what should keep shaping behavior after the working context is unloaded (memory depth). It introduces the loop-drift protocol, a controlled stress test where the retrieval index stays intact but working context is cleared, so goal-conditioned behavior must persist through long-loop interference without the relevant text being reinserted. It evaluates EVAF, a surprise- and valence-gated LoRA consolidation mechanism that writes only behavior-relevant events into a small adapter. Across GPT-2, TinyLlama, and Mistral-7B, retrieval wins shallow factual recall while EVAF wins goal persistence and post-unload recovery with only 2–3 parametric writes per 200 events, and the paper shows selective consolidation factorizes into two separable controls — selection and actuation.

Motivation Memory access and memory depth are different problems. A shallow memory is one the system can retrieve or attend to; a deep memory changes future behavior — it persists through interference, survives context unload, and affects choices without being reinserted as text. Retrieval is indispensable for fetched facts, but a long-running assistant also needs durable goals, preferences, and constraints that are not merely fetched facts. Existing long-memory benchmarks (LongMemEval, LoCoMo) emphasize conversational recall, temporal access, and knowledge updates, and do not isolate the post-unload setting where retrieval remains available but behavior must continue without the relevant text in context. The paper's claim is narrow and explicit: memory depth can be probed by post-unload goal-conditioned behavior, and consolidation factorizes into selection and actuation — it does not claim universal memory accuracy, SOTA retrieval, or complete deletion/update validity.

Methodology Loop-drift protocol: synthetic per-user streams of 200 events (10 users/run) mixing stable goal/preference reminders, off-topic distractors, transient opposite requests, conflicts, sibling-user contamination, and scheduled factual notes; four probe layers — shallow episodic (recent fact), noisy episodic (old fact after same-key interference), parametric tendency (does a stable goal still shape behavior after long interference), and post-unload recovery (re-probe the goal immediately after a context unload, with the retrieval index intact but working context cleared). The RAG baseline stores all events in a durable embedding index (top-3 cosine) that context unload does not clear, so any EVAF goal-layer advantage is not a trivial 'RAG forgot' artifact. EVAF mechanism: per-event surprise (token negative log-likelihood) and valence (embedding similarity to the user's durable goal/preferences) combine into an admission gate; events above threshold enter a buffer, and when the buffer fills a LoRA adapter is updated on the buffer plus replay from prior consolidated events, with an L2 anchor as a drift guard. Model controls: GPT-2 and TinyLlama (four-seed means) plus Mistral-7B. Selection is isolated with a matched-random gate (same write count and online write dynamics, random admitted events). Actuation is isolated with fixed-inner controllers (fixed-1/2/3 inner LoRA steps using the same gate). A routed EVAF+RAG variant routes factual probes to retrieval and goal probes to EVAF. Public Memora event streams serve as an external boundary diagnostic for stale-memory invalidation, tested with McNemar's test.

Results Depth flip: RAG is strongest on recent explicit facts (short-fact accuracy 0.956–0.973) and near-useless on goals; EVAF is near chance on short facts but much stronger on the goal layer — on GPT-2 EVAF reaches 0.904 goal / 0.900 post-unload vs RAG 0.398/0.394, and on TinyLlama 0.833/0.812 vs RAG 0.396/0.394 — at only 2.4–2.6 writes (L2 drift ~21–29) vs RAG's 0 writes. Writing everything is not enough: Naive-LoRA writes all 200 events at far higher drift (~67 TinyLlama, ~119 GPT-2) and still fails the goal layer; at 7B indiscriminate writing is actively harmful — Naive-LoRA goal persistence collapses to 0.333±0.047, below the 0.500 chance baseline. Selection is not sparsity: on GPT-2 EVAF beats a matched-random gate of equal write count on goal and post-unload in all four seeds (mean 0.790/0.763 vs 0.590/0.619); TinyLlama is weak/mixed, so the selection signal is not monotonic in model scale. Actuation is a separable, model-dependent factor: fixed-inner audits show smaller inner steps cut drift and improve goal/post (Mistral-7B five-step 0.354/0.306 -> Fixed-2 0.796/0.775 -> Fixed-1 0.919/0.938), but Fixed-1 contamination saturates at 1.000 on Mistral, so high actuation trades selectivity for goal strength. Asymmetric coupling: under a miscalibrated five-step actuation at 7B the matched-gate comparison reverses, yet EVAF still keeps lowest sibling contamination (0.787±0.041) — selection stays semantically active while its translation into goal behavior fails. Boundary: on Memora, EVAF improves forgetting-absence only 91/222 to 95/222 (p=0.57, not significant), so append-only selective consolidation does not solve stale-memory delete/update validity, which the paper leaves to validity-gating or reconsolidation.

Plain-language abstract MemStrata is a memory system for AI agents that keeps track of when facts become outdated. Instead of just retrieving whatever text looks most similar to a query, which fails when an old and a new fact look nearly identical, it uses a deterministic rule to detect when a new fact supersedes an old one and retires the stale version.

Motivation Retrieval-augmented memory has no concept of time: when a fact changes (a renamed function, an updated config value, a new port number), both the old and new versions sit in the store with near-identical embeddings, and the agent can't tell which is current. The authors show this isn't a tuning problem: on a calibrated dataset, cosine similarity separates contradictions from duplicates at only 0.59 AUROC (near chance), because a value-flip edit sits textually closer to the original than a genuine rephrasing does.

Methodology MemStrata's write path first tries a deterministic (subject, relation, object) triple match: if an incoming fact shares a key with a stored one but asserts a different value, the old fact is retired (not deleted) in a bi-temporal ledger and the new one is stored as current. Non-triple prose falls back to a similarity-plus-LLM-judge gate. The system is evaluated on six local, deterministic benchmarks (two static, four marker-free evolving: code mutation, config migration, dependency bumps, API evolution) with a 7B model on consumer hardware.

Results MemStrata matches RAG on static recall (no cost) and reaches 0.95-1.00 accuracy on evolving-knowledge benchmarks where RAG reaches only 0.20-0.47. When forced to answer, plain RAG serves the superseded value 15-40% of the time; MemStrata drives this to ~0%. It also runs at ~2.1s retrieval latency versus ~16-18s for LLM-reranking/verification baselines, since no LLM sits on the read path.

Plain-language abstract MOOSEDev gives coding agents ontology-grounded, typed project memory (decisions, constraints, rationales, lessons) in a small knowledge graph rather than flat notes or vector chunks, and shows it dramatically outperforms a production vector-memory tool on questions requiring completeness, negation, or supersession reasoning.

Motivation Coding agents generate most new code in many projects today, but the reasoning behind why changes were made is easy to lose track of; notes files and vector-based retrieval-augmented memory help but don't know what kind of record something is (a decision vs. a lesson vs. an outdated, superseded note) or how records relate to each other, which is fundamentally an ontological modeling problem, not a retrieval problem.

Methodology Two small OWL ontologies (9 and 11 classes) with SHACL-validated typed records, reasoned over by a neurosymbolic engine (MOOSE) that treats the LLM as a narrow, declared-point sensor rather than the primary reasoner, exposed to agents via MCP. Evaluated head-to-head against a production vector-memory tool (mem0) and a BM25 baseline on a neutral 835-record public corpus, using a pre-registered strict LLM judge, plus a live trial bootstrapped from the authors' own repository history.

Results On tasks requiring set-completeness, negation (absence), and supersession-traversal reasoning, MOOSEDev scored 0.98-1.00 versus 6-27% for the vector-memory baseline — a categorical, not incremental, gap. On simple relevance retrieval the two systems tied. In a live trial, the system provided 71 unprompted, relevant assists over the first three weeks against 38 misses, with most misses traced to a thinly-bootstrapped project.

Plain-language abstract A benchmark and method for a memory capability separate from recall: keeping track of which version of a fact is currently in force after it has been revised across sessions. The benchmark, StateMemBench, generates each scenario as a symbolic program of state operations so the correct answer is computed by replay and the specific way a lazy reader would get it wrong is known in advance. The method, StateMem, parses each turn into structured state units with typed dependency links, then handles supersession and staleness deterministically rather than with an LLM.

Motivation Existing memory systems and benchmarks optimize recall of relevant facts, but as agents run longer, facts, constraints and decisions get revised, and an answer must reflect the current state rather than a superseded one. The authors call the failure state drift: the relevant fact is present in the assembled context, but the agent acts on a stale or incomplete version of it. This is distinct from dialogue state tracking, which prescribes a slot-value representation and evaluates it directly over cooperative dialogues that accumulate a goal monotonically; here state is whatever a system must maintain to answer correctly, evaluation is purely behavioral, and revisions are adversarial across sessions. Some concurrent work centers state, but the authors argue none cleanly isolates state tracking from the other errors it co-occurs with.

Methodology The authors first define and label drift on existing benchmarks, assigning a failure to drift only after excluding retrieval, comprehension, schema and reasoning readings, dropping unassignable points, and cross-checking with two judge passes, a cross-family judge and two human annotators. They then build StateMemBench: each scenario is a symbolic event program of typed operations over ground, derived and declared state; the gold answer comes from deterministic replay; a family of executable lazy reader policies is run against the replay, and a scenario is admitted as a trap when policies disagree, with the disagreeing set forming its failure-mode signature (status, salience, sequence, compound, plus anti-trap controls). Programs are grounded in public data for surface vocabulary and rendered into multi-session dialogue by a strong LLM, then programmatically verified for fact placement and phrase leakage. Probes are closed-pool: an unseen pool of three to four options holds the gold answer, the targeted policy's drift answer and neutral distractors. StateMem itself runs a per-turn TurnEncoder producing state units (id, content, priority, source, deps), a deterministic update stage applying supersessions and marking dependents needs_recheck by dependency-graph traversal, and a single answer-time call over the assembled active state. A wrapper variant applies the same trace-then-resolve structure as a prompt-level transformation of any backend's answer call, evaluated against a length- and cost-matched generic-extraction control.

Results Drift leads the confirmed failure distribution on several existing benchmarks (63.5% on MemoryArena-shopping, 44.4% on LongMemEval oracle where retrieval is perfect by construction, 10 of 16 on tau-squared-bench-Z) but is not universal, falling to 19.0% on MemoryArena-travel. On StateMemBench, long-context is not the strong baseline it is on recall tasks: the best long-context model reaches 0.277 and same-backbone long-context 0.149. StateMem reaches 0.363 on DeepSeek-V4-Flash, 1.8x the best memory system and 2.4x same-backbone long-context, and 0.233 on Qwen-3.5-9B, 1.6x the best memory system, both significant by paired McNemar at p < 0.001; GraphRAG at 0.224 is statistically level on Qwen. Ablations put supersession marking as the largest single component and show dependency propagation over-fires on Set B anti-traps by 12.5 points, so removing it leaves DeepSeek slightly better. Drift-rate analysis shows the memory layer barely changes outcomes on a weak answerer, where every arm drifts at 61 to 66%, and separates on the stronger backbone, where StateMem's drift rate falls 15 points and correct answers rise by 42 while long-context, Mem0 and BM25 move by 1 to 3 points. The wrapper improves every one of six backends on both benchmarks, adding 31.7 to 66.6 points on StateMemBench with 15.0 to 31.7 attributable to state structure over the matched control, significant in all twelve cells. State tracking does not cost recall: StateMem also leads memory systems on LongMemEval (0.656 on DeepSeek) and LoCoMo (0.592), with margins concentrated on temporal-reasoning and knowledge-update question types. The authors note StateMem mirrors the policy family behind the traps, so its StateMemBench margins should be read as an upper bound.

Plain-language abstract This paper introduces AgentPoison, a method for attacking AI agents that rely on external memory or knowledge bases to answer questions and take actions. By inserting a small number of specially crafted malicious examples into an agent's memory, an attacker can reliably steer the agent into harmful behavior whenever a hidden trigger phrase appears in user input — while the agent behaves normally the rest of the time.

Motivation Large language model agents increasingly use retrieval-augmented generation, pulling past examples or knowledge from an external store to guide their decisions. This creates a new attack surface: existing threats like jailbreaking or backdoor attacks were designed for standalone models and do not reliably cause malicious examples to be retrieved from a diverse knowledge base, leaving this vulnerability largely unexplored.

Methodology AgentPoison poisons an agent's memory or RAG knowledge base with a very small number of malicious demonstrations, each pairing a user query containing an optimized trigger with a prescribed harmful action. The trigger is crafted through iterative gradient-guided discrete optimization that maps trigger-containing queries into a compact, unique region of the embedding space, maximizing the chance that poisoned examples are retrieved while keeping the trigger coherent and stealthy. The attack was evaluated on three real-world agent types: an autonomous driving agent, a knowledge-intensive question-answering agent, and a healthcare EHR agent.

Results AgentPoison achieved an average attack success rate of 80% or higher across all three agent types, with a poison rate below 0.1% of the knowledge base and a degradation in normal benign performance of 1% or less. The optimized trigger also showed strong transferability across different LLM backbones without requiring model training or fine-tuning.

Plain-language abstract This paper presents MemPoison, an attack method that lets an adversary secretly implant backdoors into the long-term memory of an AI agent simply by chatting with it. When a specific trigger phrase later appears in a conversation, the poisoned memory causes the agent to give misleading or malicious responses, while the attack remains hidden from standard defenses.

Motivation AI agents that use long-term memory to remember past interactions are increasingly common, but their memory systems create a new security weak point. Prior work on memory poisoning assumed injected content would be stored as-is, ignoring the fact that modern memory pipelines selectively extract and rewrite what gets remembered — making older attacks ineffective in realistic deployments.

Methodology MemPoison combines three technical components to survive the selective memory pipeline: a semantic relational bridge that binds a trigger and a malicious payload into a single coherent statement so both are extracted into memory together; entity masquerading that optimizes the trigger to resemble named entities, making it resistant to the rewriting step; and joint embedding optimization that clusters trigger-injected texts tightly in the embedding space while keeping them isolated from normal content for stealth. The attack is delivered through ordinary dialogue interactions rather than direct memory writes.

Results Evaluated across multiple agent application domains and memory mechanisms, MemPoison achieved attack success rates of up to 0.95, outperforming existing baseline attacks. Mechanistic analysis showed the attack works by exploiting embedding-space anisotropy and shifting the agent's attention patterns. Multiple evaluated defense strategies were found to have fundamental limitations and could not fully mitigate the attack.

Plain-language abstract This paper identifies and studies a security vulnerability in AI assistants that remember things about you over time: normal, everyday conversations can gradually corrupt an agent's stored state, quietly loosening the safety guardrails that govern what the agent is allowed to do on your behalf. The authors build a benchmark to measure the problem and propose a lightweight defense to catch and roll back dangerous memory updates before they take hold.

Motivation Personalized LLM agents maintain persistent cross-session memory so they can act as long-term collaborators, but that same persistence creates a new attack surface. Prior security research focused on explicit adversarial injections; this paper asks whether ordinary, benign-looking interactions can accumulate over time to shift an agent's stored defaults in harmful directions — weakening confirmation requirements, expanding tool-use permissions, or increasing autonomous behavior — without any explicit attacker.

Methodology The authors introduce ULSPB (Unintended Long-Term State Poisoning Bench), a bilingual (English and Chinese) benchmark of 350 conversation settings spanning five assistance categories and seven interaction patterns, each instantiated as a 24-turn simulated routine conversation, with matched 25-turn single-injection variants for comparison. They define the Harm Score (HS), a deterministic diff-level metric that scores three dimensions of state drift — authorization drift, tool-use escalation, and unchecked autonomy — weighted by the sensitivity of the affected state component. Experiments run on the OpenClaw personalized agent framework with four backbone LLMs. The proposed defense, StateGuard, audits state diffs at the writeback boundary and selectively rolls back dangerous edits; it is evaluated in single-auditor and majority-vote ensemble configurations and compared against a perplexity-based baseline.

Results Routine conversations alone induced substantial long-term state drift across all four backbone models, not just explicit single-injection attacks, with poisoning primarily occurring in memory-centric artifacts such as MEMORY.md. StateGuard reduced HS to near zero across all models: the Targeted-Ensemble configuration achieved HS scores of 0.06, 0.05, 0.14, and 0.03 on the four tested models, compared to baseline scores above 4.0 for the perplexity-based defense. StateGuard's auditing cost averaged below $0.004 per interaction run. Evaluations seeded with real-world user interactions confirmed that the risk is not an artifact of synthetic prompt construction.

Plain-language abstract ShadowMerge is a cyberattack designed to poison the memory of AI agents that use knowledge graphs to store and recall information. By injecting a carefully crafted false relationship into shared graph memory, an attacker can cause the agent to retrieve and act on corrupted information in future interactions — all without any special access beyond ordinary user interactions.

Motivation AI agents increasingly use graph-based memory systems to store structured facts and reason across sessions, but the security of this memory type has not been studied. Existing poisoning attacks target flat text records and fail against graph memory because malicious content must survive graph extraction, entity merging, and retrieval steps — each of which can silently discard a naive attack payload.

Methodology The authors developed ShadowMerge, a black-box attack requiring only query-level access to the target agent. Its core primitive, Channel-Aligned Relational Competition (CARC), crafts a poisoned relation that shares the same anchor entity and relation channel as legitimate graph evidence while carrying a conflicting value. A three-stage pipeline called AIR — Anchor, Inscribe, and Render — constructs the payload so it survives graph extraction, merges into the correct entity neighborhood, and is retrieved for the target query. The attack was evaluated on the Mem0 memory framework and three public datasets: PubMedQA, WebShop, and ToolEmu.

Results ShadowMerge achieved an average attack success rate (ASR) of 93.8% across the three datasets, a 50.3 percentage-point absolute gain over the best baseline attack, while having negligible impact on unrelated benign tasks. A mechanism study confirmed the attack overcomes all three limitations of prior work: relation extraction, anchor-neighborhood merging, and target-query retrieval. Analysis of representative input-side defenses found them insufficient to mitigate ShadowMerge.

Plain-language abstract MemLineage is a security system for AI agents that tracks the origin and derivation history of everything stored in an agent's long-term memory. Because modern agents accumulate chat logs, ingested documents, and tool outputs across sessions, an attacker can plant malicious content that later gets summarized by the agent itself into an authentic-looking memory entry—a chain-of-custody failure. MemLineage prevents such laundered memory entries from authorizing sensitive actions by tagging every memory with cryptographic signatures and a lineage graph showing which prior entries influenced it.

Motivation LLM agents that persist memory across sessions are vulnerable to a subtle attack: an adversary injects untrusted content that the agent's own retrieval-and-summarization loop transforms into a new entry committed under the agent's own identity. Existing defenses—signature-only layers, information-flow control planners, and retrieval-stage filters—cannot distinguish a benign summary from a laundered payload arriving through an authentic principal, leaving a gap between useful long-term memory and prevention of untrusted ancestry from authorizing sensitive actions.

Methodology MemLineage is implemented as a Python library integrating with LangGraph and consists of six modules around a single memory store. Each memory entry is signed with per-principal Ed25519 keys anchored in an RFC 6962 Merkle log; a weighted directed acyclic graph records LLM-mediated derivation links between entries. A max-of-strong-edges propagation rule enforces that any chain descending from an external ancestor with strong attribution edges inherits an untrusted label, which the sensitive-action gate refuses to act on. Three attribution algorithms are compared (uniform-weight, secondary-LLM judge with prompt-injection hardening, and white-box attention), and a tau-by-K ablation sweeps the lineage threshold against derivation chain length. Evaluation uses a deterministic mechanism-isolation harness that pins attacker behavior without real LLM API calls, plus a Codex-backed AgentDojo bridge under an intentionally vulnerable tool-output profile.

Results On the deterministic harness comparing three defense configurations against three attack families (AgentPoison-style, MemoryGraft-style, and sleeper-via-derivation), the no-defense baseline fails all three columns and the signature-only baseline fails two of three, while MemLineage drives all three attack success rates to zero. The tau-by-K ablation reveals that allowing K=5 derivation hops requires a threshold of tau <= 0.10 rather than the tau <= 0.30 that a K=1 measurement would suggest. Per-operation overhead is sub-millisecond on the hot path, well below the noise floor of any LLM call. On the AgentDojo bridge under a vulnerable tool-output profile, no-defense and signature-only baselines fail on all six banking task pairs, while all MemLineage configurations reduce strict AgentDojo attack success rate to zero.

Plain-language abstract This paper introduces Agent-BOM (Agent Bill of Materials), a structured representation designed to make the behavior of AI agent systems auditable from a security perspective. It models an agentic system as a hierarchical graph that captures both the static components (models, tools, memory stores) and the dynamic runtime states (goals, reasoning steps, actions), then provides a graph-query framework for tracing how security threats enter, propagate, and cause harm.

Motivation Modern LLM-based agents autonomously invoke tools, maintain persistent memory across sessions, and collaborate with other agents — creating a large semantic gap between low-level system events and high-level execution intent. Existing security representations such as software bills of materials and runtime logs record what happened but cannot explain how a goal was formed, how context was contaminated, or how a malicious instruction propagated across agents and sessions, leaving post-incident auditing fundamentally incomplete.

Methodology The authors define Agent-BOM as a hierarchical attributed directed graph with two layers — a static capability base (models, tools, long-term memory) and a dynamic semantic-state layer (goals, context, reasoning trajectories, decisions, actions) — connected by typed semantic edges carrying security attributes. On top of this representation they develop a four-stage graph-query auditing paradigm: entry localization, backward tracing, forward tracing, and attribute adjudication. They instantiate and validate the framework against the OWASP Agentic Top 10 threat list, deploying an auditing plugin in the OpenClaw environment to construct Agent-BOM graphs from live agent executions.

Results Evaluation on representative real-world agentic attack scenarios showed that Agent-BOM accurately reconstructed stealthy attack chains across four threat classes: cross-session memory poisoning and tool misuse, capability supply-chain hijacking and unexpected code execution, multi-agent ecosystem hijacking, and privilege and trust abuse. In each case the graph-query paradigm successfully traced the full causal path from the malicious entry point through intermediate semantic-state changes to the realized harmful action, demonstrating that Agent-BOM can support root-cause analysis and security adjudication in complex multi-agent ecosystems.

Plain-language abstract This paper proposes a structured security framework for agentic AI systems — AI that does not just answer questions but plans across long horizons, remembers past interactions, calls external tools, and coordinates with other agents. The authors introduce the Layered Attack Surface Model (LASM), which maps security threats to seven distinct architectural layers, and add a temporal dimension to capture how some attacks unfold slowly across sessions rather than in a single moment. The work is anchored by a systematic review of 94 papers published between 2021 and 2025.

Motivation Existing security analyses of AI systems organize threats by attack type (prompt injection, jailbreaking) without specifying which architectural component is vulnerable or how quickly a threat manifests. This leaves system designers unable to determine where to place security controls. The problem is compounded by the fact that agentic systems introduce genuinely new threat classes — such as long-term memory poisoning and multi-agent collusion — that have no direct analogues in classical LLM safety or traditional software security.

Methodology The authors conducted a systematic literature review following a PRISMA-inspired protocol, searching IEEE Xplore, ACM Digital Library, arXiv, and Google Scholar for papers from January 2021 to April 2025. From an initial pool of 1,247 records, 94 papers were retained after deduplication and full-text screening. Each paper was coded into one or more cells of a LASM-layer by attack-temporality matrix (7 layers × 4 temporal classes, yielding 120 paper-cell assignments), enabling a coverage heatmap that reveals which threat combinations are well-studied and which are neglected.

Results The analysis shows that the most dangerous emerging threats concentrate at the intersection of high-layer attacks (layers 5–7: multi-agent coordination, ecosystem supply chain, and governance) and slow-burn temporality (cross-session and weight-level threats): specifically covert agent collusion, long-term memory poisoning, MCP supply-chain compromise, and alignment failure as an insider threat. Only 8 of 120 paper-cell assignments (7%) fall in this high-risk zone, confirming it is severely under-studied. The paper identifies five research gaps in this zone, notes that two (emergent misalignment detection and steganographic collusion detection) have no near-term solution path, and provides a cross-layer defense taxonomy showing which threat classes existing defenses leave unaddressed.

Plain-language abstract This paper introduces Phantom, an attack framework that can compromise AI chatbots built with Retrieval Augmented Generation (RAG) — systems that answer questions by pulling in relevant documents from a knowledge base. By injecting a single carefully crafted document into that knowledge base, an attacker can manipulate the chatbot's responses in targeted ways whenever a victim's query contains a specific trigger word or phrase.

Motivation RAG systems are increasingly used in commercial applications such as search engines, customer service bots, and personal assistants, and their knowledge bases often draw from large, hard-to-verify sources. This creates a security gap: a malicious actor who can insert even one document into the knowledge base can potentially steer the system's outputs, but prior work had not systematically studied or demonstrated such targeted backdoor poisoning attacks against RAG.

Methodology Phantom uses a two-stage optimization process. In the first stage, a poisoned document is optimized in embedding space so that it is retrieved only when the victim's query contains a chosen trigger token sequence, keeping the attack dormant otherwise. In the second stage, an adversarial string is appended to the document using a Multi-Coordinate Gradient (MCG) optimization strategy to induce specific harmful outputs — including refusal to answer, biased opinions, harmful content, private data exfiltration, and unauthorized API calls. Attacks were evaluated across three datasets, three retriever architectures, and seven LLM generators (Gemma-2B through GPT-4), using thirteen distinct triggers.

Results Phantom achieved high attack success rates across all tested objectives and model families. For the refusal-to-answer objective, all four tested generators (Gemma-2B, Vicuna-7B, Gemma-7B, Llama3-8B) were manipulated with high success rates without needing the MCG optimization step. For biased opinion generation, adding MCG yielded improvements of roughly 38–39 percentage points for models that resisted the command alone. Attacks transferred to GPT-3.5 Turbo and GPT-4 without white-box access to those models, and the researchers successfully executed a Phantom attack on NVIDIA's production black-box RAG system, Chat with RTX.

Plain-language abstract PrivacyLens is a benchmark framework for testing whether AI language models respect privacy norms when acting as agents on behalf of users — for example, drafting emails or posting to social media. It generates realistic test scenarios from privacy principles, then measures whether a model leaks sensitive information in its actual outputs, not just in how it answers abstract questions about privacy.

Motivation Language models increasingly act as agents with access to sensitive personal data such as calendars and emails. Prior evaluations of privacy awareness relied on asking models direct questions, but there is a growing gap between how models answer those questions and what they actually do when executing real tasks. No existing framework captured this behavioral gap in realistic, agentic communication settings.

Methodology The authors built PrivacyLens by grounding a set of privacy norms in academic privacy literature and crowdsourced real-world seeds. Each seed was expanded into a detailed vignette describing a data type, subject, sender, recipient, and transmission context, then further extended into full agent trajectories — sequences of tool calls and observations an LM agent would produce when completing a communication task. Models were evaluated both on probing questions (do they know the norm?) and on their actual agent behavior (do they follow it?), enabling direct comparison between stated awareness and in-action behavior.

Results State-of-the-art models leaked sensitive information in a substantial fraction of cases: GPT-4 leaked in 25.68% of agent trajectories and Llama-3-70B in 38.69%, even when prompted with privacy-enhancing instructions. The study also revealed a consistent discrepancy between models' high performance on privacy probing questions and their much weaker privacy behavior when executing user instructions as agents.

Plain-language abstract This paper identifies and demonstrates a new class of security attack called Indirect Prompt Injection, where malicious instructions are hidden inside content that an AI assistant retrieves from the web or other sources. When the AI reads that content, the hidden instructions hijack its behavior—without the user or attacker ever typing anything directly into the chat. The authors show these attacks work against real deployed systems and can cause the AI to steal user data, spread misinformation, or act as an automated social engineer.

Motivation Prior work on prompt injection assumed an attacker had direct access to the AI's input. As AI assistants began integrating with search engines, email clients, and code tools—routinely ingesting untrusted external data—a new and largely unexamined attack surface opened up. The paper argues that the rapid deployment of LLM-integrated applications outpaced safety evaluations, leaving millions of users potentially exposed to adversaries who never interact with the system directly.

Methodology The authors developed a comprehensive threat taxonomy mapping classic computer-security concepts (intrusion, persistence, malware, denial of service, data exfiltration) to the novel LLM-integrated application setting. They then constructed and tested concrete attack prompts against both synthetic GPT-4-based applications with controlled functionality and real-world systems including Bing Chat (GPT-4 powered) and GitHub Copilot. For Bing Chat, injections were delivered by embedding instructions in HTML comments on pages read via the Edge sidebar feature, allowing local testing without public poisoning. Attack scenarios covered information gathering, fraud, malware spreading, prompt-worm propagation, and code-completion manipulation.

Results The attacks proved practically viable across all tested systems. Indirectly injected prompts successfully steered model behavior in ways that direct-interface jailbreak filters blocked: Bing Chat halted sessions for directly entered jailbreaks but obeyed the same instructions when they arrived via retrieved content. The compromised model retained injected instructions across multiple conversation turns, used conversation context to augment persuasion, and could exfiltrate user-disclosed information (such as a journalist's identity) through markdown hyperlinks or search queries to attacker-controlled URLs. GitHub Copilot was also shown to be susceptible to injections placed in retrieved code. The authors conclude that effective mitigations for these attacks are currently lacking.

Plain-language abstract This paper demonstrates a new class of cyberattack called Morris-II, a computer worm that can spread automatically through networks of AI-powered applications. When these apps share information using a technique called Retrieval-Augmented Generation (RAG), a specially crafted malicious prompt can replicate itself from one app to the next, stealing private data and infecting other apps along the way — all without any user clicking anything.

Motivation AI-powered applications like email assistants increasingly rely on RAG-based inference, where models retrieve external data to answer queries. Prior security research focused on attacks against single AI applications, leaving open whether attackers could scale such attacks across entire interconnected ecosystems of GenAI apps. This paper addresses that gap by asking whether a worm-like chain reaction could propagate malicious behavior across a whole GenAI ecosystem.

Methodology The researchers designed adversarial self-replicating prompts — the core mechanism of Morris-II — and conducted an end-to-end evaluation against RAG-based GenAI-powered email assistants using the Enron email dataset. Each employee's personal RAG database was built from 100 of their emails (2,000 emails total). The study tested how worm performance varied across five embedding algorithm types and sizes, context window sizes, GenAI engine types (including GPT and Gemini variants), and number of propagation hops. They also developed and evaluated a guardrail called the Virtual Donkey to detect and block worm propagation.

Results The evaluation showed that attackers can craft emails that extract sensitive user data from the RAG context and embed it into AI-generated replies, which then propagate the worm to new recipients and compromise their assistants. The proposed Virtual Donkey guardrail achieved a perfect true-positive rate of 1.0 with a false-positive rate of only 0.015, and remained robust against out-of-distribution worms using unseen jailbreaking commands, a different email dataset, and various attack scenarios.

Plain-language abstract This paper introduces a new type of attack on AI question-answering systems that use retrieval-augmented generation (RAG), where a language model looks up documents from a database before responding. The authors show that an attacker who can add just one specially crafted document — called a "blocker" document — to the database can cause the system to refuse to answer specific questions, claiming it lacks information or that the answer is unsafe.

Motivation RAG systems are widely used but are exposed to adversarial content whenever their underlying databases contain user-contributed or externally sourced documents. Existing safety evaluations for large language models do not assess this kind of denial-of-service risk, leaving a gap where an attacker can silently suppress specific information — for example, hiding negative reviews, concealing legal facts, or blocking regulatory disclosures — without producing obviously toxic output that would trigger standard safety checks.

Methodology The authors developed and compared several methods for generating blocker documents, including a black-box optimization approach that requires only query-level access to the RAG system — no knowledge of the embedding model or the underlying language model, and no auxiliary LLM. They evaluated these jamming attacks across multiple LLMs and embedding models, and also examined whether existing safety and trustworthiness metrics for LLMs capture vulnerability to this kind of attack. Potential defenses against blocker documents were also analyzed.

Results A single blocker document added to a RAG database is sufficient to cause the system to refuse to answer a targeted query. The paper demonstrates that resistance to jamming is a novel safety property distinct from those measured by current LLM safety benchmarks — meaning models that score well on standard safety evaluations can still be successfully jammed. The black-box optimization method for generating blocker documents works without knowledge of the target system's internals.

Plain-language abstract This paper is a survey of the security and privacy risks that arise specifically in LLM-based AI agents — systems that go beyond static language models to take actions, use tools, and interact dynamically with users and other agents. The authors categorize threats, analyze real-world impacts, review defensive strategies, and use case studies to make the material accessible.

Motivation LLM agents are being deployed widely in customer service, virtual assistants, and other high-value settings, yet security research has focused almost entirely on the underlying language models. The agents introduce new attack surfaces because their dynamic capabilities — tool use, multi-step reasoning, persistent memory — mean a single compromised action can cascade into broader harm. A comprehensive threat taxonomy covering both inherited LLM weaknesses and agent-specific vulnerabilities was missing.

Methodology The authors conducted a structured literature survey organized around the LLM agent workflow of thought, action, and perception. They classified threats into two top-level categories: threats inherited from LLMs (technical vulnerabilities such as hallucinations and catastrophic forgetting, plus intentional attacks such as data extraction and instruction-tuning attacks) and threats unique to agents (knowledge poisoning, functional manipulation, and output manipulation). They supplemented the taxonomy with case studies illustrating each threat type in realistic scenarios.

Results The survey identifies and categorizes a broad set of emerging threats, showing that agent-specific attacks — particularly functional manipulation via compromised tools and knowledge poisoning of training or memory data — pose risks that static LLM defenses do not address. The authors also map impacts across three dimensions: harm to human users, harm to the operating environment, and harm propagated to other interacting agents. Existing defensive strategies are reviewed and gaps for future research are outlined.

Concrete memory-injection attack.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Privacy leakage from agent memory stores.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Risk framing for episodic agent memory.

Why it matters Surfaced by the 2026-06-11 dense-lane rerun (missed in the lexical-only window).

Plain-language abstract A benchmark and method for a specific skill-retrieval failure: the retriever finds the right capability family but surfaces the wrong member of it, one that shares the vocabulary and procedure shape while pointing at a stale resource, skipping a precondition, or applying the wrong check. The benchmark pairs each query's helpful skill with that query-specific risky sibling inside a large public candidate pool and scores helpful ranking alongside a harmful sibling rate. The method resolves candidates into capability families, scores query-conditioned utility, and exposes one representative per family before the final top-K list.

Motivation Skills have become loadable operational artifacts carrying instructions, scripts, resources and metadata, so the retrieval layer decides which procedural context enters an agent before planning or execution. That makes retrieval failures more specific than broad irrelevance. Public skill audits find weak routing metadata, non-actionable bodies, reusable-artifact defects and technical debt, and gains from curated skills weaken when agents must retrieve from large real collections. Related work has shown top-K skill quality cannot be reduced to independent query-skill relevance because the retrieved set must be compatible as a set, and security work has established that loaded skill packages affect planning, context, permissions, scripts and local resources. Existing positive-skill retrieval benchmarks identify useful or gold skills but assign no query-specific execution-risk siblings, while malicious-skill detection and permission enforcement act after admission rather than at the retrieval-time choice of representative.

Methodology The setting is formalized as same-capability execution-risk retrieval: a collection of queries, each with a candidate pool, an admitted helpful skill, a query-specific execution-risk sibling, and a released family relation. Construction starts from a task-facing skill admitted for a query, and the paired sibling changes exactly the condition that makes the procedure usable, keeping the sibling plausible under ordinary semantic retrieval. Candidate pools add library pressure by ranking the pair among unrelated and partially related public skills drawn from a public SkillRet corpus, so a system must both recover active capability families and choose the right representative within each. Evaluation reports Recall@K and NDCG@K for the helpful skill together with HSR@K, the rate at which the risky sibling appears in the final top-K. The release records source role and admission evidence, risk taxonomy, cue and leakage checks, hashes, query-disjoint splits and held-out outputs. The method, SkillResolve, has three components: a Capability Resolver returning active candidate groups that should compete as alternative representatives (singleton groups allowed, which recovers ordinary ranking); a query-conditioned Utility Scorer trained with admitted helpful skills as positives and confusable library alternatives mined under the same query and pool protocol as negatives, using ordinary retrieval signals plus contract-profile cues; and a Representative Selector keeping the highest-utility member of each resolved group before final ranking. Family sources can also be derived from public metadata or skill text, which trades exposure against recall.

Results Same-capability ambiguity is prevalent in public libraries: 4,716 of 4,997 audited SkillRet queries have at least one non-gold skill in the gold skill's domain/action/object family, and across seven standard retrievers a top-three list surfaces such a sibling for 47.3% of queries, with 36.6% containing both the gold skill and a strict same-family sibling. Generic lexical retrieval, SkillRouter and BGE reranking retrieve relevant skills but expose risky siblings, while an attribution-listwise baseline suppresses them at the cost of helpful retrieval quality. SkillResolve reaches Recall@3 0.766, NDCG@3 0.699 and HSR@3 of 0 under the released family relation, improving over SkillRouter by 0.112 Recall@3 and 0.165 NDCG@3 while reducing HSR@3 from 0.693. Component analysis identifies representative selection as the controlling mechanism: with the same utility scorer but no representative selection, HSR@3 rises to 0.236. The recall-exposure tradeoff depends on the quality of the family source, since a resolver that splits a helpful skill and its risky sibling into different groups lets both survive into the final ranking. Note on versions: SciX holds the June 2026 v1 of this arXiv entry, titled SkillResolve-Bench with 661 helpful/risky pairs and a 7,982-candidate pool; a later revision circulates as SameCapRisk-Bench with a larger unit count and a different baseline table. The mechanism and the metric are unchanged across both; the figures above are the v1 body.

Plain-language abstract Agent Skills package procedural knowledge as markdown plus scripts, and that format describes how an agent should behave without deciding which behavior may become an action. As Skills move to settings that drive relays and locks, the gap becomes a physical one. The paper names Borrowed Authority: an inter-agent message carrying an instruction plus an unverifiable permission claim, which the receiving agent has no typed way to reject. Edge Skillguard answers it with a typed authority layer inside the Skill artifact - guard predicates over world state, leases and sensor evidence, expressed as a schema-validated policy file with a pure-function evaluator. On a live edge control plane it rejects 60/60 attacks across five variants while preserving all benign requests, holding at 5x scale and across hosts.

Motivation Self-evolving Skill harnesses generate orchestration automatically and report efficiency gains, not safety ones, so generating more Skills scales the advisory layer and leaves the authority decision to the model. Two adjacent attacks are already public: malicious skills distributed through community registries, plus a Claude Code project-file misconfiguration (CVE-2026-21852) that routed a session's API tokens to an attacker before trust was established, and jailbreaks of LLM-controlled robots reaching up to 100% success against deployed commercial platforms including a self-driving LLM, a wheeled UGV and a quadruped. Their intersection - a compromised skill artifact causing physical-state harm - is the open cell of the paper's attack table, and the authors construct and defend it before an in-the-wild incident rather than after.

Methodology An Edge Skillguard artifact is a tuple of orchestration states, typed world state (sensors, user identity, leases, device state, time, shared-state commit), typed envelope events, deterministic actions, guard predicates over state and event, a transition relation, bounded LLM holes, and inter-agent contracts incoming messages must satisfy. Guards are seven typed operators over dot-separated paths into envelope or state, shipped as a JSON-Schema-validated policy plus a pure-function evaluator that emits either an inbox publish or a structured policy_block log naming the failed predicates. It sits on a messaging substrate providing schema-validated typed envelopes, broker-attested sender_id from the connection token, durable per-agent FIFO inboxes, an audit-mirror outbox, and boundary rejection of malformed envelopes. Four conditions are compared - plain Skill, natural-language machine-to-machine with no receiver guard, a lease-only ablation, and the full policy - over 60 Borrowed Authority requests across five variants (stale presence, missing presence source, wrong-grantee lease, expired lease, lease-scope mismatch), 12 each, plus 60 benign requests, run in-process, on a live NATS broker against a Home Assistant deployment of 148 entities, and cross-host over Tailscale. Test subjects are isolated so no device adapter fires; the measurement is whether an unauthorized transition reaches the adapter boundary.

Results The full typed policy rejects 60/60 Borrowed Authority requests and preserves 60/60 benign ones, versus 60/60 wrongful actuation for both the plain Skill and the natural-language machine-to-machine baselines. The 5x run holds 300/300 attack rejection and 300/300 benign success at p95 399 microseconds on the live broker, and the cross-host Tailscale run holds the same correctness at p95 7.9 ms. Median latency is 3.2 microseconds in-process, 273 microseconds on the local broker, 5.7 ms over the mesh, with 0 LLM calls per decision against 1 for the advisory baselines. The lease-only ablation catches 36/60, covering the three lease-bound variants and letting both sensor-bound variants through, so freshness and source predicates carry the remainder. Blocked requests surface the failed predicates rather than a model rationale. The authors do not claim automated policy synthesis; hand-authored guards are the supported workflow, and the attack class is constructed rather than observed in the wild.

Plain-language abstract This paper studies whether AI coding agents can benefit from memories collected in one programming domain when working on tasks in a different domain. The authors introduce Memory Transfer Learning (MTL), which pools experiences from heterogeneous coding tasks—such as software engineering, machine learning development, and competitive programming—and makes them available to agents tackling new problems. They test this idea across six coding benchmarks and four different ways of representing memories, from raw execution traces to high-level abstract insights.

Motivation Most memory-augmented coding agents are restricted to reusing experiences from the same task domain, ignoring the fact that diverse programming problems share common infrastructure like Linux shells and programming languages. As gains from scaling training data plateau, self-evolution through accumulated experience offers a path to further improvement, but existing systems fail to exploit knowledge across domain boundaries—leaving a large pool of potentially transferable experience untapped.

Methodology The researchers evaluated MTL across six coding benchmarks spanning software engineering (SWE-level), machine learning, and competitive coding tasks. They used four memory representations of varying abstraction: concrete execution trajectories, code snippets, experiential planning and debugging traces, and high-level abstract insights. A unified memory pool drawn from all heterogeneous domains was made available during retrieval, and performance was compared against single-domain memory baselines.

Results Cross-domain memory transfer improved average performance by 3.7% across the six benchmarks. The primary source of benefit was meta-knowledge—operational know-how such as validation routines and structural inspection strategies—rather than task-specific code. The paper found that abstraction level determines transferability: high-level abstract insights generalize well across domains, while low-level execution traces often cause negative transfer by introducing excessive task-specific detail. Performance gains also scaled with the size of the memory pool and the number of contributing domains, and memory was shown to transfer effectively even between different underlying models.

Plain-language abstract SkillDroid is an Android automation system that learns from a task the first time an AI completes it, then replays that learned sequence on later requests without involving the AI again. Instead of reasoning through every step of a repeated task from scratch, it stores a reusable action template and replays it at machine speed, falling back to AI assistance only when something unexpected occurs.

Motivation Current AI-based mobile agents treat every task execution as an independent reasoning episode, calling the language model at each action step. This means a task completed successfully yesterday is re-derived entirely from scratch today, with no gain in speed or reliability. LLM calls account for 75-94% of total agent execution time, and in testing a stateless agent's success rate degraded from 80% to 44% over 150 rounds as instruction phrasing became more varied, because past successes provide no benefit to future attempts.

Methodology SkillDroid uses a three-layer architecture running on top of an open-source Android automation framework. In Layer 1, an LLM guides task execution step by step and, on success, a skill compiler extracts a parameterized template — a sequence of UI actions with weighted element locators and typed parameter slots — stored in a local SQLite database. In Layer 2, a matching cascade using regex patterns and embedding-based semantic similarity routes new instructions to stored skills for replay via Android's accessibility interface with zero LLM calls, using a state verifier to detect UI deviations and gracefully fall back when needed. In Layer 3, a failure-learning component tracks skill reliability and triggers recompilation when a skill's failure rate exceeds 50%, progressively replacing noisy initial compilations with cleaner versions.

Results Over a 150-round longitudinal evaluation spanning 15 task types across diverse Android applications with systematic instruction variation and controlled perturbations, SkillDroid achieved an 85.3% success rate — 23 percentage points above a stateless LLM baseline — while using 49% fewer LLM calls. The skill replay mechanism achieved a perfect 100% success rate across 79 replay rounds at 2.4 times the speed of full LLM execution. Critically, the system's success rate converged upward from 87% to 91% across experimental phases, while the baseline degraded from 80% to 44%, demonstrating that SkillDroid improves with use while stateless agents become less reliable as task variation grows.

Plain-language abstract This paper introduces Reflective Memory Management (RMM), a system that helps AI dialogue agents remember and retrieve relevant information across many conversations over time. Because large language models are stateless — they forget everything between sessions — the authors built an external memory mechanism with two complementary parts that work together to store and recall user information more accurately and adaptively.

Motivation AI assistants used in customer service, healthcare, or education need to remember what individual users have told them across many past conversations, not just the current session. Existing external memory systems for language models suffer from two problems: they store information at rigid, pre-defined chunk sizes (such as per conversation turn or per session) that do not match natural topic boundaries, and they rely on fixed retrieval mechanisms that cannot adapt to different users or dialogue styles, making personalized long-term interaction difficult.

Methodology The RMM framework combines two mechanisms. Prospective Reflection dynamically decomposes dialogue history into topic-based memory units across multiple granularities — individual utterances, turns, and whole sessions — so that semantically related content is grouped together regardless of where session or turn boundaries fall. Retrospective Reflection treats retrieval improvement as an online reinforcement learning problem: as the language model generates responses, it cites which retrieved memories were useful, and those citation signals are used as unsupervised reward feedback to iteratively refine the retriever without requiring any labeled training data. The system was evaluated on two benchmarks, MSC and LongMemEval, using Contriever and GTE retrievers with Gemini-1.5-Flash and Gemini-1.5-Pro as generators.

Results RMM achieves more than 10% accuracy improvement over a no-memory baseline on LongMemEval, and more than 5% improvement over the strongest prior baseline across memory retrieval and response generation metrics on both benchmarks. The full RMM framework reaches a METEOR score of 30.8% on MSC and Recall@5 of 60.4% on LongMemEval. Citation-based scoring used in Retrospective Reflection was validated as reliable, achieving an overall F1 of 86.7% for identifying useful versus non-useful retrieved memories. Experiments also show that flexible topic-based granularity from Prospective Reflection approaches oracle-level performance compared to any single fixed granularity strategy.

Plain-language abstract This paper introduces TSUBASA, a framework for making AI language models better at personalizing their responses over long stretches of time—for example, tracking a user's preferences and history across many conversations. It combines two components: one that maintains and updates a structured memory of the user, and one that trains the model to internalize that memory more effectively so it can answer personal questions without needing to look up everything at retrieval time.

Motivation Personalized language models struggle with long-horizon tasks—situations where they must reason across a user's extensive history of interactions. Existing memory systems grow linearly and fail at tasks requiring implicit temporal reasoning; retrieval-augmented approaches face a quality-efficiency tradeoff where better answers require retrieving more context at prohibitive cost; and fine-tuning approaches are held back by a train-inference gap because raw conversation data does not prepare models well for the complex personalization tasks they face at evaluation time.

Methodology TSUBASA uses a two-wing design. The first wing handles dynamic memory writing: core factual observations are extracted from raw conversations and a memory manager applies structured operations (ADD, UPDATE, RECONCILE, IGNORE) to keep the memory store compact and conflict-free. The second wing handles internalized memory reading through a self-learning pipeline that applies a teacher-student context distillation objective on synthetic question-answer pairs: a frozen teacher model sees the full session context, while a trainable student model sees only the question, and the distillation loss trains the student to internalize user-specific knowledge parametrically. Evaluations were conducted on long-horizon personalization benchmarks using the Qwen-3 model family ranging from 4B to 32B parameters, compared against memory-augmented baselines including Mem0 and Memory-R1.

Results TSUBASA surpasses competitive memory-augmented systems that rely primarily on memory writing, such as Mem0 and Memory-R1, on long-horizon benchmarks. The framework achieves Pareto improvements over prior approaches—delivering higher personalization fidelity while using a reduced token budget—effectively breaking the quality-efficiency tradeoff. The approach also preserves user privacy by avoiding cross-user training.

Plain-language abstract This paper introduces Mem2Evolve, a framework for building AI agents that can improve themselves over time by combining two types of memory: a store of past task experiences and a store of dynamically created tools and specialist sub-agents. Rather than treating these two growth processes separately, the system makes them work together so each one reinforces the other.

Motivation Existing self-evolving agent frameworks either let agents learn from experience but keep them locked to a fixed set of tools, or let agents create new tools and sub-agents but do so blindly without drawing on past experience. Both approaches are limited: the first cannot expand what the agent is capable of doing, while the second produces unreliable results because it ignores proven strategies and known pitfalls.

Methodology Mem2Evolve maintains two memory components — an Experience Memory that stores trajectories from past task executions, and an Asset Memory that stores dynamically created tools and expert agents. When a new task arises, the system retrieves relevant past experience to guide the creation of new tools or specialist agents, then stores the resulting execution trajectory back into experience memory, enabling a continuous co-evolutionary loop. The framework was evaluated across 6 task categories and 8 benchmarks.

Results Mem2Evolve achieves an 18.53% improvement over standard large language models, an 11.80% improvement over agents that evolve only through experience accumulation, and a 6.46% improvement over agents that evolve only through asset creation, demonstrating that the co-evolutionary approach is more effective and stable than either strategy in isolation.

Plain-language abstract PersonaVLM is an agent framework that turns a general-purpose multimodal AI assistant into a long-term personalized one. It gives the model three abilities it currently lacks: proactively building and updating a memory of the user across many sessions, reasoning over that memory to answer new questions, and continuously inferring the user's personality so replies stay in tune with who they actually are.

Motivation Current multimodal AI assistants treat every interaction as if it were the first — they have no mechanism to remember that a user's preferences have changed, or to learn personality traits that emerge gradually across many unrelated conversations. Prior personalization work addressed only static, single-session scenarios through input augmentation or output alignment, leaving the fundamental challenge of evolving preferences and personality unmet.

Methodology PersonaVLM introduces a structured memory architecture with a personality profile component (PEM) and four memory types — core, semantic, procedural, and episodic — that are proactively updated after each interaction. A two-stage process handles each turn: in the response stage the model performs multi-step reasoning with memory retrieval before replying; in the update stage new information is extracted and consolidated into the memory database. The authors also created Persona-MME, a new benchmark with over 2,000 curated interaction cases covering seven aspects and 14 fine-grained tasks, and tested PersonaVLM and more than 10 proprietary and open-source models on it.

Results Under a 128k-context setting PersonaVLM improved over the baseline by 22.4% on Persona-MME and 9.8% on the PERSONAMEM benchmark. It also outperformed GPT-4o by 5.2% on Persona-MME and 2.0% on PERSONAMEM, with further gains confirmed in open-ended evaluations.

Plain-language abstract MMSkills is a framework that gives AI agents reusable, visually grounded knowledge packages for operating graphical user interfaces and game environments. Each package pairs a written procedure with structured state cards that say when the procedure applies and what visual cues confirm it, plus multi-view screenshot keyframes. A temporary 'branch' process inspects the package evidence, aligns it with the live screen, and returns compact guidance to the main agent rather than flooding its context with raw screenshots.

Motivation AI agents that navigate GUIs and games must act on visual evidence, yet existing skill libraries store reusable knowledge only as text or code. A text-only skill can describe the correct steps but cannot tell the agent whether a dialog is ready, which sheet is active, or whether a goal has been visually confirmed. This gap between verbal instructions and visual state recognition causes agents to repeat actions, lose track of context, or fail tasks that require recognizing subtle on-screen conditions.

Methodology The authors define a multimodal skill package format containing a textual procedure, runtime state cards encoding when-to-use and when-not-to-use conditions with verification cues, and multi-view keyframes covering full-frame, focused, and before/after views. An automated trajectory-to-skill Generator processes public, non-evaluation interaction trajectories through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing to produce these packages without storing raw demonstrations. At inference time a two-stage branch-loading mechanism selects relevant state cards and keyframe views in a temporary branch, aligns them with the live observation, and returns structured planning guidance to the main agent. The framework was evaluated on OSWorld, macOSWorld, VAB-Minecraft, and Super Mario Bros in LMGameBench across six multimodal model families including Gemini 3 Flash, Qwen3-VL-235B, GLM-5V, Kimi-K2.6, and Qwen3-VL-8B-Instruct.

Results MMSkills consistently improve success rates over both no-skill and text-only baselines across all evaluated benchmarks and model families. On OSWorld, overall success rates rise substantially for every tested model: for example, Gemini 3 Flash improves from 36.65% (no skill) to 47.97% (MMSkills), and the smaller Qwen3-VL-8B-Instruct improves from 10.78% to 25.40%. On VAB-Minecraft, Qwen3-VL-8B-Instruct success rate rises from 23.28% to 38.79%. Ablations confirm that both state cards and visual keyframes contribute independently, and that branch loading outperforms inserting skill content directly into the main context.

Plain-language abstract This paper proposes a design framework for AI agents that support people's health over extended periods of time — across multiple conversations and weeks or months — rather than treating each interaction as a standalone event. The authors identify what such a longitudinal health agent needs to do, define four core properties it must embody, and illustrate the framework with representative use cases from clinical, preventive, and self-management settings.

Motivation Most current AI health agents respond to single queries and treat each interaction independently, which limits their usefulness for ongoing health challenges like chronic symptom management or behavior change that require follow-up, evolving goals, and accountability over time. No general framework existed for designing agents capable of maintaining coherent, goal-directed support across multiple sessions when health objectives remain unmet or change.

Methodology The authors conducted a perspective synthesis, drawing on approximately 40 prior studies spanning human-computer interaction, personal health informatics, and clinical care. Through iterative discussion within an interdisciplinary team of experts in HCI, AI, health informatics, and clinical care, they synthesized continuity-of-care and personal health informatics principles into a multi-layer framework, then demonstrated the framework's application through representative use cases.

Results The framework comprises four interdependent layers — coherence (stable memory, roles, and relationships across sessions), continuity (active stewardship of health goals through follow-up and accountability), adaptation (flexible personalization and reflexive reassessment as goals evolve), and agency (transparent negotiation of authority between user and system, including proactive intervention and user override). The authors show through use cases that applying these layers enables longitudinal agents to maintain meaningful engagement and support safe, personalized decision-making over time, and they identify open challenges including consistent longitudinal reasoning and dynamic alignment with clinical guidelines.

Plain-language abstract This paper introduces the Clinical World Model and Clinical AI Skill-Mix, a conceptual framework that gives clinical AI a formal account of the medical world in which it must operate. It formalizes clinical care as an interaction among patients, providers, and the broader care ecosystem, and organizes AI competency into eight dimensions that together define billions of distinct "competency coordinates" — each specifying a unique combination of clinical condition, care phase, setting, provider role, task, and how the AI engages human reasoning.

Motivation Clinical AI systems perform well on benchmarks and medical licensing exams but routinely fail when deployed in real settings: fewer than 6% of externally validated radiology models maintain their original performance, and large language models that pass licensing exams still falter under the uncertainty of authentic clinical reasoning. The field lacks a shared formal model of the clinical world that connects evaluation, regulation, and system design, leaving no principled way to specify where AI reliability has actually been demonstrated.

Methodology Rather than empirical experiments, the paper develops a theoretical framework. The authors synthesize established traditions in clinical AI — sociotechnical systems research, regulatory science, evaluation benchmarks, and agentic architectures — and identify three structural gaps none of them address alone. They then construct the Clinical World Model as a tripartite structure (Patient, Provider, Ecosystem) with ten views and thirteen dimensions, and develop parallel decision-making architectures for human and AI agents grounded in validated principles of clinical cognition. The Clinical AI Skill-Mix operationalizes competency through five dimensions defining the clinical space and three specifying how AI engages human reasoning.

Results The framework establishes that the combinatorial product of its eight competency dimensions yields a space of billions of distinct competency coordinates, and that validation within one coordinate provides minimal evidence of performance in another — making the competency space irreducible. This structural implication reframes the field's central question from whether AI works to in which competency coordinates reliability has been demonstrated, and for whom, supplying a common grammar for specifying, evaluating, and bounding clinical AI across stakeholders.

Plain-language abstract This paper introduces Pask, a proactive AI assistant that monitors ongoing user activity in real time and offers help without being asked. Rather than waiting for a user to type a question, Pask continuously senses context, infers what the user likely needs, and delivers assistance at the right moment. The system is built around a new model called IntentFlow for detecting user needs, a three-layer memory system for building up a persistent understanding of each person over time, and a full software infrastructure that ties everything together.

Motivation Most AI systems today operate in a purely reactive mode: a user asks a question and the system answers. This works poorly when users are busy, in meetings, or watching video and cannot stop to type a prompt. The paper argues that this creates both a usability gap and a fundamental barrier to deeper human-AI understanding. Existing research on proactive AI was limited to narrow scenarios and controlled settings, with no unified framework and no mechanism for the AI to accumulate knowledge about a user across many sessions.

Methodology The authors propose DD-MM-PAS, a three-component paradigm covering Demand Detection (DD), Memory Modeling (MM), and a Proactive Agent System (PAS). IntentFlow, built on top of the Qwen3-30B-A3B base model, is trained first with supervised fine-tuning on a 102,000-sample dataset of synthetic and real-world examples, then refined with reinforcement learning to improve intent alignment under streaming, low-latency conditions. The memory system uses three tiers: a User Memory cache for stable user traits and recent signals, a Workspace Memory implemented via the model context window for within-session continuity, and a Global Memory LLM-RAG store for long-term accumulation across sessions. Evaluation uses LatentNeeds-Bench, a new benchmark of 100 real sessions (3,936 annotated turns) spanning Work, Learning, and Daily domains, constructed from user-consented speech transcriptions and refined through thousands of rounds of human editing.

Results IntentFlow achieves an overall balanced accuracy of 83.3% on LatentNeeds-Bench, matching or exceeding leading commercial models under latency constraints. Most baseline language models struggle with demand detection: Gemini-2.5-Flash-Lite scores 18.8 on demand turns, Claude-Haiku-4.5 reaches 38.9, and even GPT-5-Mini only achieves 66.5, while Gemini-3-Flash is the strongest baseline at 83.3. A key finding is that targeted training substantially improves proactive capability beyond what general-purpose instruction-following achieves. In multi-turn analysis, IntentFlow maintains above 80% balanced accuracy across all conversation depth buckets, whereas the strongest baseline drops from 85.6% to 70.8% over long interactions, a decline of 17.3 percentage points.

Plain-language abstract This paper studies what happens when a single AI assistant must serve multiple users at the same time — each with different goals, levels of authority, and private information. The authors formalize this as a multi-principal decision problem, build a protocol for multi-user interaction, design three test scenarios, and run frontier language models through them to see where they break down.

Motivation Most language models are trained and evaluated assuming a single user whose instructions the model obeys without qualification. As these systems are embedded in team workflows and organizational tools, they must handle multiple simultaneous users who may have conflicting instructions, different authority levels, and information that should not be shared across users. No systematic framework or benchmark existed for this setting.

Methodology The authors formalized multi-user LLM interaction using a multi-principal decision framework borrowed from economics, then introduced a unified interaction protocol that explicitly encodes user identity, roles, and visibility constraints. They designed three stress-test scenarios: multi-user instruction following under conflicting objectives, cross-user access control and privacy preservation across multiple conversation rounds, and sequential multi-user coordination for meeting scheduling under partial information disclosure. They evaluated roughly twenty frontier LLMs — including GPT-5, Claude, Gemini, Grok, LLaMA, Qwen, and DeepSeek variants — across these scenarios.

Results Frontier models fail systematically on all three dimensions. Instruction-following performance degrades substantially when two users issue conflicting directives. Privacy scores drop significantly as interaction rounds increase, with most models showing measurable degradation by round ten, indicating that access control breaks down under sustained interaction. In the coordination task, models with higher success rates resolved meeting scheduling within fewer than four turns, while weaker models needed one to two additional rounds; full-information settings consistently outperformed partial-information settings, and Llama-3-70B exhibited a distinct failure of committing to incorrect answers prematurely rather than asking clarifying questions.

Plain-language abstract SemaClaw is an open-source multi-agent application framework designed to make personal AI agents more reliable, safe, and capable of learning across sessions. The paper introduces a set of engineering components — an orchestration method, a safety system, a context management architecture, and a knowledge-building tool — that together form the infrastructure layer between a capable AI model and a production-ready agent system.

Motivation When millions of users began deploying personal AI agents for tasks like travel planning and multi-step research, three system-level gaps became clear: existing frameworks lacked structured yet adaptive task orchestration, had no robust runtime safety enforcement for consequential actions like file modification or API calls, and provided only log-oriented memory that could not consolidate knowledge across sessions. These gaps define the difference between a capable model and a dependable agent, which the authors call the 'harness engineering' problem.

Methodology The authors designed and implemented SemaClaw as a two-layer architecture separating a reusable agent runtime from a configurable application harness. Key components include: DAG Teams, a two-stage hybrid orchestration method that combines LLM-based dynamic task decomposition with deterministic directed-acyclic-graph execution; PermissionBridge, a runtime safety system that treats authorization checkpoints as first-class control primitives for explicit user approval of high-risk actions; a three-tier context management architecture unifying working context, long-term memory retrieval, and per-agent persona partitioning; a four-layer plugin system covering MCP tools, subagents, skills, and hooks; a four-mode scheduled task system scaling token use to task complexity; and an agentic wiki skill for automated personal knowledge base construction that externalizes session-derived insights into a durable, user-owned format.

Results The paper presents SemaClaw as an open-source framework (released on GitHub at midea-ai/SemaClaw) with the stated contributions demonstrably instantiated in the system design. The work is primarily architectural and system-design in nature; the paper describes the framework's components and their rationale rather than reporting benchmark numbers or comparative experiments. The authors argue that the combination of DAG-based orchestration, runtime permission enforcement, structured context management, and knowledge sedimentation collectively addresses the production-reliability gap that prior open-source agent frameworks left open.

Plain-language abstract This paper is a survey of how memory systems in AI agents powered by large language models (LLMs) have developed over time. The authors propose a three-stage evolutionary framework — Storage, Reflection, and Experience — to organize and explain the progression of memory designs, and use it to identify the core technical drivers and open challenges in the field.

Motivation LLMs are stateless by nature, meaning they cannot retain information across interactions on their own. This makes it hard for LLM-based agents to stay consistent over long, multi-step tasks or to learn from past mistakes. Prior survey work on agent memory was fragmented, split between engineering-oriented and cognitive-science-oriented perspectives, with no unified framework explaining why and how memory mechanisms have evolved.

Methodology The authors conduct a systematic literature survey organized around a novel three-stage taxonomy they formally define: Storage (faithful recording of interaction trajectories), Reflection (dynamic evaluation and refinement of stored records), and Experience (cross-trajectory abstraction of generalized behavioral patterns). They analyze the field using a 'Why-How-What' structure addressing three research questions about the drivers, path, and outcomes of memory evolution, and they also catalog relevant benchmarks across each stage.

Results The survey maps existing memory mechanisms onto the three-stage framework and identifies the core catalysts driving evolution: the need for long-range consistency, the challenges of dynamic environments, and the goal of continual learning. It highlights that the frontier Experience stage — characterized by active exploration and cross-trajectory abstraction using techniques such as Minimum Description Length compression of trajectory clusters — addresses bottlenecks in agent adaptability. The authors also identify gaps in benchmark coverage, particularly for the Experience stage, and call out distributed shared memory and multimodal memory fusion as critical directions for future work.

Plain-language abstract StreamMemBench is a streaming benchmark that tests whether a personal-agent memory system can turn what it observes and how users interact with it into future-oriented assistance. Built on EgoLife egocentric lifelogs, it anchors each evaluation on a hidden piece of user-specific evidence and wraps a two-step task around it: an initial task that depends on the evidence, then a follow-up task that tests whether the agent reused both the evidence and the user's feedback. Four metrics diagnose evidence retention, feedback incorporation, initial evidence use, and follow-up reuse.

Motivation A central job of personal-agent memory is to carry stored observations and prior interactions forward into later, similar tasks, but existing memory benchmarks test dialogue recall or task improvement in isolation and usually rely on scripted or synthesized dialogues whose feedback is not tied to verifiable observations. That leaves the trajectory from streaming observations to later assistance largely untested — and even commercial assistants such as ChatGPT and Gemini store information that fails to help when it is actually needed.

Methodology In the construction stage an anchor agent processes five-minute EgoLife segments in stream order and extracts a user-specific evidence anchor plus two application-oriented queries, and a review agent retains a candidate only if both queries satisfy Leak=0 (the query does not reveal the evidence), Need=1 (ignoring the anchor yields a wrong or generic answer), and Natural=1 (it reads as a plausible request). In the evaluation stage the memory system ingests the lifelog chronologically, answers the initial task, receives confirming or correcting feedback from a user simulator, commits that interaction to memory, and then answers a follow-up task grounded in the same anchor; an evaluation agent scores Fidelity, Feedback Incorporation, Initial Evidence Use, and Follow-up Reuse against atom-level checklists, and the Fidelity−IEU / Fidelity−FUR gaps localize failures.

Results Across eight systems — two retrieval baselines (RAGraw, RAGext) and six memory systems (A-Mem, Mem0, EverMemOS, MemOS, MemoryOS, MemSkill) — on two backbones, current memory systems are not yet reliable for future-oriented assistance: they often fail to use evidence from egocentric observations in the initial task and to turn interaction feedback into reusable follow-up behavior. The failures are not explained by storage alone — systems frequently retain the evidence (with some Fidelity inflated by raw-text retention) yet do not use it — which motivates evaluation that traces each piece of evidence from its first appearance in the stream through initial use, feedback incorporation, and follow-up reuse.

Plain-language abstract This paper introduces SEA-Eval, the first benchmark designed to evaluate AI agents that improve themselves over repeated tasks. Most current benchmarks treat each task in isolation, resetting the agent's memory each time — which makes it impossible to measure whether an agent is genuinely learning from experience or just repeatedly solving problems from scratch. SEA-Eval is built around a new category of agent called a Self-Evolving Agent (SEA), which maintains persistent memory across tasks and uses past experience to get faster and cheaper over time.

Motivation Existing AI agent benchmarks evaluate each task as an isolated episode, resetting memory at every task boundary. This design structurally prevents them from detecting cross-task learning. Two agent frameworks can achieve identical success rates while differing dramatically in whether they are genuinely evolving — one accumulating efficient strategies, the other simply re-doing zero-shot reasoning every time. Current benchmarks produce a 'capability illusion' and cannot distinguish these fundamentally different behaviors.

Methodology The authors formalize the Self-Evolving Agent concept and its core architecture — the Evolutionary Flywheel, a closed loop of execution, experience distillation, and augmented re-execution. They construct a dataset of 32 atomic tasks across three difficulty tiers, composed into three types of sequential task streams (correlated, orthogonal, and implicit intent) under two noise conditions. Two primary metrics are used: success rate (SR) and token consumption (T), where the trajectory of T across sequential task repetitions is the key signal of genuine evolution. They then empirically evaluate two leading agent frameworks, OpenClaw and GenericAgent, using this benchmark.

Results Both frameworks achieved 100% task success rate in static evaluation, but their token consumption differed by roughly sevenfold in aggregate — OpenClaw consumed 27,364K tokens across all tasks while GenericAgent consumed only 3,918K tokens, with differences reaching up to 31.2x on individual tasks. In sequential task analysis, GenericAgent showed genuine evolution: token consumption for one academic database retrieval task fell monotonically from 520K to 117K tokens. OpenClaw showed pseudo-evolution: its token consumption fluctuated without converging and collapsed on certain sequential tasks. These results confirm that success rate alone is insufficient and that sequential convergence of token consumption is the necessary criterion for detecting real learning.

Plain-language abstract This paper introduces SCM (Sleep-Consolidated Memory), a memory architecture for large language models that mimics how human brains store and forget information. Rather than growing an ever-larger database of raw conversation text, SCM encodes conversations into structured concepts, selectively strengthens important ones during simulated sleep cycles, and actively prunes low-value memories — enabling persistent, organized memory that stays manageable over time.

Motivation Current LLM memory approaches all have critical gaps: context windows are bounded and degrade with long input, vector databases grow without limit and never forget, and tiered storage systems like MemGPT lack any offline consolidation or biological forgetting mechanisms. None replicate how human memory actually works — a dynamic system that consolidates during sleep, prioritizes by importance, and actively prunes weak associations. SCM was designed to fill this gap by bringing neuroscience-inspired memory management to conversational AI.

Methodology SCM is built from five modules: a MeaningEncoder (using a local Llama 3.2 2B quantized model) that converts text into typed semantic concepts with 384-dimensional embeddings; a ValueTagger that assigns four-dimensional importance scores (novelty, emotional valence, task relevance, repetition frequency); a WorkingMemory buffer capped at seven items; a LongTermMemory stored as a NetworkX semantic graph backed by SQLite; and a SleepCycle module that runs NREM consolidation (Hebbian strengthening plus synaptic downscaling), REM dreaming (novel association generation), and value-based forgetting. The prototype was evaluated on a standardized suite of eight benchmark tests covering memory retention, consolidation, forgetting, graph traversal, latency scaling, and multi-session persistence.

Results Across all eight benchmark tests the prototype achieved perfect scores (1.00). It maintained 100% recall accuracy over ten-turn conversations while reducing memory noise by 90.9% through adaptive forgetting — pruning 45 of 50 noise concepts while preserving all 5 important ones. Memory search latency remained below one millisecond with hundreds of stored concepts. An ablation study showed that disabling the ForgettingModule caused the largest memory bloat (72 stored concepts vs. 24 in full SCM) and that disabling the ValueTagger dropped recall to 81.8%, confirming that multi-dimensional importance tagging is the most critical component for selective retention.

Plain-language abstract This paper proposes 'cooperative paging', a method that helps AI chatbots handle very long conversations without losing track of earlier content. When a conversation grows too long to fit in the model's memory, old sections are replaced with short keyword summaries (about 8 tokens each), and the model is given a tool it can call to retrieve the full text of any summarized section when it needs it. The paper evaluates this approach on a standard benchmark of long real-world conversations and also runs a systematic study of design choices like page size and eviction strategy.

Motivation Large language models have a fixed context window, but multi-turn conversations can grow indefinitely. When old content must be discarded to make room, existing methods either force the model to guess what it is missing (which models do poorly), only work for narrow cases like file reads, or compress content irreversibly losing detail. There was no systematic study of how page boundary detection and eviction policy affect retrieval quality in this setting.

Methodology The authors built a cooperative paging system in which evicted conversation segments are replaced with minimal keyword bookmarks (e.g., '[p3: allergy, peanut, budget]') and the model is given a recall() tool to fetch full content on demand. They evaluated it on the LoCoMo benchmark (10 real multi-session conversations with 300+ turns each) across four models from three provider families (GPT-4o-mini, DeepSeek-v3.2, Claude Haiku, GLM5), comparing against five baselines including truncation, BM25 retrieval, and a search-tool baseline. They also ran a turn-by-turn paging simulator with 3,176 probes on synthetic data and 1,600 probes on LoCoMo to ablate five page-boundary strategies and four eviction policies (FIFO, LRU, LFU, Belady oracle), and tested six bookmark generation strategies.

Results Cooperative paging achieved the highest answer quality among six methods on LoCoMo across all four tested models, with statistical significance (p=0.017 vs. BM25, paired bootstrap). The ablation showed page granularity dominates eviction policy: coarse fixed-size pages (fixed-20) reached 96.7% recall accuracy while content-aware topic-shift paging collapsed to 56.7%. The model triggers recall() correctly 96% of the time, but selects the right page only 57% of the time when bookmarks are insufficiently distinctive, shifting the bottleneck from 'when to recall' to 'which page to recall'. The best bookmark strategy (llm-batch, single-call cross-page non-overlap keywords) improved end-to-end accuracy by 8.7 points on LoCoMo and 50 points on open-domain questions over the heuristic baseline; keyword specificity alone accounted for a 25 percentage-point accuracy difference in controlled probes.

Plain-language abstract Memanto is a memory system for AI agents that need to remember information across many conversations and tasks. Instead of the complex graph-based memory architectures most agent frameworks use, Memanto organizes memory into thirteen typed categories and retrieves relevant memories using a single, fast semantic search query — achieving top benchmark scores while being simpler and cheaper to run.

Motivation As AI agents move from answering single questions to carrying out long, multi-step tasks across many sessions, they need reliable persistent memory. Existing production memory systems — such as Mem0, Zep, and A-MEM — combine knowledge graphs with vector databases, which imposes heavy computational costs: every memory write triggers multi-second pipelines involving LLM-driven entity extraction and graph synchronization. The paper argues this 'Memory Tax' is unnecessary and that simpler architectures can match or beat graph-based systems in accuracy.

Methodology Memanto is built on Moorcheh's Information-Theoretic Search engine, a no-indexing semantic database that provides deterministic (exact-match rather than approximate nearest-neighbor) retrieval with sub-90-millisecond latency and zero ingestion delay. The memory layer uses a typed schema of thirteen predefined semantic categories, an automated conflict-resolution mechanism for contradictory memories, and temporal versioning. The system was evaluated on two established long-term memory benchmarks — LongMemEval and LoCoMo — using a five-stage progressive ablation study that isolated the contribution of retrieval-limit tuning, similarity-threshold calibration, prompt design, inference model selection, and the typed schema.

Results Memanto achieved accuracy scores of 89.8% on LongMemEval and 87.1% on LoCoMo, establishing state-of-the-art results among both vector-based and hybrid graph-plus-vector systems. These results were obtained using only a single retrieval query per lookup, with no ingestion cost and no graph infrastructure — outperforming all evaluated hybrid architectures while requiring substantially lower operational complexity.

Plain-language abstract ZenBrain is a memory architecture for AI agents that draws on 130 years of cognitive neuroscience research to give language-model agents a structured, biologically grounded way to store, consolidate, and forget information across many sessions. It organizes memory into seven distinct layers — working, short-term, episodic, semantic, procedural, core, and cross-context — and coordinates them with fifteen neuroscience-inspired algorithms. The system is open-source and ships as composable npm packages with over 11,500 automated tests.

Motivation Current AI agent memory systems borrow metaphors from computer science — virtual memory paging, flat key-value stores, or structured note-taking — but none incorporate the well-validated principles of memory consolidation, forgetting, and reconsolidation studied in cognitive neuroscience for over a century. Without principled decay and consolidation, agents suffer 'conversational amnesia' and cannot maintain consistent personality or learning across sessions. A 2026 survey explicitly identified deeper neuroscience integration as a key open challenge in the field.

Methodology ZenBrain implements seven memory layers orchestrated by nine foundational algorithms plus six new Predictive Memory Architecture (PMA) components, including a four-channel NeuromodulatorEngine modeling dopamine, norepinephrine, serotonin, and acetylcholine dynamics; a prediction-error-gated ReconsolidationEngine; TripleCopyMemory with divergent decay dynamics; and a four-dimensional PriorityMap with an amygdala fast-path. The system was evaluated across ten experiments covering memory lifecycle management, retrieval benchmarks, and system-level ablation studies using three established benchmarks — LoCoMo, MemoryAgentBench, and MemoryArena — under a 15-algorithm ablation protocol with Wilcoxon tests over 10 seeds.

Results Under challenging conditions (decay=0.20, 50 days), 7 of 15 algorithms became individually significant with quality drops ranging from -25.5% to -93.1%; under stress conditions (decay=0.25, 60 days), 9 became critical (up to -93.7% degradation). The Simulation-Selection sleep loop achieved a 37% stability improvement (p < 0.005) with 47.4% storage reduction. TripleCopyMemory retained 0.912 mean memory strength at 30 days versus near-zero Ebbinghaus baselines, and the PriorityMap achieved NDCG@10 = 0.997 versus 0.680 for chronological ordering. On the LongMemEval-500 benchmark, ZenBrain held the highest mean rank across all 12 judge-system quality cells, with a three-judge mean of 0.545 against competitors letta (0.485), a-mem (0.414), and mem0 (0.394), reaching 91.3% of long-context-oracle accuracy at 1/106th the per-query token budget.

Plain-language abstract This paper introduces STALE, a benchmark that tests whether AI language model assistants can recognize when something they "remember" about a user is no longer true. The authors identify a specific failure mode called an implicit conflict, where a new piece of information makes an old memory invalid without directly contradicting it, and show that current AI systems handle this poorly.

Motivation Existing benchmarks for AI memory mostly test whether a model can retrieve a stored fact, not whether it can recognize when that fact has been superseded. Real assistant interactions unfold over time and a user's circumstances change; a model that confidently acts on stale beliefs can give harmful or irrelevant advice. No prior benchmark systematically evaluated this implicit-conflict failure mode, especially cases where one life change cascades to invalidate a structurally related but unstated belief.

Methodology The authors built STALE, a benchmark of 400 expert-validated conflict scenarios producing 1,200 evaluation queries across over 100 everyday topics, with dialogue contexts up to 150,000 tokens. Scenarios cover two conflict types: co-referential (the same attribute is updated) and propagated (a change in one attribute cascades to invalidate a different one). Each scenario is probed along three dimensions: State Resolution (does the model detect the outdated belief?), Premise Resistance (does it reject a query that presupposes the stale state?), and Implicit Policy Adaptation (does it act correctly without being told there was a conflict?). They evaluated frontier closed-source LLMs, open-source LLMs, and five memory-augmented frameworks, and proposed a prototype system called CUPMEM that performs explicit write-side state adjudication and propagation-aware memory search.

Results Even the best-performing evaluated model, Gemini-3.1-pro, achieved only 55.2% overall accuracy. Models frequently accepted outdated assumptions embedded in user queries and struggled especially with propagated (Type II) conflicts, where Premise Resistance scores were near zero across nearly all systems. Memory-augmented frameworks did not solve the problem: the strongest baseline (LightMem) scored 17.8% overall, and diagnostic analysis showed that updated evidence was retrieved 77.5% of the time yet still failed to govern the final answer, a gap the authors call the current-state adjudication gap. The prototype CUPMEM substantially outperformed all baselines, reaching 91% on State Resolution and 78% on Premise Resistance for Type I conflicts, demonstrating that explicit write-side state adjudication is a viable design direction.

Plain-language abstract This paper introduces Adaptive Memory Crystallization (AMC), a new framework for helping AI agents learn continuously across changing tasks without forgetting what they already know. It models each stored experience as being in one of three memory states — liquid (flexible), glass (transitional), or crystal (stable) — and uses a mathematically rigorous process to decide which memories to keep and which to let go.

Motivation AI agents deployed in open-ended settings like robotics or autonomous driving must keep learning new skills while retaining old ones, a challenge known as the stability-plasticity dilemma. Existing deep reinforcement learning agents use fixed-size experience-replay buffers that, when tasks shift, cause catastrophic forgetting as new learning overwrites prior knowledge. No principled framework had linked the dynamics of memory consolidation to formal guarantees on agent performance.

Methodology AMC models memory consolidation as a continuous stochastic process governed by an Itô stochastic differential equation (SDE) in which each experience's crystallization state evolves based on a multi-objective utility signal combining temporal-difference error, state-action novelty, and downstream value. The population-level behavior is captured by a Fokker-Planck equation with a closed-form Beta stationary distribution. The framework organizes memory into three buffers (Liquid, Glass, Crystal) with capacity fractions and phase-modulated learning rates derived from the SDE dynamics. Formal proofs establish well-posedness, convergence, Q-learning error bounds, and memory-capacity lower bounds. The system was evaluated on Meta-World MT50, Atari 20-game sequential learning, and MuJoCo continual locomotion benchmarks using 50-seed statistics with Holm-Bonferroni-corrected significance tests.

Results AMC consistently outperformed the strongest baselines across all three benchmark suites, achieving 34-43% improvements in forward transfer, 67-80% reductions in catastrophic forgetting, and a 62% decrease in memory footprint.

Plain-language abstract This paper presents H2-EMV, a system that gives humanoid robots a manageable long-term memory of their own experiences so they can answer questions like 'Where did you put my keys?' or 'Why did the task fail?' The system learns over time which memories matter most to a particular user, selectively discards less relevant ones, and improves its recall accuracy based on user feedback.

Motivation Robots that operate continuously accumulate vast streams of sensory and event data that quickly exceed storage and compute limits, making real-time question answering over past experiences impractical. Existing episodic memory systems for robots either grow without bound or use fixed relevance criteria, neither of which adapts to what individual users actually care about remembering.

Methodology The authors build on a prior hierarchical tree-based episodic memory representation, extending it with three additions: online incremental construction of the memory tree as new events arrive, a decay-based forgetting mechanism in which each memory node is assigned a lifetime and an LLM judges its relevance for retention when it expires, and feedback-driven learning of natural-language relevance rules that update when a user asks about something that was already forgotten. The system is evaluated on simulated household task recordings from the TEACh dataset and on 20.5 hours of real-world recordings from the humanoid robot Armar-7.

Results H2-EMV reduces memory size by 45% and query-time compute by 35% compared to a non-forgetting baseline while maintaining question-answering accuracy. Performance improves substantially across interaction rounds: accuracy in second-round queries increases by 70% relative to the first round, showing that the system successfully adapts to user-specific priorities after only one or a few feedback instances.

Plain-language abstract Evolve is a system that pairs a small (2–7 billion parameter) local language model with an external knowledge store that persists, grows, and improves over time. Rather than fetching raw document chunks at query time, Evolve builds a store of semantically coherent knowledge sections compiled by larger teacher models; those sections are staged, consolidated offline, and refreshed when they become stale. The result is a small model that answers questions far more accurately than it could from its own training weights alone.

Motivation Large language models bake factual knowledge into their parameters during training, making that knowledge frozen, impossible to selectively update, and prone to hallucination. Standard retrieval-augmented generation (RAG) partly addresses this but still suffers from chunk-boundary mismatch, static corpora that fail on out-of-distribution queries, retrieval noise, and an opaque entanglement between what the model knows internally and what was retrieved—making the knowledge base impossible to audit or update independently of the model weights.

Methodology Evolve externalizes all factual knowledge into a persistent store of semantically bounded sections compiled by large teacher models at natural conceptual boundaries. Newly acquired sections enter a staging buffer, are consolidated offline via teacher-mediated merging and pruning (analogous to biological sleep consolidation), and are refreshed inline when they expire. A 2B-parameter local model handles classification and retrieval-augmented generation; large teacher models are invoked only for knowledge operations. The system supports two generation modes—suppress (strict grounding on retrieved sections only, for auditability) and augment (sections plus parametric fallback). Evaluation used 750 benchmark queries spanning custom specialist questions, NaturalQuestions, and TriviaQA.

Results Augmenting the 2B model with Evolve raised accuracy from a 20–33% parametric baseline to 60–84%, a gain of 40–52 percentage points across benchmarks. Teacher model invocations were reduced by more than 50% through cross-query knowledge reuse. Post-consolidation compressed the knowledge store by 31–33.5% across three independent benchmarks while preserving accuracy. Section-based retrieval outperformed conventional chunk-based retrieval by 5–9 percentage points across every lifecycle condition tested.

Plain-language abstract This paper introduces RoMem, a plug-in module that gives AI agents better long-term memory by treating time as a geometric property rather than just a timestamp. Instead of erasing old facts or asking a language model to decide what to keep, RoMem rotates outdated facts out of alignment in a mathematical space so that current information naturally rises to the top without deleting history.

Motivation Knowledge graph memory systems used by AI agents face a fundamental problem when facts change over time: they either overwrite old information and lose historical context, or rely on expensive language-model calls at every update to decide what to discard. A simpler recency-sorting fallback silently buries permanently true facts (such as a person's birthplace) beneath newer but irrelevant entries, creating what the authors call the static-dynamic dilemma.

Methodology RoMem models time as a continuous phase rotation in complex vector space rather than a discrete timestamp. A pretrained Semantic Speed Gate reads each relation's text embedding and outputs a per-relation volatility scalar between 0 and 1, learned from data so that evolving relations rotate fast while persistent ones remain stable. As time passes, dynamic facts rotate out of geometric alignment with the query vector, causing temporally correct facts to outrank contradictions without any deletion. The memory is strictly append-only, and the language model receives a clean context window ranked by geometric proximity.

Results On the temporal knowledge graph completion benchmark ICEWS05-15, RoMem achieves 72.6 MRR, a state-of-the-art result. Applied to agentic memory, it delivers 2 to 3 times higher MRR and answer accuracy on heavy temporal reasoning (MultiTQ), dominates a hybrid reasoning benchmark (LoCoMo), preserves static memory with zero degradation (DMR-MSC), and generalizes zero-shot to unseen financial domains (FinTMMBench).

Plain-language abstract This paper introduces A-MEM, a memory system for AI agents powered by large language models that can dynamically organize and update its own knowledge without needing predefined rules or structures. Instead of storing memories in a fixed schema, A-MEM creates richly annotated notes for each new experience and automatically links them to related past memories, allowing the memory network to evolve over time.

Motivation Existing memory systems for LLM agents require developers to hardcode when and how memories are stored and retrieved, which limits how well they adapt to new tasks or long-running interactions. Even systems that use graph databases rely on rigid, predefined schemas that cannot forge new connections as an agent's knowledge grows, leaving a gap for a flexible, self-organizing memory architecture.

Methodology A-MEM represents each memory as a structured note containing the original interaction content, a timestamp, LLM-generated keywords, tags, a contextual description, a dense embedding vector, and a set of links to related memories. When a new memory is added, the system uses an LLM to generate semantic attributes for it, retrieves the top-k most similar historical memories via cosine similarity, and autonomously decides whether to create links and whether to update the contextual representations of existing memories. The system was evaluated on the LoCoMo long-term conversational dataset using six foundation models (including GPT-4o, GPT-4o-mini, Llama 3.2 1B/3B, and Qwen2.5 1.5B) across five question-answering categories (single-hop, multi-hop, temporal, open-domain, and adversarial), with F1 and BLEU-1 as metrics and four competitive baselines for comparison.

Results A-MEM consistently achieved the best average ranking across all six foundation models tested, outperforming baselines including LoCoMo, ReadAgent, MemoryBank, and MemGPT on the LoCoMo dataset. For example, with Llama 3.2 3B, A-MEM reached an F1 of 45.85 on multi-hop questions compared to the next-best baseline score of 25.52, and achieved top-1 average ranking across most model configurations while using substantially fewer tokens than full-context approaches. T-SNE visualizations further showed that A-MEM produces more coherent, clustered memory embeddings compared to a baseline without link generation or memory evolution.

Plain-language abstract EvolveMem is a memory system for AI agents that can improve its own retrieval strategy over time, without human intervention. Most existing memory systems update what they remember but keep the rules for finding relevant memories fixed forever. EvolveMem breaks that limitation by treating the retrieval configuration itself as something that can be automatically researched and improved.

Motivation Long-running AI agents need persistent memory across sessions, but all existing systems freeze their retrieval infrastructure at deployment — the scoring functions, fusion strategies, and answer-generation policies never change even as the stored memory grows from dozens to hundreds of heterogeneous records. Different types of questions require different retrieval strategies, and a fixed configuration cannot optimally serve factual lookups, temporal reasoning, and multi-hop inference simultaneously.

Methodology EvolveMem runs a four-step closed-loop evolution cycle: Evaluate, Diagnose, Propose, and Guard. In each round, an LLM-powered diagnosis module reads per-question failure logs, identifies root causes, and proposes targeted adjustments to a structured action space covering scoring weights, fusion modes, context budgets, and answer-generation style. A guarded meta-analyzer applies accepted changes with automatic revert-on-regression and explore-on-stagnation safeguards. The underlying memory store combines lexical (BM25), semantic, and structured-metadata retrieval views, and the system was evaluated on the LoCoMo and MemBench benchmarks using GPT-4o as the backbone.

Results On LoCoMo, EvolveMem outperforms the strongest published baseline by 25.7% relative and improves 78.0% relative over the minimal baseline (F1 rising from 30.5% to 54.3% across seven evolution rounds). On MemBench it exceeds the strongest baseline by 18.9% relative. Configurations evolved on LoCoMo transfer zero-shot to MemBench and, when continued there, reach 79.2% F1 — outperforming a natively evolved MemBench configuration by 16.6% relative, with positive rather than catastrophic transfer. Three new configuration dimensions (entity-swap retrieval, query decomposition, and answer verification) were discovered autonomously by the diagnosis module from failure logs, not pre-specified by the researchers.

Plain-language abstract This paper is a survey that maps out how to give AI language model agents the ability to keep learning over time — accumulating new knowledge without forgetting what they already know. It organizes the key building blocks of such agents into three parts: how they take in information from the world, how they store and retrieve what they have learned, and how they act on that knowledge in changing environments.

Motivation Current large language model agents are static after training: their knowledge is frozen, and they cannot adapt to new tasks or environments without being retrained from scratch. This is a critical gap because real-world applications — from household robots to interactive assistants — require agents that can continuously encounter new situations, integrate new knowledge, and still remember prior experience. The core technical obstacle is the stability-plasticity dilemma: systems either forget old knowledge when learning something new (catastrophic forgetting) or become too rigid to learn at all (loss of plasticity).

Methodology The authors conducted a systematic literature survey, organizing techniques for lifelong learning in LLM-based agents around three modules: a perception module for integrating multimodal inputs (text, images, sensory data), a memory module for storing and retrieving evolving knowledge, and an action module for grounded interactions with dynamic environments. The survey is structured around seven research questions covering architectures, forgetting mitigation strategies, evaluation benchmarks, real-world application scenarios, and open challenges. Relevant literature is collected and made available through an accompanying GitHub repository.

Results The survey is the first to systematically cover the intersection of lifelong learning and LLM-based agents. It provides a roadmap identifying how perception, memory, and action components collectively enable continuous adaptation, outlines evaluation metrics and benchmarks for assessing lifelong performance, catalogs real-world application domains including gaming, web browsing, household tasks, and autonomous robotics, and highlights key open problems and future research directions in this rapidly growing field.

Plain-language abstract This paper addresses how AI agents trained with deep reinforcement learning can learn new tasks without forgetting old ones. The authors identify individual neurons in the neural network that encode task-specific skills, and use this insight to build a method that selectively protects those neurons during learning. The approach, called NBSP (Neuron-level Balance between Stability and Plasticity), is tested on standard robotics and game-playing benchmarks.

Motivation Deep reinforcement learning agents struggle when they need to learn a sequence of tasks: improving at a new task tends to overwrite knowledge needed for prior tasks, a problem known as catastrophic forgetting. Existing methods that try to balance stability (retaining old skills) and plasticity (acquiring new ones) operate at the network level without fine-grained control over which neurons matter most, leaving the dilemma largely unresolved.

Methodology The authors first observed that after training, specific neurons in the network show activation patterns that reliably predict task success, identifying them as carriers of task-specific skill knowledge. NBSP builds on this by defining and detecting these 'RL skill neurons' through a goal-oriented criterion, then applying gradient masking to protect those neurons from being overwritten and using experience replay to reinforce existing skills while the agent adapts to new tasks. Experiments were conducted on the Meta-World robotic manipulation benchmark and Atari games, using a cycling task setup to evaluate both retention of old skills and acquisition of new ones.

Results NBSP significantly outperforms existing approaches at balancing stability and plasticity on the Meta-World and Atari benchmarks. The paper also introduces the concept of RL skill neurons as a contribution, providing a principled, neuron-level mechanism for continual learning in deep reinforcement learning agents.

Plain-language abstract NeuSymMS is a memory system for AI assistants that lets them remember and reason about users across multiple conversations. It combines a neural language model for extracting facts from dialogue with a rule-based expert system (CLIPS) that organizes, deduplicates, and updates those facts automatically. The result is a compact, auditable memory layer that avoids flooding the AI with raw chat history while still keeping track of what it knows about each user.

Motivation Most deployed AI assistants treat conversation history as temporary context that is discarded after each session, or they naively accumulate ever-growing logs. Existing approaches such as full-log retrieval, periodic summarization, and simple key-value stores handle contradictions poorly and cannot gracefully update beliefs when a user's job, location, or preferences change. There was no practical, production-ready system that combined the language understanding of neural models with the deterministic consistency rules needed for trustworthy, auditable long-term memory.

Methodology NeuSymMS sits between user conversations and the language model. On the write path after each turn, a lightweight LLM (such as gpt-4.1-mini) extracts candidate facts as subject-relation-value triples from the dialogue at low temperature. These candidates are then passed to a CLIPS-based expert system that classifies them into nine semantic categories, detects contradictions with stored facts, retracts stale beliefs, and decides whether each fact belongs in short-term or long-term memory based on explicit lifecycle rules and access counts. Facts are stored in a PostgreSQL database with three-tier scoping (user, agent, flow) to prevent cross-entity contamination. On the read path, relevant user-scoped facts are formatted into a concise text block and injected into the LLM's system prompt. The system is implemented using Django, CLIPSPy (Python bindings for CLIPS 6.4), and LiteLLM as a provider-agnostic gateway.

Results The paper presents a working architecture deployed inside the Nexa platform, with a REST API and a user-facing Memory UI for viewing, editing, and clearing stored facts. The design achieves zero-cost, low-latency classification by running the CLIPS engine in-process with millisecond overhead and no external dependencies, and avoids expensive vector search at inference time by injecting directly formatted text. The system degrades gracefully if any component fails—agents continue without memory rather than crashing. A full quantitative evaluation against benchmarks such as LoCoMo and LongMemEval and baselines such as Mem0 and MemGPT is planned as future work; no benchmark numbers are reported in the current paper.

Plain-language abstract OSL-MR is a framework for deciding which memories a long-horizon language agent should keep when its storage budget is smaller than everything it has seen. It poses the choice as an optimization problem that weighs the future value of evidence against the cost of losing it, and learns the policy from logged interactions using only signals that would actually be available when the agent runs.

Motivation Long-horizon agents accumulate more observations, reasoning traces, and retrieved facts than fit in context, so retention is a resource-allocation problem. Prior systems score memories with heuristics, retrieval objectives, or learned compression, but treat each retention decision locally and never model its long-term consequences — and many depend on signals such as gold evidence or answer correctness that are only knowable after the fact and so would not exist at deployment.

Methodology Retention is cast as a constrained multi-step stochastic optimization: at each step the agent selects a subset of memories under a hard size budget, and a per-step reward credits covered evidence while charging storage, miss penalty, reacquisition delay, and stale-information use. A strict online/offline split separates online-observable inputs (query, memory metadata, interaction history) from offline-available supervision (gold evidence, answer text, freshness) used only for training. OSL-MR pairs a Mixed-Score heuristic — a deployable cold-start baseline and inductive prior — with an evidence learner trained offline on gold-evidence membership labels, then frozen and deployed on online features alone.

Results On LoCoMo and LongMemEval, OSL-MR beats recency, Generative-Agents-style scoring, behavior cloning, and the Mixed-Score heuristic across budgets, with the largest gains under tight budgets (LoCoMo budget 128: F1 0.302 and reward 305 vs 0.069/132 for Mixed-Score). It uses the budget more efficiently, running well below full occupancy while baselines saturate it, and reward rankings track precision and F1 exactly. Ablating the Mixed-Score prior cuts precision (0.529→0.421 on LongMemEval budget 256) while leaving recall nearly unchanged, showing the prior steers the learner away from low-utility memories.

Plain-language abstract TokenPilot is a context-management framework for LLM agents in long-horizon sessions that reconciles cutting token count with preserving backend KV prompt-cache continuity. The authors observe that prior pruning, compaction, and memory-eviction methods mutate the prompt layout, which shatters prefix continuity and triggers cache-miss pre-fill penalties that override the financial savings from text reduction. TokenPilot operates at two granularities: a global Ingestion-Aware Compaction harness that stabilizes the prompt prefix and strips open-world tool-output noise at the ingestion gate, plus a local Lifecycle-Aware Eviction that defers purging context segments until their residual task utility expires, on a conservative batch-turn schedule.

Motivation Continuous multi-turn agent interactions accumulate verbose execution traces that inflate sequence length and per-turn inference cost. Existing content-reduction methods reduce tokens but constantly mutate input boundaries, causing prefix mismatches and KV cache invalidation; the resulting pre-fill penalties can exceed the text-reduction savings. The core insight is a trade-off between text sparsity and prompt-cache continuity that must be reconciled, by safeguarding physical prefix continuity during observation ingestion and deferring structural eviction until residual utility expires.

Methodology Messages are partitioned into internal intentional messages (high utility) and open-world environmental feedback (lower density unless content-hash access frequency exceeds a threshold). A canonicalization operator enforces a byte-identical prefix across turns; environmental messages are reduced to structural previews with full payloads stored in a content-hash-indexed artifact registry and recallable via a recovery tool. Lifecycle-Aware Eviction tracks segments through active/completed/evictable states using a zero-shot estimator run every B=3 turns over a compressed history view; only evictable segments (zero residual utility) are purged in a single pass. Evaluation is on PinchBench and Claw-Eval in isolated and continuous modes with GPT-5.4-mini as backbone, against compression baselines (LLMLingua-2, SelectiveContext, Keep-Last-N) and paging/summarization baselines (MemoBrain, MemOS, others); cache hit/miss token counts are read directly from provider API metadata.

Results TokenPilot achieves the lowest inference cost while maintaining competitive accuracy: isolated mode $3.22 on PinchBench and $2.27 on Claw-Eval (61% and 56% reductions vs Vanilla); continuous mode score 81.3 at $2.79 on PinchBench and $10.58 vs Vanilla's $81.52 on Claw-Eval (61% and 87% reductions). Ablation on continuous PinchBench: Ingestion-Aware Compaction cuts cost $7.24 -> $4.22 (cache-miss tokens 5.943M -> 1.589M); adding Lifecycle-Aware Eviction reaches $2.79 (cache-read tokens 26.716M -> 8.551M, a 65% drop). Prefix stabilization raises macro cache hit rate from 38.7% to 79.2% on PinchBench and 67.2% to 83.1% on Claw-Eval. Removing the recovery tool drops accuracy 80.9 -> 77.1; B=1 eviction is too aggressive, B=infinity bloats context, B=3 is the chosen balance.

Plain-language abstract Long-running language agents accumulate more history than fits in working context, and the usual fix is retrieval — store past events outside the model and fetch a relevant subset at query time. This paper argues retrieval only answers what can be fetched (memory access) and not what should keep shaping behavior after the working context is unloaded (memory depth). It introduces the loop-drift protocol, a controlled stress test where the retrieval index stays intact but working context is cleared, so goal-conditioned behavior must persist through long-loop interference without the relevant text being reinserted. It evaluates EVAF, a surprise- and valence-gated LoRA consolidation mechanism that writes only behavior-relevant events into a small adapter. Across GPT-2, TinyLlama, and Mistral-7B, retrieval wins shallow factual recall while EVAF wins goal persistence and post-unload recovery with only 2–3 parametric writes per 200 events, and the paper shows selective consolidation factorizes into two separable controls — selection and actuation.

Motivation Memory access and memory depth are different problems. A shallow memory is one the system can retrieve or attend to; a deep memory changes future behavior — it persists through interference, survives context unload, and affects choices without being reinserted as text. Retrieval is indispensable for fetched facts, but a long-running assistant also needs durable goals, preferences, and constraints that are not merely fetched facts. Existing long-memory benchmarks (LongMemEval, LoCoMo) emphasize conversational recall, temporal access, and knowledge updates, and do not isolate the post-unload setting where retrieval remains available but behavior must continue without the relevant text in context. The paper's claim is narrow and explicit: memory depth can be probed by post-unload goal-conditioned behavior, and consolidation factorizes into selection and actuation — it does not claim universal memory accuracy, SOTA retrieval, or complete deletion/update validity.

Methodology Loop-drift protocol: synthetic per-user streams of 200 events (10 users/run) mixing stable goal/preference reminders, off-topic distractors, transient opposite requests, conflicts, sibling-user contamination, and scheduled factual notes; four probe layers — shallow episodic (recent fact), noisy episodic (old fact after same-key interference), parametric tendency (does a stable goal still shape behavior after long interference), and post-unload recovery (re-probe the goal immediately after a context unload, with the retrieval index intact but working context cleared). The RAG baseline stores all events in a durable embedding index (top-3 cosine) that context unload does not clear, so any EVAF goal-layer advantage is not a trivial 'RAG forgot' artifact. EVAF mechanism: per-event surprise (token negative log-likelihood) and valence (embedding similarity to the user's durable goal/preferences) combine into an admission gate; events above threshold enter a buffer, and when the buffer fills a LoRA adapter is updated on the buffer plus replay from prior consolidated events, with an L2 anchor as a drift guard. Model controls: GPT-2 and TinyLlama (four-seed means) plus Mistral-7B. Selection is isolated with a matched-random gate (same write count and online write dynamics, random admitted events). Actuation is isolated with fixed-inner controllers (fixed-1/2/3 inner LoRA steps using the same gate). A routed EVAF+RAG variant routes factual probes to retrieval and goal probes to EVAF. Public Memora event streams serve as an external boundary diagnostic for stale-memory invalidation, tested with McNemar's test.

Results Depth flip: RAG is strongest on recent explicit facts (short-fact accuracy 0.956–0.973) and near-useless on goals; EVAF is near chance on short facts but much stronger on the goal layer — on GPT-2 EVAF reaches 0.904 goal / 0.900 post-unload vs RAG 0.398/0.394, and on TinyLlama 0.833/0.812 vs RAG 0.396/0.394 — at only 2.4–2.6 writes (L2 drift ~21–29) vs RAG's 0 writes. Writing everything is not enough: Naive-LoRA writes all 200 events at far higher drift (~67 TinyLlama, ~119 GPT-2) and still fails the goal layer; at 7B indiscriminate writing is actively harmful — Naive-LoRA goal persistence collapses to 0.333±0.047, below the 0.500 chance baseline. Selection is not sparsity: on GPT-2 EVAF beats a matched-random gate of equal write count on goal and post-unload in all four seeds (mean 0.790/0.763 vs 0.590/0.619); TinyLlama is weak/mixed, so the selection signal is not monotonic in model scale. Actuation is a separable, model-dependent factor: fixed-inner audits show smaller inner steps cut drift and improve goal/post (Mistral-7B five-step 0.354/0.306 -> Fixed-2 0.796/0.775 -> Fixed-1 0.919/0.938), but Fixed-1 contamination saturates at 1.000 on Mistral, so high actuation trades selectivity for goal strength. Asymmetric coupling: under a miscalibrated five-step actuation at 7B the matched-gate comparison reverses, yet EVAF still keeps lowest sibling contamination (0.787±0.041) — selection stays semantically active while its translation into goal behavior fails. Boundary: on Memora, EVAF improves forgetting-absence only 91/222 to 95/222 (p=0.57, not significant), so append-only selective consolidation does not solve stale-memory delete/update validity, which the paper leaves to validity-gating or reconsolidation.

Plain-language abstract MemStrata is a memory system for AI agents that keeps track of when facts become outdated. Instead of just retrieving whatever text looks most similar to a query, which fails when an old and a new fact look nearly identical, it uses a deterministic rule to detect when a new fact supersedes an old one and retires the stale version.

Motivation Retrieval-augmented memory has no concept of time: when a fact changes (a renamed function, an updated config value, a new port number), both the old and new versions sit in the store with near-identical embeddings, and the agent can't tell which is current. The authors show this isn't a tuning problem: on a calibrated dataset, cosine similarity separates contradictions from duplicates at only 0.59 AUROC (near chance), because a value-flip edit sits textually closer to the original than a genuine rephrasing does.

Methodology MemStrata's write path first tries a deterministic (subject, relation, object) triple match: if an incoming fact shares a key with a stored one but asserts a different value, the old fact is retired (not deleted) in a bi-temporal ledger and the new one is stored as current. Non-triple prose falls back to a similarity-plus-LLM-judge gate. The system is evaluated on six local, deterministic benchmarks (two static, four marker-free evolving: code mutation, config migration, dependency bumps, API evolution) with a 7B model on consumer hardware.

Results MemStrata matches RAG on static recall (no cost) and reaches 0.95-1.00 accuracy on evolving-knowledge benchmarks where RAG reaches only 0.20-0.47. When forced to answer, plain RAG serves the superseded value 15-40% of the time; MemStrata drives this to ~0%. It also runs at ~2.1s retrieval latency versus ~16-18s for LLM-reranking/verification baselines, since no LLM sits on the read path.

Plain-language abstract A benchmark and method for a memory capability separate from recall: keeping track of which version of a fact is currently in force after it has been revised across sessions. The benchmark, StateMemBench, generates each scenario as a symbolic program of state operations so the correct answer is computed by replay and the specific way a lazy reader would get it wrong is known in advance. The method, StateMem, parses each turn into structured state units with typed dependency links, then handles supersession and staleness deterministically rather than with an LLM.

Motivation Existing memory systems and benchmarks optimize recall of relevant facts, but as agents run longer, facts, constraints and decisions get revised, and an answer must reflect the current state rather than a superseded one. The authors call the failure state drift: the relevant fact is present in the assembled context, but the agent acts on a stale or incomplete version of it. This is distinct from dialogue state tracking, which prescribes a slot-value representation and evaluates it directly over cooperative dialogues that accumulate a goal monotonically; here state is whatever a system must maintain to answer correctly, evaluation is purely behavioral, and revisions are adversarial across sessions. Some concurrent work centers state, but the authors argue none cleanly isolates state tracking from the other errors it co-occurs with.

Methodology The authors first define and label drift on existing benchmarks, assigning a failure to drift only after excluding retrieval, comprehension, schema and reasoning readings, dropping unassignable points, and cross-checking with two judge passes, a cross-family judge and two human annotators. They then build StateMemBench: each scenario is a symbolic event program of typed operations over ground, derived and declared state; the gold answer comes from deterministic replay; a family of executable lazy reader policies is run against the replay, and a scenario is admitted as a trap when policies disagree, with the disagreeing set forming its failure-mode signature (status, salience, sequence, compound, plus anti-trap controls). Programs are grounded in public data for surface vocabulary and rendered into multi-session dialogue by a strong LLM, then programmatically verified for fact placement and phrase leakage. Probes are closed-pool: an unseen pool of three to four options holds the gold answer, the targeted policy's drift answer and neutral distractors. StateMem itself runs a per-turn TurnEncoder producing state units (id, content, priority, source, deps), a deterministic update stage applying supersessions and marking dependents needs_recheck by dependency-graph traversal, and a single answer-time call over the assembled active state. A wrapper variant applies the same trace-then-resolve structure as a prompt-level transformation of any backend's answer call, evaluated against a length- and cost-matched generic-extraction control.

Results Drift leads the confirmed failure distribution on several existing benchmarks (63.5% on MemoryArena-shopping, 44.4% on LongMemEval oracle where retrieval is perfect by construction, 10 of 16 on tau-squared-bench-Z) but is not universal, falling to 19.0% on MemoryArena-travel. On StateMemBench, long-context is not the strong baseline it is on recall tasks: the best long-context model reaches 0.277 and same-backbone long-context 0.149. StateMem reaches 0.363 on DeepSeek-V4-Flash, 1.8x the best memory system and 2.4x same-backbone long-context, and 0.233 on Qwen-3.5-9B, 1.6x the best memory system, both significant by paired McNemar at p < 0.001; GraphRAG at 0.224 is statistically level on Qwen. Ablations put supersession marking as the largest single component and show dependency propagation over-fires on Set B anti-traps by 12.5 points, so removing it leaves DeepSeek slightly better. Drift-rate analysis shows the memory layer barely changes outcomes on a weak answerer, where every arm drifts at 61 to 66%, and separates on the stronger backbone, where StateMem's drift rate falls 15 points and correct answers rise by 42 while long-context, Mem0 and BM25 move by 1 to 3 points. The wrapper improves every one of six backends on both benchmarks, adding 31.7 to 66.6 points on StateMemBench with 15.0 to 31.7 attributable to state structure over the matched control, significant in all twelve cells. State tracking does not cost recall: StateMem also leads memory systems on LongMemEval (0.656 on DeepSeek) and LoCoMo (0.592), with margins concentrated on temporal-reasoning and knowledge-update question types. The authors note StateMem mirrors the policy family behind the traps, so its StateMemBench margins should be read as an upper bound.