\newblock Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.
\newblock In \emph{Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 12805--12825, 2026.
\bibitem[Yang et~al.(2024)Yang, Ding, Lin, Zhang, and Zhang]{yang-etal:2024regularizing}
title={Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning},
author={Yan, Sikuan and Yang, Xiufeng and Huang, Zuchao and Nie, Ercong and Ding, Zifeng and Li, Zonggen and Ma, Xiaowen and Bi, Jinhe and Kersting, Kristian and Pan, Jeff Z and others},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={12805--12825},
year={2026}
}
@article{chhikara-etal:mem0,
title={Mem0: Building production-ready ai agents with scalable long-term memory},
author={Chhikara, Prateek and Khant, Dev and Aryan, Saket and Singh, Taranjeet and Yadav, Deshraj},
journal={arXiv preprint arXiv:2504.19413},
year={2025}
}
@inproceedings{li-etal:word2world,
title={From word to world: Can large language models be implicit text-based world models?},
author={Li, Yixia and Wang, Hongru and Qiu, Jiahao and Yin, Zhenfei and Zhang, Dongdong and Qian, Cheng and Li, Zeping and Ma, Xiaoteng and Chen, Guanhua and Ji, Heng},
@@ -7,7 +7,7 @@ Using RL to train agents, often referred to as \textit{agentic RL}, is not a new
With the rise of LLMs as core components of autonomous systems, RL has been increasingly used to adapt these models for sequential decision-making in dynamic and open-ended environments. In this setting, agentic RL addresses a key limitation of LLMs: although they encode extensive world knowledge, their outputs are not inherently aligned with long-horizon task requirements. For example, a supervised fine-tuned LLM may correctly generate a script for calling an external API, but it cannot reliably self-correct when the environment returns unexpected errors. This creates new challenges for enabling LLM-based agents to plan autonomously, adapt to environmental feedback, and safely improve their strategies through iterative interaction.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory updates, skill optimization, and trajectory refinement without altering the internal model weights. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory management, skill optimization, and trajectory refinement without altering the internal model weights. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
\subsection{Building Agent Capabilities}
...
...
@@ -317,9 +317,46 @@ The three approaches mentioned above can be implemented through various techniqu
\subsubsection{Memory Management}
However, this approach introduces a key challenge in memory management: deciding when and what information should be updated in memory.
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
\begin{itemize}
\item\textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory to support the current interaction. The retrieved information can take various forms depending on how the memory is constructed. For example, the memory may contain successful trajectories from similar tasks, which can provide reusable solutions for current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles.
\item\textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
\item\textbf{Memory Update.} The agent updates the memory by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in\{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory unchanged. We can typically achieve this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
Memory-R1 addresses this problem by introducing reinforcement learning into memory management \citep{yan-etal:memoryr1}. The key idea is to make memory operations learnable. Given new information $x$ and an existing memory bank $\mathcal{M}_{old}$, the Memory Manager acts as a policy that selects a memory operation and produces updated memory content:
where $o$ is one of \texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, and \texttt{NOOP}, and $m'$ is the updated memory content. After applying this operation, the updated memory bank is used by an Answer Agent to solve downstream questions.
The reward is defined according to the final task outcome. If the updated memory helps the Answer Agent produce the correct answer, the Memory Manager receives a positive reward. Otherwise, it receives a lower reward. A simple reward can be written as
where $y_{\text{pred}}$ is the predicted answer and $y_{\text{gold}}$ is the ground-truth answer. This reward does not require manually labeling every memory operation. Instead, the system learns which memory operations are useful by observing whether they improve downstream task performance.
Memory-R1 also optimizes how retrieved memories are used. In many cases, retrieval may return a large number of candidate memories, only a few of which are truly useful. If all retrieved memories are directly appended to the prompt, the agent may be distracted by irrelevant information. Therefore, Memory-R1 introduces an Answer Agent that performs memory distillation. It selects and reasons over the most relevant memories before generating the final answer. This makes memory utilization an active decision process rather than a passive retrieval step.
memory-R1
Overall, memory management turns agentic experience into persistent knowledge. Mem0 shows how an agent can extract and update memory through structured operations, while Memory-R1 further shows how these operations can be optimized with RL. From the perspective of agentic experience learning, the key point is that the agent improves not only by changing its model parameters, but also by changing what it remembers and how it uses those memories in future tasks.
