@@ -319,9 +319,9 @@ The three approaches mentioned above can be implemented through various techniqu
...
@@ -319,9 +319,9 @@ The three approaches mentioned above can be implemented through various techniqu
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
\begin{itemize}
\begin{itemize}
\item\textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory to support the current interaction. The retrieved information can take various forms depending on how the memory is constructed. For example, the memory may contain successful trajectories from similar tasks, which can provide reusable solutions for current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles.
\item\textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory bank to support the current interaction. The retrieved information can take various forms depending on how the memory bank is constructed. For example, the memory bank may contain successful trajectories from similar tasks, which can provide reusable solutions for current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles.
\item\textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
\item\textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
\item\textbf{Memory Update.} The agent updates the memory by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in\{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory unchanged. We can typically achieve this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
\item\textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in\{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically achieve this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
\end{itemize}
\end{itemize}
...
@@ -339,25 +339,19 @@ Memory management is a direct way for agents to learn from agentic experience. D
...
@@ -339,25 +339,19 @@ Memory management is a direct way for agents to learn from agentic experience. D
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience.
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.
Memory-R1 addresses this problem by introducing reinforcement learning into memory management \citep{yan-etal:memoryr1}. The key idea is to make memory operations learnable. Given new information $x$ and an existing memory bank $\mathcal{M}_{old}$, the Memory Manager acts as a policy that selects a memory operation and produces updated memory content:
where $o$ is one of \texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, and \texttt{NOOP}, and $m'$ is the updated memory content. After applying this operation, the updated memory bank is used by an Answer Agent to solve downstream questions.
The reward is defined according to the final task outcome. If the updated memory helps the Answer Agent produce the correct answer, the Memory Manager receives a positive reward. Otherwise, it receives a lower reward. A simple reward can be written as
where $y_{\text{pred}}$ is the predicted answer and $y_{\text{gold}}$ is the ground-truth answer. This reward does not require manually labeling every memory operation. Instead, the system learns which memory operations are useful by observing whether they improve downstream task performance.
Memory-R1 also optimizes how retrieved memories are used. In many cases, retrieval may return a large number of candidate memories, only a few of which are truly useful. If all retrieved memories are directly appended to the prompt, the agent may be distracted by irrelevant information. Therefore, Memory-R1 introduces an Answer Agent that performs memory distillation. It selects and reasons over the most relevant memories before generating the final answer. This makes memory utilization an active decision process rather than a passive retrieval step.
Overall, memory management turns agentic experience into persistent knowledge. Mem0 shows how an agent can extract and update memory through structured operations, while Memory-R1 further shows how these operations can be optimized with RL. From the perspective of agentic experience learning, the key point is that the agent improves not only by changing its model parameters, but also by changing what it remembers and how it uses those memories in future tasks.
