@@ -339,11 +339,20 @@ Memory management is a direct way for agents to learn from agentic experience. D
...
@@ -339,11 +339,20 @@ Memory management is a direct way for agents to learn from agentic experience. D
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
\caption{Overview of training a memory manager with RL.}
\label{fig:reinforced-memory}
\end{figure}
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience.
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
