Chenglong Wang, Yifu Huo, Yang Gan, Qiaozhi He, Qi~Meng, Bei Li, Yan Wang, Junfu Liu, Tianhua Zhou, Jingbo Zhu, et~al.
...
...
@@ -594,6 +594,11 @@ Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi C
\newblock In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), \emph{Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023}, 2023{\natexlab{d}}.
where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.
After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. As a result, we can consider memory as a core component of agent learning rather than a passive storage module \citep{xu-etal:a-mem}. In this way, instead of treating memory updates as isolated operations for individual tasks, we can view memory management as an ongoing learning process, where agents gradually organize past experiences into reusable knowledge.
After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately \citep{xu-etal:a-mem}. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. This indicates that memory can serve as an important component of agent learning, beyond simply storing past information.
Beyond optimizing memory update operations, another important question is how to construct and organize memories from accumulated experiences. Instead of treating memory construction as a fixed preprocessing step, we can make an agent to learn effective memory construction strategies through RL. In this way, a straightforward approach is to formulate memory construction as a sequential decision-making process, where the agent learns how to select, organize, and transform interaction experiences into useful memories \citep{wang-etal:mem}.
% In this way, instead of treating memory updates as isolated operations for individual tasks, we can view memory management as an ongoing learning process, where agents gradually organize past experiences into reusable knowledge. As a result, we can consider memory as a core component of agent learning rather than a passive storage module \citep{xu-etal:a-mem}.
% 是否插入memory之后，我们就不管了近一步，我们还可以组织Memory
\subsubsection{Skill Optimization}
Skill optimization compresses repeated experience into reusable behavioral modules. A skill can be represented as a structured instruction package that tells the agent when the skill should be invoked, what workflow it should follow, which tools are appropriate, and how the final answer should be verified. For example, a reasoning-oriented skill may be written as follows:
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\parindent}{0pt}
\setlength{\leftskip}{2.5em}
\setlength{\rightskip}{2.5em}
\uline{Name: Math-Logic-Reasoning}\par
\textit{Description}: Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Use when a task requires multi-step arithmetic, symbolic reasoning, calculator-backed verification, table analysis, optimization, scheduling or allocation constraints, Pareto trade-offs, proof checking, or translating word problems into equations, code, formulas, or executable checks.\par
\vspace{2mm}
\uline{Math Logic Reasoning}\par
\vspace{0.2em}
Use this skill for reasoning tasks where a wrong intermediate assumption can silently break the final answer. Prefer a compact formal model plus executable verification over unsupported mental arithmetic.\par
\vspace{0.4em}
\uline{Workflow}\par
1. Restate the target quantity, decision, or claim.\par
2. Extract givens, units, constraints, and hidden assumptions.\par
3. Convert the problem into equations, inequalities, tables, spreadsheet formulas, graphs, search spaces, constraint satisfaction problems, or short verification scripts.\par
4. Solve with the simplest reliable tool.\par
5. Independently verify the result by recomputing, checking edge cases, or testing constraints.\par
6. Explain the final answer with only the necessary derivation.\par
\vspace{0.4em}
\uline{Tool Choices}\par
1. Use Python as a calculator for multi-step arithmetic, combinatorics, simulations, optimization search, or exact rational checks.\par
2. Use Fraction, Decimal, or sympy when exactness matters.\par
3. Use pandas for complex tables, joins, grouped summaries, rankings, or data-cleaning logic.\par
4. Use brute force, dynamic programming, or linear/integer programming when choices are constrained.\par
\vspace{0.4em}
\uline{Verification Checklist}\par
1. Recalculate the final numeric answer using a second method when feasible.\par
2. Test boundary cases: zero, negative values, equality limits, duplicate entries, and empty sets.\par
3. Confirm units after every transformation.\par
$\cdots\cdots$\par
\vspace{0.4em}
\uline{Output Style}\par
Give the answer first when possible. Then include a concise derivation, the checked constraints, and any caveats.
\endgroup
\end{tcolorbox}
Given such a skill, the agent does not need to store every previous reasoning trajectory in full. Instead, successful experience can be distilled into reusable procedures and verification rules. During later interactions, the agent can retrieve the skill, instantiate it for the current task, and update it when repeated failures reveal missing assumptions or insufficient checks.
Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
Reinforcement Learning for Self-Improving Agent with Skill Library
