\subsection{Fundamentals of Large Language Models}
\subsection{Fundamentals of Large Language Models}
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in future sections.
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in future sections.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question-answer, and machine translation.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_0...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
Specifically, let $\mathbf{x}=x_0...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
@@ -213,6 +213,10 @@ where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference be
...
@@ -213,6 +213,10 @@ where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference be
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
\subsubsection{Rubric-based Reward Models}
\subsubsection{Reward Model Evaluation}
\subsubsection{Reward Model Evaluation}
After training a reward model, an important question is \textit{how to evaluate whether the learned reward function can accurately capture human preferences}. Unlike conventional supervised models that directly predict explicit labels, reward models learn to assign scores that reflect the relative quality of different outputs. As a result, existing evaluation approaches mainly focus on measuring the preference modeling ability of reward models or their effectiveness in downstream RL optimization. Figure~\ref{fig:rm-evaluation} compares three mainstream evaluation paradigms, including RL-based evaluation, pairwise ranking evaluation, and listwise ranking evaluation. We summarize these commonly used evaluation approaches as follows.
After training a reward model, an important question is \textit{how to evaluate whether the learned reward function can accurately capture human preferences}. Unlike conventional supervised models that directly predict explicit labels, reward models learn to assign scores that reflect the relative quality of different outputs. As a result, existing evaluation approaches mainly focus on measuring the preference modeling ability of reward models or their effectiveness in downstream RL optimization. Figure~\ref{fig:rm-evaluation} compares three mainstream evaluation paradigms, including RL-based evaluation, pairwise ranking evaluation, and listwise ranking evaluation. We summarize these commonly used evaluation approaches as follows.
@@ -325,6 +325,14 @@ The third approach is to learn from experience via \textbf{trajectory refinement
...
@@ -325,6 +325,14 @@ The third approach is to learn from experience via \textbf{trajectory refinement
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~(\ref{eq:tau-define}) and~(\ref{eq:reward-optimization-for-agent}), where agents improve their behaviors via feedback from experience.
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~(\ref{eq:tau-define}) and~(\ref{eq:reward-optimization-for-agent}), where agents improve their behaviors via feedback from experience.
\begin{figure}[!t]
\centering
\input{section6/Figures/memory-and-retrieval.tex}
\caption{
An overview of a memory system for agentic interaction. We take a travel-planning scenario as an example, where the user asks the agent to plan a trip to Kyoto. The agent first retrieves relevant user preferences from the memory bank, such as vegetarian meals, window seats, and budget hotels, and uses them to support the current conversation. During the interaction, the user provides new information, such as a preference for quiet ryokans near Kyoto Station. This new information is extracted as memory and passed to a memory manager, which updates the memory bank by selecting an operation such as \texttt{ADD}. \colorbox{blue!30}{Blue} highlights denote retrieved memories, while \colorbox{red!10}{red} highlights denote newly extracted memories and their update path back to the memory bank.
}
\label{fig:memory-and-retrieve}
\end{figure}
\subsubsection{Memory Management}
\subsubsection{Memory Management}
\label{sec:memory-management}
\label{sec:memory-management}
...
@@ -337,18 +345,15 @@ Memory management is a direct way for agents to learn from agentic experience. D
...
@@ -337,18 +345,15 @@ Memory management is a direct way for agents to learn from agentic experience. D
\end{itemize}
\end{itemize}
\begin{figure}[!t]
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
\centering
\input{section6/Figures/memory-and-retrieval.tex}
\caption{
An overview of a memory system for agentic interaction. We take a travel-planning scenario as an example, where the user asks the agent to plan a trip to Kyoto. The agent first retrieves relevant user preferences from the memory bank, such as vegetarian meals, window seats, and budget hotels, and uses them to support the current conversation. During the interaction, the user provides new information, such as a preference for quiet ryokans near Kyoto Station. This new information is extracted as memory and passed to a memory manager, which updates the memory bank by selecting an operation such as \texttt{ADD}. \colorbox{blue!30}{Blue} highlights denote retrieved memories, while \colorbox{red!10}{red} highlights denote newly extracted memories and their update path back to the memory bank.
}
\label{fig:memory-and-retrieve}
\end{figure}
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
\begin{figure}[!t]
\begin{figure}[!t]
\centering
\centering
...
@@ -357,14 +362,6 @@ In learning from experience, it is easy to observe that the performance of memor
...
@@ -357,14 +362,6 @@ In learning from experience, it is easy to observe that the performance of memor
\label{fig:reinforced-memory}
\label{fig:reinforced-memory}
\end{figure}
\end{figure}
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
