\subsection{Fundamentals of Large Language Models}
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in future sections.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question-answer, and machine translation.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_0...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
@@ -210,7 +210,11 @@ We can then define a new loss function for training the generative reward model
\end{eqnarray}
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches. The basic idea is to use only the ranking capability of generative reward models to derive rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
\subsubsection{Rubric-based Reward Models}
\subsubsection{Reward Model Evaluation}
...
...
@@ -234,7 +238,7 @@ After training a reward model, an important question is \textit{how to evaluate
\begin{itemize}
\item\textbf{RL-based Evaluation.}
A straightforward approach to evaluate a reward model is to measure its effectiveness in downstream RL training. Specifically, given multiple reward models $\{\mathcal{M}_r^1,\cdots,\mathcal{M}_r^k\}$, we use each reward model to provide reward signals for training a policy model while keeping the training data, RL algorithm, and hyperparameters identical \citep{wang-etal:probing,frick-etal:ppe}. After RL training, we obtain a set of policies $\{\mathcal{P}_1,\cdots,\mathcal{P}_k\}$, which are evaluated on human preference benchmarks or downstream tasks. The performance of each optimized policy is then used as an indirect measure of the corresponding reward model quality. In this way, we consider that a high-quality reward model should provide more reliable reward signals, enabling the policy to achieve better final performance.
A straightforward approach to evaluate a reward model is to measure its effectiveness in downstream RL training. Specifically, given multiple reward models $\{\mathcal{M}_r^1,\cdots,\mathcal{M}_r^k\}$, we use each reward model to provide reward signals for training a policy model while keeping the training data, RL algorithm, and hyperparameters identical \citep{wang-etal:probing,frick-etal:ppe}. After RL training, we obtain a set of optimized policies $\{\mathcal{P}_1,\cdots,\mathcal{P}_k\}$ and evaluate them on human preference benchmarks or downstream tasks. The performance of each policy then serves as an indirect measure of the quality of its corresponding reward model. In this way, we consider that a high-quality reward model should provide more reliable reward signals, enabling the policy to achieve better final performance.
\item\textbf{Pairwise Ranking Evaluation.}
Although RL-based evaluation directly measures whether a reward model can improve downstream policy optimization, it suffers from two limitations. First, it introduces significant evaluation costs. Since RL training requires substantial computational resources and time, it is difficult to efficiently compare different reward models. Second, the evaluation results can be sensitive to other factors in the RL pipeline, such as the choice of RL algorithms, which may introduce additional variations beyond the quality of the reward model itself. To address these limitations, a more efficient approach is to directly evaluate the preference ranking ability of reward models. Given a prompt $\mathbf{x}$ and two candidate outputs $\mathbf{y}^{+}$ and $\mathbf{y}^{-}$, where $\mathbf{y}^{+}$ is preferred over $\mathbf{y}^{-}$ according to human preference annotations, the reward model predicts their relative preference. Specifically, for discriminative reward models, we compare the predicted scores of the two outputs. If the reward model assigns a higher score to $\mathbf{y}^{+}$, its prediction is consistent with human preference. For generative reward models, the model directly selects $\mathbf{y}^{+}$ as the preferred output based on its generated preference. Otherwise, the prediction is considered incorrect. By constructing a large number of pairwise ranking samples, we can evaluate the reward model using ranking accuracy:
@@ -129,8 +129,9 @@ One is that large-scale RL requires a highly generalizable reward model. As the
Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behaviour region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between keeping stability and fostering adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
\subsection{On-Policy Distillation}
While large-scale RL has shown strong potential for improving model capabilities, it often relies on large amounts of verifiable training data, where the correctness of model outputs can be reliably evaluated. Several approaches have been explored to relax this requirement. A straightforward solution is to construct proxy rewards from the model's own outputs. For example, majority voting can be used to derive pseudo-gold answers for computing accuracy-based rewards, while uncertainty-related signals such as output entropy can also serve as reward signals. However, these approaches largely depend on the model's own capabilities and may provide unreliable supervision. A more promising direction is \textit{on-policy distillation} (OPD), which introduces supervision from a stronger teacher model while preserving on-policy exploration of the student model\footnote{In standard knowledge distillation settings, we typically refer to the ``smaller'' model as the \textit{student model} and the ``larger'' model as the \textit{teacher model}.}. The key idea is to let the student model generate reasoning trajectories on-policy and then use the teacher model to provide token-level distributional supervision at the states actually visited by the student.
While large-scale RL has shown strong potential for improving model capabilities, it often relies on large amounts of verifiable training data. Several approaches have been explored to relax this requirement. A straightforward solution is to construct proxy rewards from the model's own outputs. A straightforward solution is to construct proxy rewards from the model's own outputs. For example, we can use majority voting to derive pseudo-gold answers and compute accuracy-based rewards. A more promising direction is \textit{on-policy distillation} (OPD), which introduces supervision from a stronger teacher model while preserving on-policy exploration of the student model\footnote{In standard knowledge distillation settings, we typically refer to the ``smaller'' model as the \textit{student model} and the ``larger'' model as the \textit{teacher model}.}. The key idea is to let the student model generate reasoning trajectories on-policy and then use the teacher model to provide token-level distributional supervision at the states actually visited by the student.
Compared with conventional offline distillation, OPD adopts a simple but important design: the training samples are generated online by the current student policy rather than collected offline from the teacher model. This design mainly addresses the distribution mismatch in offline distillation, where the student is trained on teacher-generated trajectories that may differ from the states it encounters during its own inference. In contrast, OPD provides teacher supervision directly on the states visited by the student, allowing the teacher to correct the student's actual behaviors. For example, consider a mathematical reasoning problem where the student has already generated the partial trajectory ``3x+7=22 $\rightarrow$ 3x=15''. At this student-generated state, the teacher model can directly provide token-level supervision that assigns a higher probability to generating ``x=5'' next, rather than an incorrect continuation such as ``x=4''. In this way, OPD teaches the student how to continue correctly from the states it actually visits, rather than only imitating complete trajectories generated by the teacher model.
@@ -325,6 +325,14 @@ The third approach is to learn from experience via \textbf{trajectory refinement
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~(\ref{eq:tau-define}) and~(\ref{eq:reward-optimization-for-agent}), where agents improve their behaviors via feedback from experience.
\begin{figure}[!t]
\centering
\input{section6/Figures/memory-and-retrieval.tex}
\caption{
An overview of a memory system for agentic interaction. We take a travel-planning scenario as an example, where the user asks the agent to plan a trip to Kyoto. The agent first retrieves relevant user preferences from the memory bank, such as vegetarian meals, window seats, and budget hotels, and uses them to support the current conversation. During the interaction, the user provides new information, such as a preference for quiet ryokans near Kyoto Station. This new information is extracted as memory and passed to a memory manager, which updates the memory bank by selecting an operation such as \texttt{ADD}. \colorbox{blue!30}{Blue} highlights denote retrieved memories, while \colorbox{red!10}{red} highlights denote newly extracted memories and their update path back to the memory bank.
}
\label{fig:memory-and-retrieve}
\end{figure}
\subsubsection{Memory Management}
\label{sec:memory-management}
...
...
@@ -337,18 +345,15 @@ Memory management is a direct way for agents to learn from agentic experience. D
\end{itemize}
\begin{figure}[!t]
\centering
\input{section6/Figures/memory-and-retrieval.tex}
\caption{
An overview of a memory system for agentic interaction. We take a travel-planning scenario as an example, where the user asks the agent to plan a trip to Kyoto. The agent first retrieves relevant user preferences from the memory bank, such as vegetarian meals, window seats, and budget hotels, and uses them to support the current conversation. During the interaction, the user provides new information, such as a preference for quiet ryokans near Kyoto Station. This new information is extracted as memory and passed to a memory manager, which updates the memory bank by selecting an operation such as \texttt{ADD}. \colorbox{blue!30}{Blue} highlights denote retrieved memories, while \colorbox{red!10}{red} highlights denote newly extracted memories and their update path back to the memory bank.
}
\label{fig:memory-and-retrieve}
\end{figure}
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''.
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
\begin{figure}[!t]
\centering
...
...
@@ -357,14 +362,6 @@ In learning from experience, it is easy to observe that the performance of memor
\label{fig:reinforced-memory}
\end{figure}
One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
