@@ -198,7 +198,7 @@ License: CC-BY 4.0, see \url{https://creativecommons.org/licenses/by/4.0/}.
RL is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles.
A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of LLMs. For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1,kimi-team:kimi}. As a result, interest in NLP has exploded. There have been many, many conference papers discussing RL in LLMs, and even more in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI. This trend is further strengthened by the rise of LLM-based agents, where models must learn from interaction and experience to make effective decisions in dynamic environments. From this perspective, RL is becoming a key technique for building more capable and adaptive intelligent systems. This view closely echoes the ``Reward Is Enough'' hypothesis proposed by \citet{silver-etal:reward}:
A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of LLMs. For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1,kimi-team:kimi}. As a result, interest in RL for LLMs has exploded. Numerous conference papers now discuss RL in LLMs, with even more appearing in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI. This trend is further strengthened by the rise of LLM-based agents, where models must learn from interaction and experience to make effective decisions in dynamic environments. From this perspective, RL is becoming a key technique for building more capable and adaptive intelligent systems. This view closely echoes the ``Reward Is Enough'' hypothesis proposed by \citet{silver-etal:reward}:
\begin{quote}
``Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.''
\end{quote}
...
...
@@ -208,7 +208,7 @@ This hypothesis highlights why RL is particularly relevant to the next stage of
Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in the era of LLMs, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
Of course, we'd like to learn about RL, which appears straightforward. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
It is natural to learn RL from standard references, which appears straightforward at first. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM research. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through a concrete example of training an LLM. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
...
...
@@ -245,7 +245,7 @@ The aim of this paper is to provide a comprehensive introduction to RL from the
\section*{Acknowledgements}
We would like to thank for their suggestions on improving the early version of this work.
We would like to thank those who provided suggestions on improving the early version of this work.
\subsection{Fundamentals of Large Language Models}
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in future sections.
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in the following sections.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_1...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
@@ -23,16 +23,16 @@ There are three tips for improving accuracy in solving math problems: \\ [1mm]
Although this output adheres to the given input, it is very short and not sufficiently detailed or comprehensive for this scenario. In practice, the ``short'' feature in the generated output stems from the training approach of the SFT LLM. During SFT, the LLM learns from a large set of labeled samples that typically contain concise and factual answers to common inputs. These samples guide the LLM to prioritize brevity and clarity, so it often generates short outputs, like the one shown above. Of course, we could annotate enough additional data to fine-tune the LLM further and adjust it to generate the desired outputs. However, this approach is limited in its ability to scale. For example, in this scenario, the input involves a variety of potential answers, and the expectations of the student can be pretty diverse. Therefore, describing the ``ideal'' output would require immense annotation effort. Consequently, collecting or annotating fine-tuning data is not as straightforward as it is with SFT, particularly when attempting to cover the breadth of possible student inputs and outputs. Instead, we can use RL to enable the model to discern outputs that better align with human preferences, such as generating more detailed and comprehensive content that not only adheres to the given input but also meets the expectations of the student in this scenario.
In the following sections, we will demonstrate how to use RL to train LLMs through a specific example—enabling the LLM to generate a longer output in the context of a homework assistant. Throughout this example, we introduce RL, including key algorithms and their improvements.
In the following sections, we will demonstrate how to use RL to train LLMs through a specific example, enabling the LLM to generate a longer output in the context of a homework assistant. Throughout this example, we introduce RL, including key algorithms and their improvements.
% policy gradient
\subsection{Policy Gradient}
\label{sec:policy-gradient}
In this section, we aim to lengthen the LLM output for the ``give me three tips'' input using a classical RL algorithm called \textbf{policy gradient}.
To define our optimization objective, let $R(\cdot)$ be the reward function that describes the goal the algorithm aims to optimize. In this scenario, the reward is designed to align the output of the LLM with a specific behaviour, that is, generating longer outputs. Therefore, we define the reward function based on a length-related metric. More specifically, the reward can be proportional to the length of the output, that is, longer outputs can receive higher rewards:
To define our optimization objective, let $R(\cdot)$ be the reward function that describes the goal the algorithm aims to optimize. In this scenario, the reward is designed to align the output of the LLM with a specific behavior, that is, generating longer outputs. Therefore, we define the reward function based on a length-related metric. More specifically, the reward can be proportional to the length of the output, that is, longer outputs can receive higher rewards:
where $\mathrm{Length}(\cdot)$ indicates the length of the given output. We can use the number of tokens in the output as the length for simplicity. This allows us to assign an immediate reward $r_t$ for the $t$-th token generated by the LLM, which is $\frac{1}{10}$.
...
...
@@ -54,7 +54,7 @@ The optimization objective can be given by
\label{fig:update-with-policy-gradient}
\end{figure*}
where $S_{x}$ indicates the input-only dataset, $\mathbf{y}\in\Omega$ indicates that output $\mathbf{y}$ is drawn from the hypothesis space $\Omega$, $T$ indicates the length of $\mathbf{y}$. $J(\theta)$ is also called the performance function. Then the training objective is maximize $J(\theta)$:
where $\mathcal{S}_{x}$ indicates the input-only dataset, $\mathbf{y}\in\Omega$ indicates that output $\mathbf{y}$ is drawn from the hypothesis space $\Omega$, and $T$ indicates the length of $\mathbf{y}$. $J(\theta)$ is also called the performance function. Then the training objective is to maximize $J(\theta)$:
\begin{eqnarray}
\hat{\theta}& = &\argmax_{\theta} J(\theta)
\end{eqnarray}
...
...
@@ -82,13 +82,13 @@ We can further refine the process when optimizing the objective using gradient d
\label{fig:understand-policy-gradient}
\end{figure*}
We assume that every output in $\mathcal{D}$ is equally probable (i.e., $\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})=1/|\mathcal{D}|$). In this case we can simplify Eq. (\ref{eq:rl-gradient-j-theta}) and need only consider the terms $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta}$ and $R(\mathbf{y})$:
Using a Monte Carlo sample average over $d$ outputs in $\mathcal{D}$, we can simplify Eq. (\ref{eq:rl-gradient-j-theta}) and need only consider the terms $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta}$ and $R(\mathbf{y})$:
Now, as illustrated in Figure \ref{fig:update-with-policy-gradient}, we have an RL approach to optimize the output of the LLM: 1) we sample some inputs from the SFT LLM; then, 2) we evaluate each input using a reward function that we define; then, 3) we update the LLM using gradient descent, following Eq. (\ref{eq:rl-j-theta-gradient-simplified}), to maximize the performance function.
Now, as illustrated in Figure \ref{fig:update-with-policy-gradient}, we have an RL approach to optimize the output of the LLM: 1) we sample inputs from the input-only dataset and sample outputs from the LLM; then, 2) we evaluate each sampled output using a reward function that we define; then, 3) we update the LLM using gradient descent, following Eq. (\ref{eq:rl-j-theta-gradient-simplified}), to maximize the performance function.
We can understand this optimization process from the perspective of hypothesis space reranking, as illustrated in Figure \ref{fig:understand-policy-gradient}. The objective is to increase the probability of longer outputs in the hypothesis space by assigning a high reward to those outputs, while simultaneously decreasing the probability of shorter outputs by assigning them a lower reward. In other words, the policy gradient approach encourages the model to generate longer outputs by reinforcing those outputs that meet the desired length criteria, and it penalizes shorter outputs that do not meet the objective.
...
...
@@ -97,13 +97,13 @@ We can understand this optimization process from the perspective of hypothesis s
\subsection{Temporal Decomposition}
\label{sec:temporal-decomposition}
While optimizing using Eq. (\ref{eq:rl-j-theta-gradient-simplified}) is intuitive, a potential issue arises: in some cases, the sampled outputs we sample may significantly overlap, yet the rewards for the overlapping parts differ. For example, as illustrated in Figure \ref{fig:an-overlap-example}, if outputs $\mathbf{y}_1$ and $\mathbf{y}_2$ both include the same tokens in the third tip ``3.Double-check your work...any-one can significantly enhance their math problem-solving abilities.'', but due to variations in the entire outputs, they receive markedly different rewards (e.g., $R(\mathbf{y}_1)=50$ vs. $R(\mathbf{y}_2)=10$). Ideally, we would want similar generational behaviours to be rewarded consistently, without significant variation, as their contributions are equivalent to the sum of rewards.
While optimizing using Eq. (\ref{eq:rl-j-theta-gradient-simplified}) is intuitive, a potential issue arises: in some cases, the sampled outputs may significantly overlap, yet the rewards for the overlapping parts differ. For example, as illustrated in Figure \ref{fig:an-overlap-example}, if outputs $\mathbf{y}_1$ and $\mathbf{y}_2$ both include the same tokens in the third tip ``3.Double-check your work...anyone can significantly enhance their math problem-solving abilities.'', but due to variations in the entire outputs, they receive markedly different rewards (e.g., $R(\mathbf{y}_1)=50$ vs. $R(\mathbf{y}_2)=10$). Ideally, we would want similar generation behaviors to be rewarded consistently, without significant variation, as their contributions are equivalent to the sum of rewards.
Before discussing an approach to solve this issue, we first perform temporal decomposition for the term $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta} R(\mathbf{y})$ in Eq. (\ref{eq:rl-j-theta-gradient-simplified}), and obtain
@@ -125,7 +125,7 @@ In fact, this simplification follows the Markov process, asserting that the futu
\subsection{Reducing Gradient Variance}
\label{sec:reduce-gradient-variance}
Returning to the case discussed in Section \ref{sec:temporal-decomposition}, while the strategy of excluding rewards for past tokens reduces gradient variance, substantial gradient variance still occurs in practice. First, if overlapping portions of the output appear early, the modified Eq. (\ref{eq:modified-gradient-simlified}) may still experience similar problems, i.e., the same tokens receive vastly different rewards. Furthermore, the reward distribution can vary significantly between different time steps within a single output. For example, consider an output sequence of 500 tokens. According to Eq. (\ref{eq:modified-gradient-simlified}), the reward at the 10th step might be significantly higher than at the 450th step, i.e., 49 vs. 5. Such varying rewards for good and poor tokens can result in a very low total reward for the entire output, even if it includes good tokens.
Returning to the case discussed in Section \ref{sec:temporal-decomposition}, while the strategy of excluding rewards for past tokens reduces gradient variance, substantial gradient variance still occurs in practice. First, if overlapping portions of the output appear early, the modified Eq. (\ref{eq:modified-gradient-simplified}) may still experience similar problems, i.e., the same tokens receive vastly different rewards. Furthermore, the reward distribution can vary significantly between different time steps within a single output. For example, consider an output sequence of 500 tokens. According to Eq. (\ref{eq:modified-gradient-simplified}), the reward at the 10th step might be significantly higher than at the 450th step, i.e., 49 vs. 5. Such varying rewards for good and poor tokens can result in a very low total reward for the entire output, even if it includes good tokens.
One simple method for further reducing the variance of the gradient is to set a baseline $b$ and subtract it from $\sum_{k=t}^{T} r_k$, resulting in $\sum_{k=t}^{T} r_k - b$.\footnote{In fact, the use of a baseline $b$ does not change the variance of the total rewards $\sum_{t=1}^{T} r_t$. However, it is important to note that while introducing a baseline does not alter the overall variance of the rewards, it helps reduce the variance of the gradient estimates. This is because subtracting the baseline from the total rewards effectively reduces fluctuations around their mean, which makes the gradient estimates more stable. In general, the operation $\sum_{k=t}^{T} r_k - b$ centers the rewards around zero (e.g., $b$ is defined as the expected value of $\sum_{k=t}^{T} r_k$), which can lead to reduced variance in the product $\sum_{k=t}^{T}\log\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})(\sum_{k=t}^{T} r_k - b)$.} Here, the baseline can be interpreted as a reference point. By centering the rewards around this baseline, we remove systematic biases in the reward.
...
...
@@ -133,7 +133,7 @@ This policy gradient model with a baseline can be given by
There are many ways to define the baseline $b$. For example, we can set the baseline $b$ to the average length of the sampled outputs across $\mathcal{D}$, i.e., $b =(\sum_{\mathbf{y}\in\mathcal{D}}\mathrm{Length}(\mathbf{y}))/d$. While this approach can reduce the relative size of the gradient variance, it does not address the issue of varying gradient variance across different time steps within a single output. To tackle this challenge, we would want a different baseline for each time step that can specifically be adjusted to reduce variance at that step. To achieve this ideal baseline, we can introduce a value function $V(\cdot)$. In the training of the LLM, this value function calculates the expected value of the sum of future rewards (or return for short) when generating $y_{t}$, given the input $\mathbf{x}$ and the tokens previously generated $\mathbf{y}_{<t}$, i.e., $V(\mathbf{x}, \mathbf{y}_{<t}, y_{t})=\mathbb{E}[\sum_{k=t}^T r_{k}]$. Hence we have
...
...
@@ -148,7 +148,7 @@ By using the advantage function $A_{t}$, the gradient of $J(\theta)$ can be writ
Based on this training objective, the loss function of training the LLM can be written in the form
...
...
@@ -186,13 +186,13 @@ or alternatively, introduce the discount factor $\gamma$ to obtain a more genera
where $\gamma\in[0,1]$ is the discount factor that adjusts the importance of future rewards. When $\gamma$ is set to less than 1, it signifies that early rewards are considered more important than future rewards. This basic idea is also applied in other fields. For example, in LLMs, it has been demonstrated that early token generation plays a crucial role, as it can influence the style and accuracy of the entire output \citep{wang-and-zhou:2024chain}.
In this subsection, we have detailed the integration of a value model in training LLMs. In practice, this training approach, represented by Eq. (\ref{eq:modified-gradient-simlified-advantage}), is known as the Advantage Actor-Critic (A2C) method \citep{mnih-etal:2016asynchronous}. The A2C method facilitates an interaction where a policy model (the actor) and a value model (the critic) learn in parallel and undergo synchronous updates. However, RL is a vast field, and many technical details cannot be covered here. The interested reader can refer to RL books for more details \citep{szepesvari:2010algorithms, Sutton-and-Barto:2018RL}.
In this subsection, we have detailed the integration of a value model in training LLMs. In practice, this training approach, represented by Eq. (\ref{eq:modified-gradient-simplified-advantage}), is known as the Advantage Actor-Critic (A2C) method \citep{mnih-etal:2016asynchronous}. The A2C method facilitates an interaction where a policy model (the actor) and a value model (the critic) learn in parallel and undergo synchronous updates. However, RL is a vast field, and many technical details cannot be covered here. The interested reader can refer to RL books for more details \citep{szepesvari:2010algorithms, Sutton-and-Barto:2018RL}.
Workflow of integrating important sampling in the training of LLMs with policy gradient. In Step 1, the current policy is synchronized to establish the reference policy. In Step 2, we sample an output $\mathbf{y}$ from this reference policy for a given input $\mathbf{x}$. Finally, in Step 3, the sequence $[\mathbf{x}, \mathbf{y}]$ is used to optimize the current policy multiple times. After optimization, the updated policy is synchronized to become the reference policy.
Workflow of integrating importance sampling in the training of LLMs with policy gradient. In Step 1, the current policy is synchronized to establish the reference policy. In Step 2, we sample an output $\mathbf{y}$ from this reference policy for a given input $\mathbf{x}$. Finally, in Step 3, the sequence $[\mathbf{x}, \mathbf{y}]$ is used to optimize the current policy multiple times. After optimization, the updated policy is synchronized to become the reference policy.
where $\theta_{\mathrm{ref}}$ denotes the parameters of the previously used LLM (also called reference policy or old policy). Here, we use $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ or $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot|\mathbf{x})$ to denote $\mathcal{D}$, distinguishing from which model the output is sampled. The ratio $\frac{\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}$, also called the ratio function, compares the log-probability of token $y_t$ under the current and reference policies. This ratio function is used to reweight the observed rewards, reflecting how much more or less likely a token is under the current policy compared to the reference policy. When this ratio is greater than 1, it indicates that the token $y_t$ is more favored by the current policy than the reference policy. Conversely, a ratio less than 1 indicates that $y_t$ is less favored by the current policy. However, when the current used$\theta_{\mathrm{ref}}$ diverges significantly from $\theta$, the accuracy of the hypothesis space estimation decreases. Therefore, we do need to resync it regularly. As illustrated in Figure \ref{fig:policy-gradient-with-importance-sampling}, one simple way is to designate the reference policy as the LLM from which we initiate updates during a training step. Note that we can conserve computational time and memory by eliminating the need for model copying in Step 1. Specifically, rather than maintaining a separate copy of the model, we directly sample from the current policy, retain the output probabilities $\{\mathrm{Pr}_{\theta}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$, and later use these stored probabilities to act as $\{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$ when updating the policy with Eq. (\ref{eq:loss-function-training-llm-importance-sampling}).
where $\theta_{\mathrm{ref}}$ denotes the parameters of the previously used LLM (also called reference policy or old policy). Here, we use $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ or $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot|\mathbf{x})$ to denote $\mathcal{D}$, distinguishing from which model the output is sampled. The ratio $\frac{\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}$, also called the ratio function, compares the probability of token $y_t$ under the current and reference policies. This ratio function is used to reweight the observed rewards, reflecting how much more or less likely a token is under the current policy compared to the reference policy. When this ratio is greater than 1, it indicates that the token $y_t$ is more favored by the current policy than the reference policy. Conversely, a ratio less than 1 indicates that $y_t$ is less favored by the current policy. However, when the current$\theta_{\mathrm{ref}}$ diverges significantly from $\theta$, the accuracy of the hypothesis space estimation decreases. Therefore, we do need to resync it regularly. As illustrated in Figure \ref{fig:policy-gradient-with-importance-sampling}, one simple way is to designate the reference policy as the LLM from which we initiate updates during a training step. Note that we can conserve computational time and memory by eliminating the need for model copying in Step 1. Specifically, rather than maintaining a separate copy of the model, we directly sample from the current policy, retain the output probabilities $\{\mathrm{Pr}_{\theta}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$, and later use these stored probabilities to act as $\{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$ when updating the policy with Eq. (\ref{eq:loss-function-training-llm-importance-sampling}).
However, an inherent issue arises when using importance sampling to update LLMs. As shown in Figure \ref{fig:terrible-step-importance-sampling}, when a particular optimization step results in poor updates (or a terrible step for short), the reference policy utilized in subsequent Step 1 of the next iteration inherits these poor characteristics. Consequently, the samples drawn in Step 2 are likely to be adversely affected, leading to a more terrible step. Unlike SFT\footnote{In supervised learning, a terrible step during a training step caused by bad samples can often be corrected in subsequent steps by good samples.}, this cycle does not correct the initial terrible step but potentially exacerbates it, creating a downward spiral in policy performance.
...
...
@@ -268,15 +268,15 @@ In Eq. (\ref{eq:importance-sampling-with-penalty}), the range of the ratio funct
where the clipping function $\mathrm{Clip}(\cdot)$ can be defined by
where the function $\mathrm{bound}(\cdot)$ constrains the ratio function to within the range $[1-\epsilon, 1+\epsilon]$. The clipping imposed by Eq. (\ref{eq:cliping-function}) is illustrated in Figure \ref{fig:ppo-clip}. This training method is also known as proximal policy optimization (PPO) \citep{schulman-etal:2017proximal}, currently the most widely used method for training LLMs with RL. Originally, PPO also introduced an adaptive $\beta$ to dynamically control this penalty. However, this adaptive approach has not been widely applied in training LLMs. This is because this process depends on the expectation of the penalty (i.e., $\mathbb{E}(\mathrm{Penalty})$), which would introduce additional computational overhead. Interested readers can refer to the original literature for more details on this adaptive approach.
where the function $\mathrm{bound}(\cdot)$ constrains the ratio function to within the range $[1-\epsilon, 1+\epsilon]$. The clipping imposed by Eq. (\ref{eq:clipping-function}) is illustrated in Figure \ref{fig:ppo-clip}. This training method is also known as proximal policy optimization (PPO) \citep{schulman-etal:2017proximal}, currently the most widely used method for training LLMs with RL. Originally, PPO also introduced an adaptive $\beta$ to dynamically control this penalty. However, this adaptive approach has not been widely applied in training LLMs. This is because this process depends on the expectation of the penalty (i.e., $\mathbb{E}(\mathrm{Penalty})$), which would introduce additional computational overhead. Interested readers can refer to the original literature for more details on this adaptive approach.
% introduce reward models
\subsection{Training Reward Models}
\label{sec:training-reward-models}
While our initial reward function focused on output length, as described in Section \ref{sec:policy-gradient}, provides effective signals in the learning process, human preferences are considerably more complex and extend beyond merely preferring longer outputs. For example, in scenarios like a homework assistant, it is essential not only to generate longer outputs but also to avoid redundancy, ensure accuracy, and maintain fluency. This complexity underscores the need for a sophisticated reward modeling technique, moving beyond simple function-based methods to more effectively capture human preferences.
While our initial reward function based on output length, as described in Section \ref{sec:policy-gradient}, provides effective signals in the learning process, human preferences are considerably more complex and extend beyond merely preferring longer outputs. For example, in scenarios like a homework assistant, it is essential not only to generate longer outputs but also to avoid redundancy, ensure accuracy, and maintain fluency. This complexity underscores the need for a sophisticated reward modeling technique, moving beyond simple function-based methods to more effectively capture human preferences.
Given these complexities, we typically train a reward model, a neural network that maps a pair of input and output token sequences to a scalar value, to capture human preferences. Given an input $\mathbf{x}$ and an output $\mathbf{y}$, the reward is expressed as $\mathrm{Reward}(\mathbf{x},\mathbf{y})$, where $\mathrm{Reward}(\cdot)$ denotes the reward model. There are many ways to implement the reward model. One simple approach is to build the reward model based on a pre-trained LLM, similar to the value model as presented in Section \ref{sec:reduce-gradient-variance}. More specifically, we employ the sequence $\mathrm{seq}_{\mathbf{x},\mathbf{y}}=[\mathbf{x}, \mathbf{y}]$ to serve as the input. We run the LLM on this sequence and obtain a representation from the top-most Transformer layer. Then, we take the representation at the last position of the output and map it to a scalar via linear transformation\footnote{In practice, when employing an LLM for training a reward model, we utilize it primarily as an encoder to encode the sequence $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$ into a representation. Here, the representation from the last position is selected as it encapsulates the semantic information of the entire sequence.}:
\begin{eqnarray}
...
...
@@ -378,7 +378,7 @@ Although the RL process introduced above seems highly promising, as it can effec
\item\vspace{0.5cm} RL is computationally expensive for training LLMs. This arises from two primary factors. One is that during the RL process, we need to perform sampling in an autoregressive mode \citep{xiao-etal:2023introduction}. This is highly computationally intensive because the policy model usually has a large number of parameters. Furthermore, during the PPO-based optimization, we need to load four LLMs simultaneously, which requires significantly more GPU memory compared to the SFT. To address these challenges, there has been significant interest in developing efficient RL strategies, such as dynamic sampling \citep{wang-etal:2024esrl} and rule-based rewards \citep{shao-etal:2024deepseekmath}, which can maintain state-of-the-art performance without requiring high computational and time costs.
\item\vspace{0.5cm} RL is often an unstable produce. While RL provides a strong theoretical foundation for the design of each component, it is also highly sensitive to changes in any part of the system. Even small adjustments to the reward model, policy model, or hyperparameters can lead to significant fluctuations in performance, making it difficult to ensure consistent and stable results. This fragility highlights the importance of carefully tuning and stabilizing the various elements involved in RL, especially when applied to complex systems like LLMs. Techniques such as reward shaping, adaptive learning rate, and more sophisticated optimization strategies are often required to mitigate instability and improve the robustness of the learning process.
\item\vspace{0.5cm} RL is often an unstable process. While RL provides a strong theoretical foundation for the design of each component, it is also highly sensitive to changes in any part of the system. Even small adjustments to the reward model, policy model, or hyperparameters can lead to significant fluctuations in performance, making it difficult to ensure consistent and stable results. This fragility highlights the importance of carefully tuning and stabilizing the various elements involved in RL, especially when applied to complex systems like LLMs. Techniques such as reward shaping, adaptive learning rate, and more sophisticated optimization strategies are often required to mitigate instability and improve the robustness of the learning process.
In the previous section, we introduced some improvements to RL, such as importance sampling and reward baseline techniques. However, directly applying them to train LLMs still presents numerous challenges. In this section, we will delve deeper into the improvements for using RL to train LLMs.
\subsection{Advanced Reward Models}
Reward models are a fundamental component in RL, as they define what a policy model optimizes for. Therefore, the quality of the reward model training directly influences the effectiveness of the LLM optimization during RLHF. A poorly trained reward model can lead to suboptimal LLM, while a well-optimized reward model ensures that the LLM is effectively aligned with the desired behaviours and objectives. Here we will introduce several methods for obtaining advanced reward models. Our discussion will be relatively general, and since the reward model is widely used in many RL problems, such as robot planning and control. This broad applicability makes it straightforward to adapt the methods discussed here to other related applications.
Reward models are a fundamental component in RL, as they define what a policy model optimizes for. Therefore, the quality of reward model training directly influences the effectiveness of LLM optimization during RLHF. A poorly trained reward model can lead to a suboptimal LLM, while a well-optimized reward model ensures that the LLM is effectively aligned with the desired behaviors and objectives. Here we will introduce several methods for obtaining advanced reward models. Our discussion will be relatively general, since reward models are widely used in many RL problems, such as robot planning and control. This broad applicability makes it straightforward to adapt the methods discussed here to other related applications.
\subsubsection{Automatic Preference Data Generation}
\label{sec:automatic-preference-data-generation}
Although learning from human preferences is an effective and popular method for aligning LLMs, annotating preference data is costly. Relying on human feedback not only faces scalability limitations but may also introduce bias, as human feedback is inherently subjective. Consequently, AI-based feedback methods offer a promising solution to address these scalability and consistency issues, avoiding the limitations associated with human annotators.
One simple method is to generate preference data using LLM. Given a set of inputs, we first use an LLM to generate pairs of outputs. Then, we prompt the LLM to label the preference between each pair of outputs, along with its corresponding input. Below is an example of prompting the LLM to generate a preference label for a pair of outputs.
One simple method is to generate preference data using an LLM. Given a set of inputs, we first use an LLM to generate pairs of outputs. Then, we prompt the LLM to label the preference between each pair of outputs, along with its corresponding input. Below is an example of prompting the LLM to generate a preference label for a pair of outputs.
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
...
...
@@ -61,7 +61,7 @@ Output B:
\end{tcolorbox}
\vspace{0.5em}
Once we collect such preference labels, we can use them, along with the output pair and input, to train the reward model. Of course, these labels are not entirely accurate either. Recent studies have shown that AI-based feedback often exhibits a location bias problem, making it more likely to prefer the output at the front \citep{zheng-etal:2023judging}. We can consider demonstrating a few examples or using advanced prompting techniques, such as Chain-of-Thought (CoT), to improve the labeling performance \citep{liu-etal:2023GEval}.
Once we collect such preference labels, we can use them, along with the output pair and input, to train the reward model. Of course, these labels are not entirely accurate either. Recent studies have shown that AI-based feedback often exhibits a position bias problem, making it more likely to prefer the output shown first \citep{zheng-etal:2023judging}. We can consider demonstrating a few examples or using advanced prompting techniques, such as Chain-of-Thought (CoT), to improve the labeling performance \citep{liu-etal:2023GEval}.
For data generation, although it is easy to scale up, it is often necessary to ensure the data is accurate and diverse. Here, the data quality and diversity issues involve not only the labeling of preferences but also the inputs and outputs of the model. Therefore, we often need to use a variety of techniques to obtain large-scale, high-quality data. For example, one can generate diverse model outputs and annotations by using different LLMs, prompts, in-context demonstrations, and so on \citep{cui-etal:2024ultra}. Furthermore, \citet{dubois-etal:2024alpacafarm} report that the variability in pairwise preference data is important for training LLMs from either human or AI feedback.
...
...
@@ -80,9 +80,9 @@ While learning from AI feedback is highly scalable and generally objective, this
\subsubsection{Reward Shaping}
\label{sec:reward-shaping}
As discussed in Section \ref{sec:training-reward-models}, while the reward model is effective in capturing human preferences, it provides sparse rewards. These rewards, known as delayed rewards, are only at the end of the output generation process, as opposed to intermediate rewards that would be distributed continuously throughout it.
As discussed in Section \ref{sec:training-reward-models}, while the reward model is effective in capturing human preferences, it provides sparse rewards. These rewards, known as delayed rewards, are received only at the end of the output generation process, as opposed to intermediate rewards that would be distributed continuously throughout it.
In fact, dealing with sparse rewards has long been a concern in RL, and has been one of the challenges in many practical applications. For example, robotics often needs to shape the reward function to ease optimization rather than relying solely on end-of-sequence rewards. Various methods have been developed to address this issue. One common approach is reward shaping, where the original function is modified to include intermediate rewards, thereby providing more immediate feedback. Here, the intermediate reward is often an indirect way to be able to improve this delayed reward. For example, as shown in Figure \ref{fig:example-shaping-rewards}, setting a length-based reward as an intermediate reward encourages the generation of more content, potentially increasing the overall quality of the final output as assessed by the delayed reward from the reward model. Additional examples of reward shaping in training LLMs can be found in \citet{kumar-etal:2024training}. In this way, we can obtain
In fact, dealing with sparse rewards has long been a concern in RL, and has been one of the challenges in many practical applications. For example, robotics often needs to shape the reward function to ease optimization rather than relying solely on end-of-sequence rewards. Various methods have been developed to address this issue. One common approach is reward shaping, where the original function is modified to include intermediate rewards, thereby providing more immediate feedback. Here, the intermediate reward is often an indirect signal for improving the delayed reward. For example, as shown in Figure \ref{fig:example-shaping-rewards}, setting a length-based reward as an intermediate reward encourages the generation of more content, potentially increasing the overall quality of the final output as assessed by the delayed reward from the reward model. Additional examples of reward shaping in training LLMs can be found in \citet{kumar-etal:2024training}. In this way, we can obtain
@@ -125,9 +125,9 @@ In addition to reward shaping, another method to address the sparse reward issue
As discussed in Section \ref{sec:training-reward-models}, reward models are trained using preference data and subsequently used to optimize LLMs. However, a problem arises: the data distribution during LLM optimization may differ from the distribution of the preference data, making it challenging for the reward model to generalize to unseen input-output pairs.
A well-known failure mode associated with this problem is commonly referred to as \textit{overoptimization} or \textit{reward hacking}, where the optimization stage improves the reward model but deteriorates the alignment with true rewards \citep{gao-etal:2023scaling,eisenstein-etal:2023helping}. This mode occurs because the reward model may incorrectly assign high rewards to unseen input-output pairs, leading the LLM to learn and optimize for behaviours that do not truly align with the desired behaviours and objectives. For example, consider a scenario from Section \ref{sec:policy-gradient} where the reward model is designed to favor informative and accurate outputs for a homework assistant. However, if the reward model fails to generalize to an overly verbose output and assigns a high reward to it (perhaps due to its length or certain keywords), the LLM may learn to prioritize generating a long output, which is not actually more informative but reward better according to the reward model. Consequently, the LLM becomes misaligned with its true objectives—delivering concise and relevant information—because it optimizes for the incorrect reward signal from the reward model with weak generalization.
A well-known failure mode associated with this problem is commonly referred to as \textit{overoptimization} or \textit{reward hacking}, where the optimization stage improves the reward model score but deteriorates the alignment with true rewards \citep{gao-etal:2023scaling,eisenstein-etal:2023helping}. This mode occurs because the reward model may incorrectly assign high rewards to unseen input-output pairs, leading the LLM to learn and optimize for behaviors that do not truly align with the desired behaviors and objectives. For example, consider a scenario from Section \ref{sec:policy-gradient} where the reward model is designed to favor informative and accurate outputs for a homework assistant. However, if the reward model fails to generalize to an overly verbose output and assigns a high reward to it (perhaps due to its length or certain keywords), the LLM may learn to prioritize generating a long output, which is not actually more informative but receives a higher reward according to the reward model. Consequently, the LLM becomes misaligned with its true objectives, delivering concise and relevant information, because it optimizes for the incorrect reward signal from the reward model with weak generalization.
Addressing this generalization problem is challenging, and no mature solution exists yet. The ideal approach would be to develop an oracle reward model that perfectly captures the true objectives of the task and generalizes across all input-output pairs during LLM optimization. However, creating such a model is extremely difficult due to the complexity of the real-world environment, as well as the challenge of collecting sufficient preference data. Instead, a more practical approach is to combine multiple reward models, improving generalization and proving more correct rewards \citep{coste-etal:2024reward}.
Addressing this generalization problem is challenging, and no mature solution exists yet. The ideal approach would be to develop an oracle reward model that perfectly captures the true objectives of the task and generalizes across all input-output pairs during LLM optimization. However, creating such a model is extremely difficult due to the complexity of the real-world environment, as well as the challenge of collecting sufficient preference data. Instead, a more practical approach is to combine multiple reward models, improving generalization and providing more accurate rewards \citep{coste-etal:2024reward}.
Given a set of reward models, combining them is straightforward, and in some cases, we can simply treat this problem as an ensemble learning problem. A simple yet common approach is to average the outputs of these models to obtain a more precise reward estimation:
\begin{eqnarray}
...
...
@@ -141,7 +141,7 @@ On the other hand, to improve the generalization of reward models, it is importa
where $\alpha$ is a balancing factor, we further illustrate the freezing parameter and regularization methods in Figure \ref{fig:improve-reward-generalization}. Note that the regularization term applies only to the LLM. That is, when optimizing this term, only the parameters of the LLM are updated, while the parameters of the reward linear map remain fixed. Therefore, in this equation, the parameter associated with the regularization term is $\theta$, which specifically refers to the parameters of the LLM.
where $\alpha$ is a balancing factor. We further illustrate the parameter freezing and regularization methods in Figure \ref{fig:improve-reward-generalization}. Note that the regularization term applies only to the LLM. That is, when optimizing this term, only the parameters of the LLM are updated, while the parameters of the reward linear map remain fixed. Therefore, in this equation, the parameter associated with the regularization term is $\theta$, which specifically refers to the parameters of the LLM.
\begin{figure}[!t]
\centering
...
...
@@ -152,7 +152,7 @@ where $\alpha$ is a balancing factor, we further illustrate the freezing paramet
\subsubsection{Generative Reward Models}
\label{sec:generative-reward-models}
Reward models are typically trained as discriminative models to assign numerical rewards to outputs and classify them as preferred or dispreferred. However, this method does not leverage the text-generation capabilities for which LLMs are fundamentally designed \citep{zhang-etal:zhang2024generative,wang-etal:wang2025gram}. For example, the discriminative reward model cannot perform CoT reasoning. To address this, an LLM can alternatively be employed as a reward model, thus endowing it with the ability to engage in text generation and reasoning, as depicted in Figure \ref{fig:generative-reward-model-architecture}. This model works as follows. First, we input a prompt $\mathbf{c}$, along with the tuple $(\mathbf{x},\mathbf{y}_{a},\mathbf{y}_{b})$, to the LLM. The prompt is a description of the task, as demonstrated in the example below.
Reward models are typically trained as discriminative models to assign numerical rewards to outputs and classify them as preferred or dispreferred. However, this method does not leverage the text-generation capabilities for which LLMs are fundamentally designed \citep{zhang-etal:zhang2024generative,wang-etal:wang2025gram}. For example, the discriminative reward model cannot perform CoT reasoning. To address this, an LLM can alternatively be employed as a reward model, thus endowing it with the ability to engage in text generation and reasoning, as depicted in Figure \ref{fig:generative-reward-model-architecture}. This model works as follows. First, we input a prompt $\mathbf{c}$, along with the tuple $(\mathbf{x},\mathbf{y}_{a},\mathbf{y}_{b})$, to the LLM. The prompt is a description of the task, as demonstrated in the example below.
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
...
...
@@ -168,11 +168,11 @@ You are given two outputs to an input. Evaluate which output is better based on
\end{tcolorbox}
\vspace{0.5em}
Then, the LLM predicts subsequent tokens based on this input sequence. Let $w$ be the label token predicted by the LLM. If $w=\text{A}$, it indicates a preference for $\mathbf{y}_a$ over $\mathbf{y}_b$; if $w=\text{B}$, then $\mathbf{y}_b$ is preferred. Note that here $\mathbf{y}_a$ and $\mathbf{y}_b$ do not have a pre-defined preference relationship as described in Section \ref{sec:training-reward-models}. Their relationship is instead represented by the label token.
Then, the LLM predicts subsequent tokens based on this input sequence. Let $w$ be the label token predicted by the LLM and $w^{*}$ be the annotated preference label. If $w^{*}=\text{A}$, it indicates a preference for $\mathbf{y}_a$ over $\mathbf{y}_b$; if $w^{*}=\text{B}$, then $\mathbf{y}_b$ is preferred. Note that here $\mathbf{y}_a$ and $\mathbf{y}_b$ do not have a pre-defined preference relationship as described in Section \ref{sec:training-reward-models}. Their relationship is instead represented by the label token.
The loss function can be defined as the log-probability of predicting `A':
The loss function can be defined as the log-probability of predicting the annotated preference label:
@@ -180,7 +180,7 @@ where $\mathbf{s}$ denotes the string $[\mathbf{c},\mathbf{x},\mathbf{y}_a,\math
Although we discuss methods for training a generative reward model here, an interesting question arises: how can we use the generative reward model in RLHF? We provide some guidance as follows. Specifically, when applying this generative reward model to provide a reward for an input-output pair $(\mathbf{x}',\mathbf{y}')$, we can generate a reference output $\mathbf{y}_{\mathrm{ref}}$ by using the LLM, for example, through greedy search, and concatenate $\mathbf{x}'$, $\mathbf{y}'$ and $\mathbf{y}_{\mathrm{ref}}$ into $\mathbf{s}' =[\mathbf{c}, \mathbf{x}', \mathbf{y}', \mathbf{y}_{\mathrm{ref}}]$.
Additionally, to mitigate the positional bias problem \citep{wang-etal:2023large}, we can introduce an alternative input order by transposing the positions of output, i.e., presenting $\mathbf{y}_{\mathrm{ref}}$ before $\mathbf{y}'$, to construct a secondary input string $\mathbf{s}'_{T}=[\mathbf{c}, \mathbf{x}', \mathbf{y}_{\mathrm{ref}}, \mathbf{y}']$.
The reward for $(\mathbf{x}', \mathbf{y}')$ is thus defined as the log-probability that $\mathbf{y}'$ is preferred over $\mathbf{y}_{\mathrm{ref}}$:
The reward for $(\mathbf{x}', \mathbf{y}')$ is thus defined as the probability that $\mathbf{y}'$ is preferred over $\mathbf{y}_{\mathrm{ref}}$:
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (also known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches. The basic idea is to use only the ranking capability of generative reward models to derive rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
...
...
@@ -374,12 +374,12 @@ It is worth noting that reasoning can also be incorporated into rubric-based eva
The quality of the rubric is crucial to the performance of rubric-based reward models. As a result, recent work has devoted increasing attention to constructing high-quality rubrics for reward modeling. Similar to many other components in machine learning, rubric acquisition generally follows two approaches: manual design and automatic generation.
For manual design, one straightforward approach is to recruit human experts to write rubrics based on their domain knowledge and task requirements. However, this way is often difficult to design a rubric that comprehensively covers the diverse cases that may arise across different inputs and tasks. This is because that evaluation criteria that are appropriate for one case may be insufficient for another. Additionally, manually designed rubrics inherit the prompt sensitivity of LLM-based evaluation: even when the underlying evaluation dimensions remain the same, small differences in wording can lead to noticeably different results.
For manual design, one straightforward approach is to recruit human experts to write rubrics based on their domain knowledge and task requirements. However, it is often difficult to design a rubric that comprehensively covers the diverse cases that may arise across different inputs and tasks. This is because evaluation criteria that are appropriate for one case may be insufficient for another. Additionally, manually designed rubrics inherit the prompt sensitivity of LLM-based evaluation: even when the underlying evaluation dimensions remain the same, small differences in wording can lead to noticeably different results.
One promising strategy is to generate rubrics dynamically based on the input. As shown in Figure~\ref{fig:rubric-based-reward-modeling}, existing approaches generally fall into two categories. The first uses an external LLM to generate task-specific rubrics from the input and evaluation requirements, which are then provided to the reward model for preference prediction. The second allows the generative reward model to generate its own rubrics for the current input and then evaluate candidate responses according to these self-generated criteria.
Both approaches formulate rubric generation as an explicit task. This naturally leads to another idea: can we optimize the rubric generation process itself to produce better rubrics? The answer is yes. Rubrics generated directly by LLMs may suffer from limited coverage, redundant criteria, or preference misalignment \citep{liu-etal:openrubrics,shen-etal:rethinking-rubric}. To improve rubric quality, we can explicitly refine the generated criteria. For example, OpenRubrics generates rubrics by contrasting preferred and rejected responses to identify more discriminative rules and principles \citep{liu-etal:openrubrics}. We can further learn the rubric generation process itself through RL training \citep{xu-etal:rubric-arm}. More recent studies go one step further by allowing rubrics to evolve with the policy, so that the evaluation criteria continue to capture new weaknesses as model improves \citep{rezaei-etal:online-rubrics,ding-etal:evorubrics,yu-etal:audio-rubrics}. Rubric optimization is becoming an active research direction, and interested readers can refer to the aforementioned works for further details.
Both approaches formulate rubric generation as an explicit task. This naturally leads to another idea: can we optimize the rubric generation process itself to produce better rubrics? The answer is yes. Rubrics generated directly by LLMs may suffer from limited coverage, redundant criteria, or preference misalignment \citep{liu-etal:openrubrics,shen-etal:rethinking-rubric}. To improve rubric quality, we can explicitly refine the generated criteria. For example, OpenRubrics generates rubrics by contrasting preferred and rejected responses to identify more discriminative rules and principles \citep{liu-etal:openrubrics}. We can further learn the rubric generation process itself through RL training \citep{xu-etal:rubric-arm}. More recent studies go one step further by allowing rubrics to evolve with the policy, so that the evaluation criteria continue to capture new weaknesses as the model improves \citep{rezaei-etal:online-rubrics,ding-etal:evorubrics,yu-etal:audio-rubrics}. Rubric optimization is becoming an active research direction, and interested readers can refer to the aforementioned works for further details.
\subsubsection{Reward Model Evaluation}
...
...
@@ -466,13 +466,13 @@ While the Monte Carlo-based estimation of advantage provides a relatively stable
\centering
\input{section4/Figures/figure-issue-monte-carlo}
\caption{
Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq. (\ref{eq:langth-based-reward-function}).
Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq. (\ref{eq:length-based-reward-function}).
Typically, if the value model is well-trained, it would predict an average $V_{t}$ of 6 at time step $t$, based on most outputs having around 60 tokens in length. However, for outlier outputs like $\mathbf{y}_{d,t:T}$, which might extend up to 300 tokens, the advantage estimation can exhibit high variance. For instance, the advantage for such an outlier could be computed as $A_{t}(\mathbf{x},\mathbf{y}_{d,<t},y_{d,t})=\sum_{k=t}^T r_{k}- V_t =30-6=24$, a stark contrast to another output, say $\mathbf{y}_{1,t:T}$, which is closer to the average length, where the advantage might be $A_{t}(\mathbf{x},\mathbf{y}_{1,<t},y_{1,t})=-1$. This discrepancy leads to high gradient variance, potentially destabilizing the learning process.
}
\label{fig:monte-carlo-advantage-issue}
\end{figure*}
To address these challenges, an alternative approach is the temporal difference-based advantage estimation \citep{sutton-and-richard:1988learning}, which allows for estimating the advantage using incomplete outputs. More specifically, this method computes the difference between the predicted values at consecutive time steps (i.e., current and next stapes), adjusting the advantage estimate based on new information as it becomes available, thus reducing the dependency on the entire output. In a practical implementation, the temporal difference-based advantage estimation can be given by
To address these challenges, an alternative approach is the temporal difference-based advantage estimation \citep{sutton-and-richard:1988learning}, which allows for estimating the advantage using incomplete outputs. More specifically, this method computes the difference between the predicted values at consecutive time steps (i.e., current and next steps), adjusting the advantage estimate based on new information as it becomes available, thus reducing the dependency on the entire output. In a practical implementation, the temporal difference-based advantage estimation can be given by
By focusing on the current and next steps, temporal difference-based advantage estimation minimizes the influence of distant future events. This leads to lower variance in optimizing the policy model and improves the stability of policy training.
\subsubsection{Generalized Advantage Estimation}
While temporal difference-based estimation effectively reduces variance, it can also increase estimation bias due to its heavy reliance on the predictions of the value model. By contrast, one effective method is generalized advantage estimation (GAE), which is one of the most popular advantage estimation methods used in LLMs and other fields \citep{schulman-etal:2015high}. This method mainly refines the advantage estimation by using exponentially weighted averages of temporal difference advantages across multiple future steps. More specifically, consider a bias $\delta_t$ in the temporal difference at time step $t$, defined as
While temporal difference-based estimation effectively reduces variance, it can also increase estimation bias due to its heavy reliance on the predictions of the value model. By contrast, one effective method is generalized advantage estimation (GAE), which is one of the most popular advantage estimation methods used in LLMs and other fields \citep{schulman-etal:2015high}. This method mainly refines the advantage estimation by using exponentially weighted averages of temporal-difference residuals across multiple future steps. More specifically, consider the temporal-difference residual $\delta_t$ at time step $t$, defined as
\begin{eqnarray}
\delta_t & = & r_t + \gamma V_{t+1} - V_{t}
\end{eqnarray}
GAE introduces parameters that dynamically balance the trade-off between bias and variance. This basic idea is to use the weighted sum of multi-step temporal difference bias as an estimation of the advantage, combining the advantages of Monte Carlo (high variance and low bias) and temporal difference methods (low variance and high bias). This combination can be expressed as
GAE introduces parameters that dynamically balance the trade-off between bias and variance. This basic idea is to use the weighted sum of multi-step temporal-difference residuals as an estimation of the advantage, combining the advantages of Monte Carlo (high variance and low bias) and temporal difference methods (low variance and high bias). This combination can be expressed as
@@ -518,7 +518,7 @@ In fact, this is entirely feasible, and one example of such an approach is \cite
\label{eq:advantage-grpo}
\end{eqnarray}
where the $\mathrm{Mean}(\cdot)$ and $\mathrm{Std}(\cdot)$ represent the mean and standard deviation functions, respectively. Note that the advantage computation used here differs slightly from that in PPO. In this case, this computed advantage is applied to each time step, whereas in the traditional PPO, we separately compute an advantage for each time step. At this point, we can also use the process reward model to supplement the advantage computation, allowing us to compute an advantage specific to each time step. A simple way to achieve this is by implementing Eq. (\ref{eq:advantage-grpo}) at each time step, where the rewards are computed using the process reward model. As a result, the objective for GRPO can be defined according to Eq. (\ref{eq:ppo-loss}):
where $\mathrm{Mean}(\cdot)$ and $\mathrm{Std}(\cdot)$ represent the mean and standard deviation functions, respectively. Note that the advantage computation used here differs slightly from that in PPO. In this case, this computed advantage is applied to each time step, whereas in the traditional PPO, we separately compute an advantage for each time step. At this point, we can also use the process reward model to supplement the advantage computation, allowing us to compute an advantage specific to each time step. A simple way to achieve this is by implementing Eq. (\ref{eq:advantage-grpo}) at each time step, where the rewards are computed using the process reward model. As a result, the objective for GRPO can be defined according to Eq. (\ref{eq:ppo-loss}):
@@ -572,7 +572,7 @@ Similarly, in mathematical reasoning tasks, where the desired outcome is a corre
In addition to improving computational efficiency, another benefit of using rule-based rewards is their stability. Compared to reward models, rule-based rewards are less prone to issues such as reward hacking. As the rules are predefined and not subject to the same complex training process as traditional models, the risk of misalignment between the optimization objectives of the policy model and the oracle rewards is reduced. This stability makes rule-based rewards a more predictable and reliable approach, ensuring that the learning process of the policy model is less influenced by prediction errors or biases that may arise from the reward model. Furthermore, recent research has shown that employing rule-based rewards can significantly enhance the reasoning capabilities of LLMs through RL, underscoring its practical effectiveness \citep{guo:2025deepseek,xie-etal:2025logic}.
Another lightweight reward method is to consider achieving a lightweight reward model with fewer parameters, which would reduce both computational time and resource usage. There are several methods to achieve this. For example, given that reward models fundamentally utilize an LLM, many efficient methods, such as Mixture of Experts (MoE) \citep{masoudnia-etal:2014mixture} and model pruning \citep{sun-etal:2023simple}, originally developed for LLMs can be adapted to the reward model to improve its efficiency. Furthermore, knowledge distillation techniques offer a pathway to construct a smaller reward model that learns to emulate the behaviour of a larger one \citep{wang-etal:2023learning}. These approaches not only improve the efficiency of the reward model but also maintain its ability to deliver accurate rewards.
Another lightweight reward method is to use a reward model with fewer parameters, which would reduce both computational time and resource usage. There are several methods to achieve this. For example, given that reward models fundamentally utilize an LLM, many efficient methods, such as Mixture of Experts (MoE) \citep{masoudnia-etal:2014mixture} and model pruning \citep{sun-etal:2023simple}, originally developed for LLMs can be adapted to the reward model to improve its efficiency. Furthermore, knowledge distillation techniques offer a pathway to construct a smaller reward model that learns to emulate the behavior of a larger one \citep{wang-etal:2023learning}. These approaches not only improve the efficiency of the reward model but also maintain its ability to deliver accurate rewards.
\subsection{Direct Preference Optimization}
...
...
@@ -581,10 +581,10 @@ Another lightweight reward method is to consider achieving a lightweight reward
\centering
\input{section4/Figures/figure-rlhf-vs-dpo}
\caption{Standard PPO vs. DPO in training LLMs. In PPO, the human preference data is used to train a reward model, which is then employed to train the policy and the value function. In DPO, the use of human preference data is more direct, and the policy is trained on this data without the need for reward model training.}
\label{fig:comparsion-rl-dpo}
\label{fig:comparison-rl-dpo}
\end{figure*}
Although learning reward models is a standard step in RL, it makes the entire training process much more complex than supervised training. Training a reliable reward model is itself not an easy task and a poorly trained reward model can greatly affect the outcome of policy learning. We now consider an alternative method, called direct preference optimization (DPO), which simplifies the training framework by eliminating the need to explicitly model rewards \citep{rafailov:2023direct}. This method directly optimizes the policy model based on user preferences rather than developing a separate reward model. As a result, we can achieve human preference alignment in a supervised learning-like fashion. Figure \ref{fig:comparsion-rl-dpo} shows a comparison of the standard PPO method and the DPO method.
Although learning reward models is a standard step in RL, it makes the entire training process much more complex than supervised training. Training a reliable reward model is itself not an easy task and a poorly trained reward model can greatly affect the outcome of policy learning. We now consider an alternative method, called direct preference optimization (DPO), which simplifies the training framework by eliminating the need to explicitly model rewards \citep{rafailov:2023direct}. This method directly optimizes the policy model based on user preferences rather than developing a separate reward model. As a result, we can achieve human preference alignment in a supervised learning-like fashion. Figure \ref{fig:comparison-rl-dpo} shows a comparison of the standard PPO method and the DPO method.
DPO simplifies the RL process for LLMs by utilizing a straightforward cross-entropy loss, which streamlines the learning process and enhances its stability. Rather than delving into the detailed derivation of the DPO loss function, we will discuss its optimization objective from the perspective of optimizing an LLM. The loss function derived from a Bradley-Terry reward model can be given by
\begin{eqnarray}
...
...
@@ -606,7 +606,7 @@ This loss function utilizes the implicit reward for DPO training, which replaces
\label{fig:understand-dpo-optimization-objective}
\end{figure*}
Here, we can consider that the DPO directly models human feedback through the relative likelihood of outputs under the policy and reference models. This method provides a more straightforward approach than traditional methods that use a separate reward model and then apply RL algorithms like PPO to adjust the probabilities of sampled outputs. In this way, we can further understand the optimization objective of the DPO by leveraging the hypothesis space as mentioned in Section \ref{sec:policy-gradient}. In practice, as illustrated in Figure \ref{fig:understand-dpo-optimization-objective}, we can see the DPO as reranking in the hypothesis space through the Bradley-Terry model approach. Specifically, this method increases the probabilities of preferred outputs in preference data while decreasing those of dispreferred ones. Similar to TRPO, the DPO also incorporates a penalty derived from the reference model, which ensures that updates remain within a trusted behaviour space for the policy model.
Here, we can consider that DPO directly models human feedback through the relative likelihood of outputs under the policy and reference models. This method provides a more straightforward approach than traditional methods that use a separate reward model and then apply RL algorithms like PPO to adjust the probabilities of sampled outputs. In this way, we can further understand the optimization objective of DPO by leveraging the hypothesis space as mentioned in Section \ref{sec:policy-gradient}. In practice, as illustrated in Figure \ref{fig:understand-dpo-optimization-objective}, we can see DPO as reranking in the hypothesis space through the Bradley-Terry model approach. Specifically, this method increases the probabilities of preferred outputs in preference data while decreasing those of dispreferred ones. Similar to TRPO, DPO also incorporates a penalty derived from the reference model, which ensures that updates remain within a trusted behavior space for the policy model.
However, there are two sides to every coin. While DPO simplifies the RL process, it also introduces limitations. Notably, since DPO eliminates the exploration phase during training, its potential peak performance is generally considered lower compared to RL-based methods like PPO, which incorporate exploration to discover more effective policies. In other words, the performance of DPO is inherently bounded by the labeled preferred outputs in the preference data. To address this limitation, current approaches focus on ensuring high-quality preferred outputs, either by employing advanced LLMs like GPT-4 or through human labeling \citep{cui-etal:2023ultrafeedback,morimura-etal:2024filtered}.
...
...
@@ -615,8 +615,8 @@ However, there are two sides to every coin. While DPO simplifies the RL process,
\resizebox{\textwidth}{!}{
\input{section4/Tables/dpo_variants}}
\caption{DPO variants and their optimization objectives.}
\label{tab:dpo_varients}
\label{tab:dpo_variants}
\end{table*}
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table \ref{tab:dpo_varients}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table \ref{tab:dpo_variants}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
\section{Reinforcement Learning for LLM Reasoning}
So far, our discussion has mainly focused on various aspects of using and improving RL for training LLMs. The methods mentioned can be easily adapted to a wild of scenarios where the correctness of an output can be examined by checking whether the desired result is included. For example, in the task of calculating a mathematical expression, a reward model can provide positive feedback if the answer is correct and negative feedback if the answer is wrong. However, in many problems that require complex reasoning, simply examining the correctness of the final answer is insufficient for learning. Imagine a student who is only given the final answer to a challenging math problem. Knowing whether the final answer is right or wrong does not help the student figure out where they went wrong and how to calculate the correct answer. A better approach would be to guide the student with a step-by-step breakdown of the problem-solving process and encourage understanding of the underlying concepts and logic behind these steps. To address this, researchers also explore the potential of RL to teach LLMs not just to generate correct answers but to develop and present coherent, step-by-step reasoning that aligns with human cognitive processes. In this regard, RL has previously demonstrated its effectiveness in training neural networks for complex planning and reasoning within game environments, as evidenced by notable successes such as AlphaGo \citep{silver-etal:2016mastering} and AlphaStar \citep{vinyals-rtal:2019grandmaster}. Given these advancements and the inherently interactive nature of problem-solving, it is also natural to consider the application of RL to LLM reasoning.
So far, our discussion has mainly focused on various aspects of using and improving RL for training LLMs. The methods mentioned can be easily adapted to a wide range of scenarios where the correctness of an output can be examined by checking whether the desired result is included. For example, in the task of calculating a mathematical expression, a reward model can provide positive feedback if the answer is correct and negative feedback if the answer is wrong. However, in many problems that require complex reasoning, simply examining the correctness of the final answer is insufficient for learning. Imagine a student who is only given the final answer to a challenging math problem. Knowing whether the final answer is right or wrong does not help the student figure out where they went wrong and how to calculate the correct answer. A better approach would be to guide the student with a step-by-step breakdown of the problem-solving process and encourage understanding of the underlying concepts and logic behind these steps. To address this, researchers also explore the potential of RL to teach LLMs not just to generate correct answers but to develop and present coherent, step-by-step reasoning that aligns with human cognitive processes. In this regard, RL has previously demonstrated its effectiveness in training neural networks for complex planning and reasoning within game environments, as evidenced by notable successes such as AlphaGo \citep{silver-etal:2016mastering} and AlphaStar \citep{vinyals-rtal:2019grandmaster}. Given these advancements and the inherently interactive nature of problem-solving, it is also natural to consider the application of RL to LLM reasoning.
In this section, we delve deeper into the application of RL to enhance the reasoning capabilities of the LLM, a topic that has recently garnered significant attention. We begin by giving a general introduction to test-time scaling that fully unleashes the reasoning potential of LLMs, including best-of-$N$ sampling, step-by-step verification, and Monte Carlo Tree Search. We then discuss two critical issues: how to scale RL effectively, and how to iterate the RL process to enhance the reasoning capabilities of LLMs. Note that the following methods are primarily illustrated through the mathematical reasoning problem, they are applicable to a broad range of decision-making problems.
In this section, we delve deeper into the application of RL to enhance the reasoning capabilities of the LLM, a topic that has recently garnered significant attention. We begin by giving a general introduction to test-time scaling that fully unleashes the reasoning potential of LLMs, including best-of-$N$ sampling, step-by-step verification, and Monte Carlo Tree Search. We then discuss two critical issues: how to scale RL effectively, and how to iterate the RL process to enhance the reasoning capabilities of LLMs. Note that the following methods are primarily illustrated through mathematical reasoning problems, but they are applicable to a broad range of decision-making problems.
\subsection{Test-time Scaling}
\label{sec:tts}
Initially, we review the fundamental objective of RL: to maximize the rewards obtained during output generation. This goal has been extensively achieved through various training-time optimization techniques, such as policy gradient or PPO algorithms. Beyond training, recent research has illuminated the benefits of scaling up test-time computation, a process also known as test-time scaling, which can significantly enhance the maximization of rewards, especially in the reasoning task. In a practical implementation, one simple approach to achieve test-time scaling is the use of prompting techniques. For example, appending the phrase ``Let's think step-by-step.'' to the input can effectively stimulate the LLM to engage in a more detailed reasoning process during the generation. While this approach can improve reasoning accuracy, it often leads to suboptimal performance because the model is not inherently trained to understand the most beneficial reasoning processes. To address this issue, we can perform the test-time scaling with guidance from a reward model, as discussed in the following subsections.
Initially, we review the fundamental objective of RL: to maximize the rewards obtained during output generation. This goal has been extensively pursued through various training-time optimization techniques, such as policy gradient or PPO algorithms. Beyond training, recent research has illuminated the benefits of scaling up test-time computation, a process also known as test-time scaling, which can significantly enhance reward maximization, especially in reasoning tasks. In a practical implementation, one simple approach to achieve test-time scaling is the use of prompting techniques. For example, appending the phrase ``Let's think step-by-step.'' to the input can effectively stimulate the LLM to engage in a more detailed reasoning process during generation. While this approach can improve reasoning accuracy, it often leads to suboptimal performance because the model is not inherently trained to understand the most beneficial reasoning processes. To address this issue, we can perform test-time scaling with guidance from a reward model, as discussed in the following subsections.
\subsubsection{Best-of-N Sampling}
\label{sec:BoN-sampling}
...
...
@@ -23,10 +23,10 @@ One approach to test-time scaling using a reward model involves sampling multipl
As illustrated in Figure \ref{fig:bon-sampling}, in the BoN sampling, we first sample $N$ different outputs $\{\mathbf{y}_{1}, \mathbf{y}_{2}, \cdots, \mathbf{y}_{N}\}$ for the input $\mathbf{x}$:
where the $\mathrm{argTopN}$ operation returns the top-$N$ outputs that maximize the function $\Pr(\mathbf{y}|\mathbf{x})$. These outputs can be sampled in various ways, depending on the search algorithm used by the model (e.g., nucleus sampling or beam search). Once the $N$-best output candidates are sampled, the reward model is used to evaluate and select the best one:
where the outputs can be sampled in various ways, depending on the decoding algorithm used by the model (e.g., nucleus sampling or beam search). Once the $N$ output candidates are sampled, the reward model is used to evaluate and select the best one:
@@ -44,12 +44,12 @@ While BoN sampling is effective for scaling test-time computation, it is ineffic
We can collect or generate reasoning paths corresponding to problems from existing datasets to train a process reward model. Human experts then annotate each step in these paths for correctness. These annotations can be used to train LLMs or as rewards in reward modeling directly. However, in practice, richer annotations are often introduced \citep{lightman-etal:2024lets}. In addition to the \textit{correct} and \textit{incorrect} labels, a step can also be labeled as \textit{neutral} to indicate that while the step may be technically correct, it might still be problematic within the overall reasoning process. Additionally, an automatic process annotation framework can be utilized, which relies on generating multiple entire reasoning paths based on the current step and using the accuracy of these paths to serve as the quality annotation of the step \citep{wang-etal:2023math}.
Given a set of step-level annotated reasoning paths and corresponding inputs, we can train a reward model to provide a reward for each step in the reasoning process. The reward model can be treated as a classification model. So its architecture can be an LLM with a Softmax layer stacked on top, akin to the architecture depicted in Figure \ref{fig:reward-model}. Here, consider an reasoning path including $n_{s}$ steps, represented as $\mathbf{y}=\{\bar{\mathbf{y}}_{1},\cdots ,\bar{\mathbf{y}}_{n_{s}}\}$. At each step $k$, the process reward model takes both the problem description, denoted by $\mathbf{x}$, and the reasoning steps generated so far, denoted by $\bar{\mathbf{y}}$, as inputs. It then outputs a probability distribution over the set of labels ${\text{\textit{correct}}, \text{\textit{incorrect}}}$, or ${\text{\textit{correct}}, \text{\textit{incorrect}}, \text{\textit{neutral}}}$, to evaluate the reasoning at that point. This model can trained by a casual classification loss, e.g., Sigmoid Cross-entropy. Once trained, the reward model can be used to evaluate reasoning paths by assessing the correctness of each step. A simple method to use log-probabilities of classification to define the reward of each reasoning step, for example, the reward of the $k$-th reasoning step can be given by
Given a set of step-level annotated reasoning paths and corresponding inputs, we can train a reward model to provide a reward for each step in the reasoning process. The reward model can be treated as a classification model. Thus, its architecture can be an LLM with a Softmax layer stacked on top, akin to the architecture depicted in Figure \ref{fig:reward-model}. Here, consider a reasoning path including $n_{s}$ steps, represented as $\mathbf{y}=\{\bar{\mathbf{y}}_{1},\cdots ,\bar{\mathbf{y}}_{n_{s}}\}$. At each step $k$, the process reward model takes both the problem description, denoted by $\mathbf{x}$, and the reasoning steps generated so far, denoted by $\bar{\mathbf{y}}$, as inputs. It then outputs a probability distribution over the set of labels ${\text{\textit{correct}}, \text{\textit{incorrect}}}$, or ${\text{\textit{correct}}, \text{\textit{incorrect}}, \text{\textit{neutral}}}$, to evaluate the reasoning at that point. This model can be trained with a standard classification loss, e.g., cross-entropy. Once trained, the reward model can be used to evaluate reasoning paths by assessing the correctness of each step. A simple method is to use classification log-probabilities to define the reward of each reasoning step. For example, the reward of the $k$-th reasoning step can be given by
where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes the probability of the \textit{correct} label generated by the reward model. The reward score $R_{\phi}(\mathbf{x},\mathbf{y})$ can then be used to select the best step while generating a reasoning path. Additionally, as discussed in Section \ref{sec:generative-reward-models}, there is the option to train a generative process reward model that further improves this performance on evaluating the reasoning step, also called generative verifier in the literature\citep{zhang-etal:2024generative}. Note that in practice, the process reward model serves not only to provide rewards for test-time scaling but also to train the model using RL, e.g., the rewards from this model can be employed as shaping rewards.
where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes the probability of the \textit{correct} label generated by the reward model. The reward score $R_{\phi}(\mathbf{x},\mathbf{y})$ can then be used to select the best step while generating a reasoning path. Additionally, as discussed in Section \ref{sec:generative-reward-models}, there is the option to train a generative process reward model, also called a generative verifier in the literature, to further improve reasoning-step evaluation\citep{zhang-etal:2024generative}. Note that in practice, the process reward model serves not only to provide rewards for test-time scaling but also to train the model using RL, e.g., the rewards from this model can be employed as shaping rewards.
\begin{figure*}[!t]
\centering
...
...
@@ -100,7 +100,7 @@ Training LLMs with RL usually follows a two-phase approach: training a pre-train
\centering
\input{section5/Figures/figure-iterative-rl}
\caption{
Illustration of iterative RL in DeepSeek-R1 \citep{guo:2025deepseek}. This method maintains multi-phase RL to develop a robust reasoning LLM with a GRPO algorithm. Initially, a pre-trained LLM is fine-tuned using a small set of high-quality labeled reasoning data with SFT as a cold start. Then, large-scale RL is applied specifically to enhance reasoning capabilities. After that, the model undergoes reject sampling across multiple tasks to refine output quality. In the final phase, the model is further trained using RL to enhance generalization across various tasks.
Illustration of iterative RL in DeepSeek-R1 \citep{guo:2025deepseek}. This method maintains multi-phase RL to develop a robust reasoning LLM with a GRPO algorithm. Initially, a pre-trained LLM is fine-tuned using a small set of high-quality labeled reasoning data with SFT as a cold start. Then, large-scale RL is applied specifically to enhance reasoning capabilities. After that, the model undergoes rejection sampling across multiple tasks to refine output quality. In the final phase, the model is further trained using RL to enhance generalization across various tasks.
}
\label{fig:iterative-rl}
\end{figure*}
...
...
@@ -118,20 +118,20 @@ An interesting issue arises with this design of iterative RL: why is RL aimed at
Since the optimization objective of each phase is different, the design of iterative RL can also be analyzed and understood from the perspective of multi-objective optimization \citep{wang-etal:2024hybrid}. In the context of multi-objective optimization, iterative RL for LLMs can be likened to interactive methods where the solution process is iterative, and preferences are actively defined and refined by the decision-maker during the search for the most preferred solutions \citep{miettinen-etal:2008introduction,deb-etal:2016multi}. More specifically, in this scenario, RL can be seen as a decision-maker, iteratively refining and enhancing the different capabilities of the LLM across different phases.
\subsection{Large-scale Reinforcement Learning}
\label{sec:large-sclae-rl}
\label{sec:large-scale-rl}
While test-time scaling is instrumental in exploring more effective reasoning paths, its potential is restricted if the model has not learned to generate high-quality reasoning paths. In other words, in practice, test-time scaling struggles to achieve desired outcomes without foundational reasoning ability, no matter how extensive it is \citep{yuan-etal:2023scaling,zeng-etal:2025revisiting}. Consequently, there has been a growing research interest in training LLMs specifically to develop reasoning abilities that mimic human-like processes. A straightforward approach is to use annotated reasoning data to train the LLM through SFT. However, a significant challenge in this approach is the high cost of annotations, especially since annotating detailed step-by-step reasoning paths is significantly more cost-intensive than annotating SFT data in other tasks like machine translation and summarization.
Another more advanced approach is to utilize large-scale RL, transitioning from human-annotated to self-reinforced learning processes. Unlike the RL used as a fine-tuning method discussed in Section \ref{sec:example-using-rl-training-llms}, which typically involves training for a few dozen or hundreds of steps at a small scale, this approach employs it on a larger scale with rewards to break free from the limitation of supervised reasoning data. This approach has enabled the development of robust reasoning models, such as OpenAI-o1 \citep{openai:2024learning}, DeepSeek-R1 \citep{guo:2025deepseek}, and Kimi-1.5 \citep{kimi-team:2025kimi}. Notably, DeepSeek-R1-Zero was able to develop a robust reasoning model by applying large-scale RL to a pre-trained LLM without the need for any annotated reasoning data. However, despite its successes, large-scale RL is not easy and introduces unique challenges that are not present at smaller scales, as follows.
Another more advanced approach is to utilize large-scale RL, transitioning from human-annotated to self-reinforced learning processes. Unlike the RL used as a fine-tuning method discussed in Section \ref{sec:example-using-rl-training-llms}, which typically involves training for a few dozen or hundreds of steps at a small scale, this approach applies RL at a larger scale with rewards to break free from the limitation of supervised reasoning data. This approach has enabled the development of robust reasoning models, such as OpenAI-o1 \citep{openai:2024learning}, DeepSeek-R1 \citep{guo:2025deepseek}, and Kimi-1.5 \citep{kimi-team:2025kimi}. Notably, DeepSeek-R1-Zero was able to develop a robust reasoning model by applying large-scale RL to a pre-trained LLM without the need for any annotated reasoning data. However, despite its successes, large-scale RL is not easy and introduces unique challenges that are not present at smaller scales, as follows.
One is that large-scale RL requires a highly generalizable reward model. As the scale of training increases, the model is trained on broader data, necessitating a reward model capable of effectively generalizing across this varied data. On the other hand, the extensive scope of training introduces significant variability in the model, leading to considerable changes in sampling behaviours. Consequently, it is crucial to ensure that the reward model possesses robust generalization capabilities to prevent overfitting and maintain the effectiveness of the learning process. There are several methods to achieve this. For example, \citet{guo:2025deepseek} incorporated rule-based rewards, as discussed in Section \ref{sec:lightweight-reward-methods}, such as format checking and answer verification in reasoning scenarios, which can provide a stable reward throughout the learning process. This suggests that in certain RL scenarios, prioritizing rule-based rewards may be beneficial if they can effectively describe human preferences. \citet{yuan-etal:2024selfrewarding} introduced a self-rewarding framework that dynamically updates the reward model based on the currently optimized policy model so that this reward model can effectively evaluate the behaviours from the current policy model.
One is that large-scale RL requires a highly generalizable reward model. As the scale of training increases, the model is trained on broader data, necessitating a reward model capable of effectively generalizing across this varied data. On the other hand, the extensive scope of training introduces significant variability in the model, leading to considerable changes in sampling behaviors. Consequently, it is crucial to ensure that the reward model possesses robust generalization capabilities to prevent overfitting and maintain the effectiveness of the learning process. There are several methods to achieve this. For example, \citet{guo:2025deepseek} incorporated rule-based rewards, as discussed in Section \ref{sec:lightweight-reward-methods}, such as format checking and answer verification in reasoning scenarios, which can provide a stable reward throughout the learning process. This suggests that in certain RL scenarios, prioritizing rule-based rewards may be beneficial if they can effectively describe human preferences. \citet{yuan-etal:2024selfrewarding} introduced a self-rewarding framework that dynamically updates the reward model based on the currently optimized policy model so that this reward model can effectively evaluate the behaviors from the current policy model.
Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behaviour region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between keeping stability and fostering adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behavior region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between stability and adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
\subsection{On-Policy Distillation}
While large-scale RL has shown strong potential for improving model capabilities, it often relies on large amounts of verifiable training data. Several approaches have been explored to relax this requirement. A straightforward solution is to construct proxy rewards from the model's own outputs. A straightforward solution is to construct proxy rewards from the model's own outputs. For example, we can use majority voting to derive pseudo-gold answers and compute accuracy-based rewards. A more promising direction is \textit{on-policy distillation} (OPD), which introduces supervision from a stronger teacher model while preserving on-policy exploration of the student model\footnote{In standard knowledge distillation settings, we typically refer to the ``smaller'' model as the \textit{student model} and the ``larger'' model as the \textit{teacher model}.}. The key idea is to let the student model generate reasoning trajectories on-policy and then use the teacher model to provide token-level distributional supervision at the states actually visited by the student.
While large-scale RL has shown strong potential for improving model capabilities, it often relies on large amounts of verifiable training data. Several approaches have been explored to relax this requirement. A straightforward solution is to construct proxy rewards from the model's own outputs. For example, we can use majority voting to derive pseudo-gold answers and compute accuracy-based rewards. A more promising direction is \textit{on-policy distillation} (OPD), which introduces supervision from a stronger teacher model while preserving on-policy exploration of the student model\footnote{In standard knowledge distillation settings, we typically refer to the ``smaller'' model as the \textit{student model} and the ``larger'' model as the \textit{teacher model}.}. The key idea is to let the student model generate reasoning trajectories on-policy and then use the teacher model to provide token-level distributional supervision at the states actually visited by the student.
Compared with conventional offline distillation, OPD adopts a simple but important design: the training samples are generated online by the current student policy rather than collected offline from the teacher model. This design mainly addresses the distribution mismatch in offline distillation, where the student is trained on teacher-generated trajectories that may differ from the states it encounters during its own inference. In contrast, OPD provides teacher supervision directly on the states visited by the student, allowing the teacher to correct the student's actual behaviors. For example, consider a mathematical reasoning problem where the student has already generated the partial trajectory ``3x+7=22 $\rightarrow$ 3x=15''. At this student-generated state, the teacher model can directly provide token-level supervision that assigns a higher probability to generating ``x=5'' next, rather than an incorrect continuation such as ``x=4''. In this way, OPD teaches the student how to continue correctly from the states it actually visits, rather than only imitating complete trajectories generated by the teacher model.
...
...
@@ -147,7 +147,7 @@ Compared with conventional offline distillation, OPD adopts a simple but importa
\end{figure*}
This implementation of OPD is relatively simple, as illustrated in Figure~\ref{fig:opd-implementation}. Given an input $\mathbf{x}$, we first sample an output $\mathbf{y}=\{y_1,\ldots,y_n\}$ from the current student model $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$. For each token position $i$, the prefix $(\mathbf{x},\mathbf{y}_{<i})$ denotes a state actually visited by the student model. We then feed the same state into the teacher model $\mathrm{Pr}_{\mathrm{tea}}$ and obtain its probability distribution over the next token. The student model is optimized to match this teacher distribution. This OPD objective can defined as:
This implementation of OPD is relatively simple, as illustrated in Figure~\ref{fig:opd-implementation}. Given an input $\mathbf{x}$, we first sample an output $\mathbf{y}=\{y_1,\ldots,y_n\}$ from the current student model $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$. For each token position $i$, the prefix $(\mathbf{x},\mathbf{y}_{<i})$ denotes a state actually visited by the student model. We then feed the same state into the teacher model $\mathrm{Pr}_{\mathrm{tea}}$ and obtain its probability distribution over the next token. The student model is optimized to match this teacher distribution. This OPD objective can be defined as:
@@ -107,7 +107,7 @@ Tool use is another fundamental capability of LLM-based agents. Tools extend the
% 2. Prompt-based Tool Use
\subsubsubsection{Prompt-based Tool Use}
Prompt-based tool is a simple approach to enable LLM-based agents to interact with external tools without updating model parameters. However, the model may not naturally know when to invoke a tool or how to organize tool-use behaviors. Thus, we usually need to provide tool descriptions and a few demonstrations in the prompt. These demonstrations show the model how to alternate between reasoning, tool invocation, and observation. At each step, the model first generates a reasoning trace, then produces a tool-use action. After the tool returns an observation, the model uses this feedback to update its next decision. For example,
Prompt-based tool use is a simple approach to enable LLM-based agents to interact with external tools without updating model parameters. However, the model may not naturally know when to invoke a tool or how to organize tool-use behaviors. Thus, we usually need to provide tool descriptions and a few demonstrations in the prompt. These demonstrations show the model how to alternate between reasoning, tool invocation, and observation. At each step, the model first generates a reasoning trace, then produces a tool-use action. After the tool returns an observation, the model uses this feedback to update its next decision. For example,
\begin{tcolorbox}[frame empty]
\begingroup
...
...
@@ -158,7 +158,7 @@ This demonstration teaches the model a tool-use pattern. The model learns to rea
To make tool use more reliable, we can further train LLMs on tool-use trajectories. In this setting, each training example contains a user task, available tool descriptions, and a demonstrated tool-use process. The model learns when to invoke a tool, which tool to select, how to construct valid arguments, and how to use the returned observation. For example, we can construct a training example as follows:
With such data, SFT teaches the model to imitate the demonstrated tool-use pattern. The model can learn the format of tool invocation and the role of observations in later generation.
...
...
@@ -208,7 +208,7 @@ In application, this reward decomposition provides denser feedback than final-an
Another direction is to use RL to improve strategic tool use. For example, ReTool first builds a cold-start model with code-augmented reasoning traces and then applies RL with real-time code execution. During rollout, the model interleaves natural language reasoning with code execution, observes execution results, and revises its subsequent reasoning. This allows the model to discover when and how to invoke a code interpreter based on outcome feedback rather than human-written rules \citep{feng-etal:retool}. Search-R1 and ReSearch follow a similar idea in retrieval-augmented reasoning. They train models to generate search queries during reasoning and use retrieved information to update later steps, rather than relying on fixed retrieval pipelines or hand-crafted search prompts \citep{jin-etal:searchr1,chen-etal:research}.
So far, we have introduced how RL can enhance the core capabilities of LLM-based agents. Since LLM-based agents are built upon underlying LLMs, their RL training can naturally leverage the general LLM-based RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}. However, agentic RL introduces additional challenges beyond standard response optimization. A key challenge is \textit{how to obtain high-quality feedbacks from environments} and \textit{how to effectively use these signals to improve agent behaviors}. As a result, recent works on agentic RL requires not only appropriate RL algorithms, but also specialized approaches for addressing the unique challenges introduced by agentic settings \citep{huo-etal:learning,liu-etal:agentic}.
So far, we have introduced how RL can enhance the core capabilities of LLM-based agents. Since LLM-based agents are built upon underlying LLMs, their RL training can naturally leverage the general LLM-based RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}. However, agentic RL introduces additional challenges beyond standard response optimization. A key challenge is \textit{how to obtain high-quality feedback from environments} and \textit{how to effectively use these signals to improve agent behaviors}. As a result, recent work on agentic RL requires not only appropriate RL algorithms, but also specialized approaches for addressing the unique challenges introduced by agentic settings \citep{huo-etal:learning,liu-etal:agentic}.
\subsection{Environment Design and Scaling}
...
...
@@ -272,7 +272,7 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
\subsection{Credit Assignment}
Credit assignment is not unique to LLM-based agents. It is a long-standing challenge in traditional RL \citep{sutton-etal:temporal,zhou-etal:learning}. When rewards are delayed, the agent must decide which past actions should be reinforced or suppressed. Classical methods, such as temporal-difference learning and eligibility traces, address this issue by propagating reward information backward along the trajectory. In agentic RL, credit assignment becomes more challenging. This is because that each interaction step may involve high-level planning, tool invocation, and environmental feedback, rather than a single low-level action. Therefore, credit assignment often needs to operate at both the trajectory level and the step level.
Credit assignment is not unique to LLM-based agents. It is a long-standing challenge in traditional RL \citep{sutton-etal:temporal,zhou-etal:learning}. When rewards are delayed, the agent must decide which past actions should be reinforced or suppressed. Classical methods, such as temporal-difference learning and eligibility traces, address this issue by propagating reward information backward along the trajectory. In agentic RL, credit assignment becomes more challenging because each interaction step may involve high-level planning, tool invocation, and environmental feedback, rather than a single low-level action. Therefore, credit assignment often needs to operate at both the trajectory level and the step level.
This challenge is especially important for agent planning. In many agent tasks, the environment only provides a sparse reward after the entire trajectory is completed. However, the final success or failure may come from an earlier planning decision, an incorrect tool invocation, or an ineffective response to an intermediate observation. If we directly assign the same trajectory-level reward to all interaction steps, the supervision can become noisy. The agent may fail to identify which decisions should be reinforced and which should be suppressed.
where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indicate that the agent correctly extracts the numerical values and performs the calculation, $r_3$ indicates that the JSON formatting is partially correct, and $r_4$ indicates that the database update fails. In this way, the agent receives more precise feedback: it should preserve the correct planning.
where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indicate that the agent correctly extracts the numerical values and performs the calculation, $r_3$ indicates that the JSON formatting is partially correct, and $r_4$ indicates that the database update fails. In this way, the agent receives more precise feedback: it should preserve the correct planning and calculation steps while improving the database update step.
...
...
@@ -317,7 +317,7 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.
The first approach is \textbf{memory management} though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The first approach is \textbf{memory management} through agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The second approach is \textbf{skill optimization}. One challenge of using memory to learn from experience is scalability. As agent experience continuously accumulates, the memory size will keep growing. Moreover, a large amount of noisy and redundant information may also be introduced into the memory. As a result, efficiently storing and retrieving relevant information becomes increasingly difficult. Instead of storing individual experiences in memory, skill aims to identify common patterns across experiences and transform them into abstract behaviors. These skills provide a more compact representation of experience and allow the agent to transfer experience across different tasks. Similar to memory optimization, we can also update and refine the skills of agents to achieve learning from experience.
...
...
@@ -339,9 +339,9 @@ The three approaches mentioned above can be implemented through various techniqu
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
\begin{itemize}
\item\textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory bank to support the current interaction. The retrieved information can take various forms depending on how the memory bank is constructed. For example, the memory bank may contain successful trajectories from similar tasks, which can provide reusable solutions for current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles.
\item\textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory bank to support the current interaction. The retrieved information can take various forms depending on how the memory bank is constructed. For example, the memory bank may contain successful trajectories from similar tasks, which can provide reusable solutions for the current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles.
\item\textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
\item\textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in\{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically achieve this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
\item\textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in\{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically implement this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
\end{itemize}
...
...
@@ -353,7 +353,7 @@ One promising approach to improve memory management is to optimize memory operat
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
where $o$ denotes the selected memory operation from the operation set $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
\begin{figure}[!t]
\centering
...
...
@@ -371,7 +371,7 @@ where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denote
After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately \citep{xu-etal:a-mem}. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. This indicates that memory can serve as an important component of agent learning, beyond simply storing past information.
Beyond optimizing memory update operations, another important question is how to construct and organize memories from accumulated experiences. Instead of treating memory construction as a fixed preprocessing step, we can make an agent to learn effective memory construction strategies through RL. In this way, a straightforward approach is to formulate memory construction as a sequential decision-making process, where the agent learns how to select, organize, and transform interaction experiences into useful memories \citep{wang-etal:mem}.
Beyond optimizing memory update operations, another important question is how to construct and organize memories from accumulated experiences. Instead of treating memory construction as a fixed preprocessing step, we can enable an agent to learn effective memory construction strategies through RL. In this way, a straightforward approach is to formulate memory construction as a sequential decision-making process, where the agent learns how to select, organize, and transform interaction experiences into useful memories \citep{wang-etal:mem}.
% In this way, instead of treating memory updates as isolated operations for individual tasks, we can view memory management as an ongoing learning process, where agents gradually organize past experiences into reusable knowledge. As a result, we can consider memory as a core component of agent learning rather than a passive storage module \citep{xu-etal:a-mem}.
...
...
@@ -379,7 +379,7 @@ Beyond optimizing memory update operations, another important question is how to
\subsubsection{Skill Optimization}
\label{sec:skill-optimization}
Skill represents a higher-level abstraction of agentic experience. Different from memory, which focuses on storing useful information extracted from previous interactions, skill aims to summarize recurring patterns across multiple experiences and transform them into reusable capabilities. Specifically, a skill can be viewed as a structured instruction package that describes when the skill should be invoked, what workflow it should follow, which tools are required, and how the final outcome should be verified. For example, a reasoning-oriented skill may be represented as follows:
A skill represents a higher-level abstraction of agentic experience. Different from memory, which focuses on storing useful information extracted from previous interactions, a skill aims to summarize recurring patterns across multiple experiences and transform them into reusable capabilities. Specifically, a skill can be viewed as a structured instruction package that describes when the skill should be invoked, what workflow it should follow, which tools are required, and how the final outcome should be verified. For example, a reasoning-oriented skill may be represented as follows:
\begin{tcolorbox}[frame empty]
\begingroup
...
...
@@ -443,7 +443,7 @@ A straightforward approach to constructing skills for an agent is to manually wr
One limitation of manual skill construction is that the quality and diversity largely depend on human experience. Therefore, if we want LLMs to handle a broad range of tasks, relying on human-annotated data for LLM fine-tuning is often inefficient. To address this limitation, an alternative approach is to automatically generate skills from agentic experiences. For example, we can prompt an LLM to summarize successful trajectories and extract reusable skills. These generated skills can then be added to the agent's skill bank and reused in future tasks.
The above way of generating skills often suffer from quality issues. First, a general LLM does not inherently know how to design effective skills without additional optimization. Second, the skill generator may optimize for the wrong objective because skill generation and skill utilization involve different contexts. Specifically, an LLM may consider a generated skill effective because it satisfies the instructions provided in the prompt, while another agent may fail to use this skill effectively in real-world tasks. As a result, skill generation should not only focus on producing well-structured skills, but also consider whether these skills can actually improve agent performance during task execution. Recent work resorts to RL to achieve this goal, where the skill generation is optimized based on the actual performance of generated skills.
The above way of generating skills often suffers from quality issues. First, a general LLM does not inherently know how to design effective skills without additional optimization. Second, the skill generator may optimize for the wrong objective because skill generation and skill utilization involve different contexts. Specifically, an LLM may consider a generated skill effective because it satisfies the instructions provided in the prompt, while another agent may fail to use this skill effectively in real-world tasks. As a result, skill generation should not only focus on producing well-structured skills, but also consider whether these skills can actually improve agent performance during task execution. Recent work has turned to RL to achieve this goal, where skill generation is optimized based on the actual performance of generated skills.
\begin{figure*}[!t]
...
...
@@ -456,7 +456,7 @@ The above way of generating skills often suffer from quality issues. First, a ge
\end{figure*}
Here we consider \textbf{SkillRL} as an example to illustrate how to optimize the skill generation through RL \citep{xia-etal:skillrl}. The idea is that instead of directly generating skills from stored trajectories, we can first abstract reusable skills from existing trajectories and uses them to guide agentic RL training. During the training process, the agent continuously discovers new skills and updates the skill bank, enabling the joint evolution of the agent and its skills. Figure~\ref{fig:skillrl} shows a schematic illustration of SkillRL. Here we give a brief outline of the key steps involved.
Here we consider \textbf{SkillRL} as an example to illustrate how to optimize skill generation through RL \citep{xia-etal:skillrl}. The idea is that instead of directly generating skills from stored trajectories, we can first abstract reusable skills from existing trajectories and use them to guide agentic RL training. During the training process, the agent continuously discovers new skills and updates the skill bank, enabling the joint evolution of the agent and its skills. Figure~\ref{fig:skillrl} shows a schematic illustration of SkillRL. Here we give a brief outline of the key steps involved.
\begin{itemize}
\item Initially, we collect interaction trajectories from environments. These trajectories can include both successful and failed experiences, which provide diverse signals for skill discovery and optimization. Successful trajectories reveal effective behaviors, while failed trajectories help identify potential weaknesses and improvement opportunities.
\item The collected trajectories are then used to extract reusable skills, which initialize the skill bank. Considering that agent capabilities usually include both general problem-solving strategies and task-specific procedures, we can organize skills at different levels of abstraction. Specifically, a hierarchical skill library can be constructed, where a general skill bank stores transferable skills learned across different tasks to improve generalization, while a task-specific skill bank preserves specialized strategies for solving particular tasks.
@@ -76,9 +76,9 @@ To address this challenge, researchers have explored RL as an effective approach
\label{eq:optimization_objective}
\end{eqnarray}
Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. For example, given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color:
Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. Given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color.
However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because that diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as a MDP. Specifically, we can consider the denoising procedure as a multi-step MDP and applies a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as an MDP. Specifically, we can consider the denoising procedure as a multi-step MDP and apply a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
@@ -102,7 +102,7 @@ where $S_{z}$ denotes the training set, $\tau_\mathrm{dm}$ denotes the denoising
\subsubsection{Flow Matching Models}
Different from diffusion models, flow matching models achieve the multimodal generation process based on ordinary differential equations (ODEs), where data transformation is modeled as a continuous-time flow. Instead of gradually adding and removing noise through a stochastic diffusion process, flow matching learns a continuous vector field that transports samples from a simple prior distribution to the target data distribution. In this subsection, we briefly introduce the training and generation processes of flow matching models, and then discuss how RL can be applied to optimize them. Note that we do not provide a detailed introduction to the underlying principles of flow matching models in this subsection. Interested readers can refer to existing tutorials for further details \citep{xiao-etal:ordinary}.
Different from diffusion models, flow matching models perform multimodal generation based on ordinary differential equations (ODEs), where data transformation is modeled as a continuous-time flow. Instead of gradually adding and removing noise through a stochastic diffusion process, flow matching learns a continuous vector field that transports samples from a simple prior distribution to the target data distribution. In this subsection, we briefly introduce the training and generation processes of flow matching models, and then discuss how RL can be applied to optimize them. Note that we do not provide a detailed introduction to the underlying principles of flow matching models in this subsection. Interested readers can refer to existing tutorials for further details \citep{xiao-etal:ordinary}.
Let $\mathbf{x}_0\sim X_0$ denote a data sample from the target distribution and $\mathbf{x}_1\sim X_1$ denote a noise sample from the prior distribution. Flow matching constructs an intermediate state by interpolating between the data and noise distributions:
\begin{eqnarray}
...
...
@@ -137,7 +137,7 @@ a_t=\mathbf{x}_{t-\Delta t}
Since the flow model deterministically predicts the velocity field, the policy can be represented as:
where $\delta(\cdot)$ denotes the Dirac delta distribution, indicating that the next state is deterministically determined by the current state and the learned velocity field.
