@@ -10,7 +10,7 @@ Understanding these basic mechanisms is important before introducing RL for LLMs
...
@@ -10,7 +10,7 @@ Understanding these basic mechanisms is important before introducing RL for LLMs
In this paper, we mainly focus on generative LLMs based on decoder-only Transformers \citep{vaswani-etal:2017attention}. Let $\mathcal{V}$ denote the vocabulary of tokens. A token is the basic unit processed by an LLM. It can be a word, a subword, a punctuation mark, or another text fragment produced by the tokenizer. Before a raw text is input into an LLM, it is first converted into a sequence of tokens from $\mathcal{V}$. Therefore, throughout this paper, all inputs and outputs of an LLM are represented as token sequences. Given a token sequence $\mathbf{z}=z_1...z_N$, a language model parameterized by $\theta$ estimates the probability of the sequence by factorizing it from left to right:
In this paper, we mainly focus on generative LLMs based on decoder-only Transformers \citep{vaswani-etal:2017attention}. Let $\mathcal{V}$ denote the vocabulary of tokens. A token is the basic unit processed by an LLM. It can be a word, a subword, a punctuation mark, or another text fragment produced by the tokenizer. Before a raw text is input into an LLM, it is first converted into a sequence of tokens from $\mathcal{V}$. Therefore, throughout this paper, all inputs and outputs of an LLM are represented as token sequences. Given a token sequence $\mathbf{z}=z_1...z_N$, a language model parameterized by $\theta$ estimates the probability of the sequence by factorizing it from left to right:
@@ -111,10 +111,10 @@ Another important concept related to prompting is in-context learning. When prom
...
@@ -111,10 +111,10 @@ Another important concept related to prompting is in-context learning. When prom
Prompting is crucial because it directly influences how the LLM understands the task and generates the output. A well-designed prompt can guide the model to produce more accurate, relevant, and appropriately formatted responses. However, since the model parameters are unchanged, the result can also be sensitive to the wording, structure, and demonstrations provided in the prompt.
Prompting is crucial because it directly influences how the LLM understands the task and generates the output. A well-designed prompt can guide the model to produce more accurate, relevant, and appropriately formatted responses. However, since the model parameters are unchanged, the result can also be sensitive to the wording, structure, and demonstrations provided in the prompt.
Following the notation used above, we denote the entire prompt by $\mathbf{x}=x_1...x_m$. Prompting asks the LLM to generate an output $\mathbf{y}=y_1...y_T$ according to the conditional distribution $\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$. The probability of the output is factorized as
Following the notation used above, we denote the entire prompt by $\mathbf{x}=x_1...x_m$. Prompting asks the LLM to generate an output $\mathbf{y}=y_1...y_T$ according to the conditional distribution $\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$. The probability of the output is factorized as
where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the tokens that have already been generated. This conditional generation formulation will be repeatedly used in the following sections. From this perspective, a prompt mainly changes the condition under which the same language model distribution is used.
where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the tokens that have already been generated. This conditional generation formulation will be repeatedly used in the following sections. From this perspective, a prompt mainly changes the condition under which the same language model distribution is used.
...
@@ -213,16 +213,27 @@ Regardless of the decoding strategy, LLM generation can be viewed as a sequence
...
@@ -213,16 +213,27 @@ Regardless of the decoding strategy, LLM generation can be viewed as a sequence
RL, supported by a well-established theoretical framework, has been widely applied across a broad range of domains. In particular, with the rapid advancement of LLMs, RL has become a standard approach in the post-training stage. Before using LLM training examples to explain RL algorithms, we first provide a brief overview of deep reinforcement learning in this subsection. We then introduce the general formulation of RL, along with key terminologies and notations that are essential for understanding RL.
RL, supported by a well-established theoretical framework, has been widely applied across a broad range of domains. In particular, with the rapid advancement of LLMs, RL has become a standard approach in the post-training stage. Before using LLM training examples to explain RL algorithms, we first provide a brief overview of deep reinforcement learning in this subsection. We then introduce the general formulation of RL, along with key terminologies and notations that are essential for understanding RL.
\subsubsection{Markov Decision Process}
\subsubsection{Markov Decision Process}
The general framework of RL is illustrated in Figure~\ref{fig:rl_framework}. An RL system primarily consists of two components: an agent and an environment. At each timestep $t$, the agent observes a state $s_t$ from the environment and selects an action $a_t$ according to a policy $\pi$, which is typically parameterized by a neural network. After executing the action, the environment transitions to a new state $s_{t+1}$ and returns a reward $r_t$ corresponding to the taken action.
The RL formulation for LLMs is illustrated in Figure~\ref{fig:llm-rl-policy-optimization}. An RL system primarily consists of two components: an agent and an environment. At each timestep $t$, the agent observes a state $s_t$ from the environment and selects an action $a_t$ according to a policy $\pi$, which is typically parameterized by a neural network. After executing the action, the environment transitions to a new state $s_{t+1}$ and returns a reward $r_t$ corresponding to the taken action.
\begin{figure}[!t]
% \begin{figure}[!t]
% \centering
% \input{section2/Figures/rl_framework}
% \caption{
% A general framework of deep reinforcement learning. The policy is parameterized by neural networks, and the agent learns an optimal policy through interactions with the environment to maximize cumulative rewards.
% }
% \label{fig:rl_framework}
% \end{figure}
\begin{figure*}[!t]
\centering
\centering
\input{section2/Figures/rl_framework}
\resizebox{\linewidth}{!}{
\caption{
\input{section2/Figures/llm-and-rl-elements.tex}
A general framework of deep reinforcement learning. The policy is parameterized by neural networks, and the agent learns an optimal policy through interactions with the environment to maximize cumulative rewards.
}
}
\label{fig:rl_framework}
\caption{A schematic illustration of RL for LLMs.
\end{figure}
% The current context defines the state $s_t=(\mathbf{x},\mathbf{y}_{<t})$, the policy corresponds to the next-token distributions $\Pr_{\theta}(\cdot \mid \mathbf{x},\mathbf{y}_{<t})$, and sampling from the policy yields the action $a_t=y_t$. The reward model provides feedback for the sampled action, while value functions estimate long-term returns for policy optimization.
}
\label{fig:llm-rl-policy-optimization}
\end{figure*}
This interaction process is commonly modeled as a Markov Decision Process (MDP), where the agent interacts with the environment over discrete timesteps \citep{Sutton-and-Barto:2018RL}. A sequence of states and actions forms a trajectory, denoted as $\tau=(s_0, a_0, s_1, a_1, \cdots, s_{H-1}, a_{H-1})$, where $H$ represents the trajectory length. Each trajectory accumulates rewards from the environment.
This interaction process is commonly modeled as a Markov Decision Process (MDP), where the agent interacts with the environment over discrete timesteps \citep{Sutton-and-Barto:2018RL}. A sequence of states and actions forms a trajectory, denoted as $\tau=(s_0, a_0, s_1, a_1, \cdots, s_{H-1}, a_{H-1})$, where $H$ represents the trajectory length. Each trajectory accumulates rewards from the environment.
The learner or decision-maker in reinforcement learning. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals. \\
The learner or decision-maker in RL. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals. \\
\midrule
\midrule
\textbf{Environment}&
\textbf{Environment}&
Everything external to the agent with which it interacts. Unlike traditional RL settings that involve physical or simulated environments, the environment in LLM-based RL is typically abstract, consisting of the training framework that provides feedback (e.g., reward models, human annotations, or evaluation metrics) for generated outputs. \\
Everything external to the agent with which it interacts. Unlike traditional RL settings that involve physical or simulated environments, the environment in LLM-based RL is typically abstract, consisting of the training framework that provides feedback (e.g., reward models, human annotations, or evaluation metrics) for generated outputs. \\
\midrule
\midrule
\textbf{State ($S$)}&
\textbf{State ($s$)}&
A state represents the current situation of the environment. For language modeling, the state at timestep $t$ can be defined as the sequence of observed tokens up to that point, i.e., the context used to predict the next token. Formally, the state can be represented as $S=(x,y_{<t})$, where $x$ denotes the input prompt and $y_{<t}$ denotes the previously generated tokens. \\
A state represents the current situation of the environment. For language modeling, the state at timestep $t$ can be defined as the sequence of observed tokens up to that point, i.e., the context used to predict the next token. Formally, the state can be represented as $s_t=(\mathbf{x},\mathbf{y}_{<t})$, where $\mathbf{x}$ denotes the input prompt and $\mathbf{y}_{<t}$ denotes the previously generated tokens. \\
\midrule
\midrule
\textbf{Action ($a$)}&
\textbf{Action ($a$)}&
An action corresponds to a decision made by the agent. In LLMs, actions are naturally defined as selecting the next token from the vocabulary, i.e., $a=y_t$. \\
An action corresponds to a decision made by the agent. In LLMs, actions are naturally defined as selecting the next token from the vocabulary, i.e., $a=y_t$. \\
...
@@ -34,16 +34,16 @@ The reward provides feedback from the environment to evaluate the quality of an
...
@@ -34,16 +34,16 @@ The reward provides feedback from the environment to evaluate the quality of an
\midrule
\midrule
\textbf{Policy ($\pi$)}&
\textbf{Policy ($\pi$)}&
The policy defines the agent's behavior, i.e., the probability of taking an action given a state. For LLMs, the policy corresponds to the conditional probability distribution over the next token given the context:
The policy defines the agent's behavior, i.e., the probability of taking an action given a state. For LLMs, the policy corresponds to the conditional probability distribution over the next token given the context:
where $a=y_t$ and $s=(x,y_{<t})$. Under this formulation, an LLM can be naturally interpreted as a parameterized policy. \\
where $a_t=y_t$ and $s_t=(\mathbf{x},\mathbf{y}_{<t})$. Under this formulation, an LLM can be naturally interpreted as a parameterized policy. \\
\midrule
\midrule
\textbf{\makecell[l]{Value Function\\($V$ and $Q$)}}&
\textbf{\makecell[l]{Value Function\\($V$ and $Q$)}}&
The value function estimates the expected cumulative reward when following a policy. The state-value function $V(s)$ measures the expected discounted return starting from state $s$:
The value function estimates the expected cumulative reward when following a policy. The state-value function $V(s)$ measures the expected discounted return starting from state $s$:
\State\%\%\%\textit{compute policy model loss and value model loss through advantages and returns}
\State\%\%\%\textit{compute policy model loss and value model loss through advantages and returns}
\State initialize the loss values: $\mathrm{loss}_{p}=0$, $\mathrm{loss}_{v}=0$
\State initialize the loss values: $\mathrm{loss}_{p}=0$, $\mathrm{loss}_{v}=0$
\For{t=1 to T}
\For{t=1 to T}
\State compute the penalty through Eq.(\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log\mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})-\log\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State compute the penalty through Eq.~(\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log\mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})-\log\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State compute the advantage $A(\mathbf{x},\mathbf{y}_{<t},y_{t})$: $A(\mathbf{x},\mathbf{y}_{<t},y_{t})=r_{t}+\gamma V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the advantage $A(\mathbf{x},\mathbf{y}_{<t},y_{t})$: $A(\mathbf{x},\mathbf{y}_{<t},y_{t})=r_{t}+\gamma V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the policy model loss through Eq.(\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p}=\mathrm{loss}_{p}+\mathrm{Clip}\Big(\frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})}\Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State compute the policy model loss through Eq.~(\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p}=\mathrm{loss}_{p}+\mathrm{Clip}\Big(\frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})}\Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State\%\%\%\textit{compute value model loss}
\State\%\%\%\textit{compute value model loss}
\State compute the return $\mathrm{Return}_{t}$: $\mathrm{Return}_{t}=A(\mathbf{x},\mathbf{y}_{<t},y_t)+V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the return $\mathrm{Return}_{t}$: $\mathrm{Return}_{t}=A(\mathbf{x},\mathbf{y}_{<t},y_t)+V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the value model loss through Eq.(\ref{eq:value-loss}) and add it to $\mathrm{loss}_{v}$: $\mathrm{loss}_v =\mathrm{loss}_v +(\mathrm{Return}_{t}-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t}))^2$
\State compute the value model loss through Eq.~(\ref{eq:value-loss}) and add it to $\mathrm{loss}_{v}$: $\mathrm{loss}_v =\mathrm{loss}_v +(\mathrm{Return}_{t}-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t}))^2$
\EndFor
\EndFor
\State update the parameters of $\pi_{\theta}(\cdot)$ through $\mathrm{loss}_{p}$
\State update the parameters of $\pi_{\theta}(\cdot)$ through $\mathrm{loss}_{p}$
\State update the parameters of $V_{\omega}(\cdot)$ through $\mathrm{loss}_{v}$
\State update the parameters of $V_{\omega}(\cdot)$ through $\mathrm{loss}_{v}$
@@ -82,24 +82,24 @@ We can further refine the process when optimizing the objective using gradient d
...
@@ -82,24 +82,24 @@ We can further refine the process when optimizing the objective using gradient d
\label{fig:understand-policy-gradient}
\label{fig:understand-policy-gradient}
\end{figure*}
\end{figure*}
Using a Monte Carlo sample average over $d$ outputs in $\mathcal{D}$, we can simplify Eq.(\ref{eq:rl-gradient-j-theta}) and need only consider the terms $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta}$ and $R(\mathbf{y})$:
Using a Monte Carlo sample average over $d$ outputs in $\mathcal{D}$, we can simplify Eq.~(\ref{eq:rl-gradient-j-theta}) and need only consider the terms $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta}$ and $R(\mathbf{y})$:
Now, as illustrated in Figure\ref{fig:update-with-policy-gradient}, we have an RL approach to optimize the output of the LLM: 1) we sample inputs from the input-only dataset and sample outputs from the LLM; then, 2) we evaluate each sampled output using a reward function that we define; then, 3) we update the LLM using gradient descent, following Eq. (\ref{eq:rl-j-theta-gradient-simplified}), to maximize the performance function.
Now, as illustrated in Figure~\ref{fig:update-with-policy-gradient}, we have an RL approach to optimize the output of the LLM: 1) we sample inputs from the input-only dataset and sample outputs from the LLM; then, 2) we evaluate each sampled output using a reward function that we define; then, 3) we update the LLM using gradient descent, following Eq.~(\ref{eq:rl-j-theta-gradient-simplified}), to maximize the performance function.
We can understand this optimization process from the perspective of hypothesis space reranking, as illustrated in Figure\ref{fig:understand-policy-gradient}. The objective is to increase the probability of longer outputs in the hypothesis space by assigning a high reward to those outputs, while simultaneously decreasing the probability of shorter outputs by assigning them a lower reward. In other words, the policy gradient approach encourages the model to generate longer outputs by reinforcing those outputs that meet the desired length criteria, and it penalizes shorter outputs that do not meet the objective.
We can understand this optimization process from the perspective of hypothesis space reranking, as illustrated in Figure~\ref{fig:understand-policy-gradient}. The objective is to increase the probability of longer outputs in the hypothesis space by assigning a high reward to those outputs, while simultaneously decreasing the probability of shorter outputs by assigning them a lower reward. In other words, the policy gradient approach encourages the model to generate longer outputs by reinforcing those outputs that meet the desired length criteria, and it penalizes shorter outputs that do not meet the objective.
\subsection{Temporal Decomposition}
\subsection{Temporal Decomposition}
\label{sec:temporal-decomposition}
\label{sec:temporal-decomposition}
While optimizing using Eq. (\ref{eq:rl-j-theta-gradient-simplified}) is intuitive, a potential issue arises: in some cases, the sampled outputs may significantly overlap, yet the rewards for the overlapping parts differ. For example, as illustrated in Figure \ref{fig:an-overlap-example}, if outputs $\mathbf{y}_1$ and $\mathbf{y}_2$ both include the same tokens in the third tip ``3. Double-check your work...anyone can significantly enhance their math problem-solving abilities.'', but due to variations in the entire outputs, they receive markedly different rewards (e.g., $R(\mathbf{y}_1)=50$ vs. $R(\mathbf{y}_2)=10$). Ideally, we would want similar generation behaviors to be rewarded consistently, without significant variation, as their contributions are equivalent to the sum of rewards.
While optimizing using Eq.~(\ref{eq:rl-j-theta-gradient-simplified}) is intuitive, a potential issue arises: in some cases, the sampled outputs may significantly overlap, yet the rewards for the overlapping parts differ. For example, as illustrated in Figure~\ref{fig:an-overlap-example}, if outputs $\mathbf{y}_1$ and $\mathbf{y}_2$ both include the same tokens in the third tip ``3. Double-check your work...anyone can significantly enhance their math problem-solving abilities.'', but due to variations in the entire outputs, they receive markedly different rewards (e.g., $R(\mathbf{y}_1)=50$ vs. $R(\mathbf{y}_2)=10$). Ideally, we would want similar generation behaviors to be rewarded consistently, without significant variation, as their contributions are equivalent to the sum of rewards.
Before discussing an approach to solve this issue, we first perform temporal decomposition for the term $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta} R(\mathbf{y})$ in Eq.(\ref{eq:rl-j-theta-gradient-simplified}), and obtain
Before discussing an approach to solve this issue, we first perform temporal decomposition for the term $\frac{\partial\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})}{\partial\theta} R(\mathbf{y})$ in Eq.~(\ref{eq:rl-j-theta-gradient-simplified}), and obtain
@@ -115,7 +115,7 @@ Example of sampled outputs demonstrating that the same tokens will receive signi
...
@@ -115,7 +115,7 @@ Example of sampled outputs demonstrating that the same tokens will receive signi
\label{fig:an-overlap-example}
\label{fig:an-overlap-example}
\end{figure*}
\end{figure*}
In this form, we observe that, when generating the token at $t$-th time step, it does not affect the term $\sum_{k=1}^{t-1} r_{k}$ and only affects the term $\sum_{k=t}^{T} r_{k}$. Therefore, we can safely remove the former part, $\sum_{k=1}^{t-1} r_{k}$, to prevent it from affecting the optimization of the current step. By doing so, we can achieve a more stable gradient computation for Eq.(\ref{eq:rl-j-theta-gradient-simplified}):
In this form, we observe that, when generating the token at $t$-th time step, it does not affect the term $\sum_{k=1}^{t-1} r_{k}$ and only affects the term $\sum_{k=t}^{T} r_{k}$. Therefore, we can safely remove the former part, $\sum_{k=1}^{t-1} r_{k}$, to prevent it from affecting the optimization of the current step. By doing so, we can achieve a more stable gradient computation for Eq.~(\ref{eq:rl-j-theta-gradient-simplified}):
@@ -125,7 +125,7 @@ In fact, this simplification follows the Markov process, asserting that the futu
...
@@ -125,7 +125,7 @@ In fact, this simplification follows the Markov process, asserting that the futu
\subsection{Reducing Gradient Variance}
\subsection{Reducing Gradient Variance}
\label{sec:reduce-gradient-variance}
\label{sec:reduce-gradient-variance}
Returning to the case discussed in Section\ref{sec:temporal-decomposition}, while the strategy of excluding rewards for past tokens reduces gradient variance, substantial gradient variance still occurs in practice. First, if overlapping portions of the output appear early, the modified Eq. (\ref{eq:modified-gradient-simplified}) may still experience similar problems, i.e., the same tokens receive vastly different rewards. Furthermore, the reward distribution can vary significantly between different time steps within a single output. For example, consider an output sequence of 500 tokens. According to Eq. (\ref{eq:modified-gradient-simplified}), the reward at the 10th step might be significantly higher than at the 450th step, i.e., 49 vs. 5. Such varying rewards for good and poor tokens can result in a very low total reward for the entire output, even if it includes good tokens.
Returning to the case discussed in Section~\ref{sec:temporal-decomposition}, while the strategy of excluding rewards for past tokens reduces gradient variance, substantial gradient variance still occurs in practice. First, if overlapping portions of the output appear early, the modified Eq.~(\ref{eq:modified-gradient-simplified}) may still experience similar problems, i.e., the same tokens receive vastly different rewards. Furthermore, the reward distribution can vary significantly between different time steps within a single output. For example, consider an output sequence of 500 tokens. According to Eq.~(\ref{eq:modified-gradient-simplified}), the reward at the 10th step might be significantly higher than at the 450th step, i.e., 49 vs. 5. Such varying rewards for good and poor tokens can result in a very low total reward for the entire output, even if it includes good tokens.
One simple method for further reducing the variance of the gradient is to set a baseline $b$ and subtract it from $\sum_{k=t}^{T} r_k$, resulting in $\sum_{k=t}^{T} r_k - b$.\footnote{In fact, the use of a baseline $b$ does not change the variance of the total rewards $\sum_{t=1}^{T} r_t$. However, it is important to note that while introducing a baseline does not alter the overall variance of the rewards, it helps reduce the variance of the gradient estimates. This is because subtracting the baseline from the total rewards effectively reduces fluctuations around their mean, which makes the gradient estimates more stable. In general, the operation $\sum_{k=t}^{T} r_k - b$ centers the rewards around zero (e.g., $b$ is defined as the expected value of $\sum_{k=t}^{T} r_k$), which can lead to reduced variance in the product $\sum_{k=t}^{T}\log\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})(\sum_{k=t}^{T} r_k - b)$.} Here, the baseline can be interpreted as a reference point. By centering the rewards around this baseline, we remove systematic biases in the reward.
One simple method for further reducing the variance of the gradient is to set a baseline $b$ and subtract it from $\sum_{k=t}^{T} r_k$, resulting in $\sum_{k=t}^{T} r_k - b$.\footnote{In fact, the use of a baseline $b$ does not change the variance of the total rewards $\sum_{t=1}^{T} r_t$. However, it is important to note that while introducing a baseline does not alter the overall variance of the rewards, it helps reduce the variance of the gradient estimates. This is because subtracting the baseline from the total rewards effectively reduces fluctuations around their mean, which makes the gradient estimates more stable. In general, the operation $\sum_{k=t}^{T} r_k - b$ centers the rewards around zero (e.g., $b$ is defined as the expected value of $\sum_{k=t}^{T} r_k$), which can lead to reduced variance in the product $\sum_{k=t}^{T}\log\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})(\sum_{k=t}^{T} r_k - b)$.} Here, the baseline can be interpreted as a reference point. By centering the rewards around this baseline, we remove systematic biases in the reward.
where $\sum_{k=t}^{T} r_k $ represents the actual return received, and $V(\mathbf{x}, \mathbf{y}_{<t}, y_{t})$ (or $V_{t}$ for short) represents the expected return at time step $t$. $A(\mathbf{x}, \mathbf{y}_{<t}, y_{t})$ (or $A_t$ for short) is called the advantage at time step $t$, which quantifies the relative benefit of the current generated $y_t$ compared to the expected return. Computing the advantage using Eq.(\ref{eq:monte-carlo-advantage}) is a method known as Monte Carlo-based advantage estimation.
where $\sum_{k=t}^{T} r_k $ represents the actual return received, and $V(\mathbf{x}, \mathbf{y}_{<t}, y_{t})$ (or $V_{t}$ for short) represents the expected return at time step $t$. $A(\mathbf{x}, \mathbf{y}_{<t}, y_{t})$ (or $A_t$ for short) is called the advantage at time step $t$, which quantifies the relative benefit of the current generated $y_t$ compared to the expected return. Computing the advantage using Eq.~(\ref{eq:monte-carlo-advantage}) is a method known as Monte Carlo-based advantage estimation.
By using the advantage function $A_{t}$, the gradient of $J(\theta)$ can be written in the form
By using the advantage function $A_{t}$, the gradient of $J(\theta)$ can be written in the form
@@ -168,7 +168,7 @@ Based on this training objective, the loss function of training the LLM can be w
...
@@ -168,7 +168,7 @@ Based on this training objective, the loss function of training the LLM can be w
\label{fig:architecture-value-function}
\label{fig:architecture-value-function}
\end{figure*}
\end{figure*}
In practice, there are many ways to implement the value function. One simple approach is to build it based on a pre-trained LLM (e.g., an SFT LLM). In this way, the value function is also called the value model (or critic model). More specifically, we can concatenate $\mathbf{x}$ and $\mathbf{y}$ to form a single token sequence $\mathrm{seq}_{\mathbf{x},\mathbf{y}}=[\mathbf{x}, \mathbf{y}]$. We run an SFT LLM on this sequence, as usual, and at each position, we obtain a representation from the top-most Transformer layer. Then, we take the representations at the output (denoted by $\{\mathbf{h}_{y_{1}},\mathbf{h}_{y_{2}},\cdots,\mathbf{h}_{y_{T}}\}$) and map them to scalars via linear transformation, respectively. Figure\ref{fig:architecture-value-function} illustrates this architecture of the value model. Hence, at $t$-th time step, we can compute the value $V_{t}$
In practice, there are many ways to implement the value function. One simple approach is to build it based on a pre-trained LLM (e.g., an SFT LLM). In this way, the value function is also called the value model (or critic model). More specifically, we can concatenate $\mathbf{x}$ and $\mathbf{y}$ to form a single token sequence $\mathrm{seq}_{\mathbf{x},\mathbf{y}}=[\mathbf{x}, \mathbf{y}]$. We run an SFT LLM on this sequence, as usual, and at each position, we obtain a representation from the top-most Transformer layer. Then, we take the representations at the output (denoted by $\{\mathbf{h}_{y_{1}},\mathbf{h}_{y_{2}},\cdots,\mathbf{h}_{y_{T}}\}$) and map them to scalars via linear transformation, respectively. Figure~\ref{fig:architecture-value-function} illustrates this architecture of the value model. Hence, at $t$-th time step, we can compute the value $V_{t}$
\begin{eqnarray}
\begin{eqnarray}
V_{t}& = &\mathbf{h}_{y_{t}}\mathbf{W}_{v}
V_{t}& = &\mathbf{h}_{y_{t}}\mathbf{W}_{v}
\end{eqnarray}
\end{eqnarray}
...
@@ -186,7 +186,7 @@ or alternatively, introduce the discount factor $\gamma$ to obtain a more genera
...
@@ -186,7 +186,7 @@ or alternatively, introduce the discount factor $\gamma$ to obtain a more genera
where $\gamma\in[0,1]$ is the discount factor that adjusts the importance of future rewards. When $\gamma$ is set to less than 1, it signifies that early rewards are considered more important than future rewards. This basic idea is also applied in other fields. For example, in LLMs, it has been demonstrated that early token generation plays a crucial role, as it can influence the style and accuracy of the entire output \citep{wang-and-zhou:2024chain}.
where $\gamma\in[0,1]$ is the discount factor that adjusts the importance of future rewards. When $\gamma$ is set to less than 1, it signifies that early rewards are considered more important than future rewards. This basic idea is also applied in other fields. For example, in LLMs, it has been demonstrated that early token generation plays a crucial role, as it can influence the style and accuracy of the entire output \citep{wang-and-zhou:2024chain}.
In this subsection, we have detailed the integration of a value model in training LLMs. In practice, this training approach, represented by Eq.(\ref{eq:modified-gradient-simplified-advantage}), is known as the Advantage Actor-Critic (A2C) method \citep{mnih-etal:2016asynchronous}. The A2C method facilitates an interaction where a policy model (the actor) and a value model (the critic) learn in parallel and undergo synchronous updates. However, RL is a vast field, and many technical details cannot be covered here. The interested reader can refer to RL books for more details \citep{szepesvari:2010algorithms, Sutton-and-Barto:2018RL}.
In this subsection, we have detailed the integration of a value model in training LLMs. In practice, this training approach, represented by Eq.~(\ref{eq:modified-gradient-simplified-advantage}), is known as the Advantage Actor-Critic (A2C) method \citep{mnih-etal:2016asynchronous}. The A2C method facilitates an interaction where a policy model (the actor) and a value model (the critic) learn in parallel and undergo synchronous updates. However, RL is a vast field, and many technical details cannot be covered here. The interested reader can refer to RL books for more details \citep{szepesvari:2010algorithms, Sutton-and-Barto:2018RL}.
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
...
@@ -199,7 +199,7 @@ In this subsection, we have detailed the integration of a value model in trainin
...
@@ -199,7 +199,7 @@ In this subsection, we have detailed the integration of a value model in trainin
\subsection{Importance Sampling}
\subsection{Importance Sampling}
\label{sec:importance-sampling}
\label{sec:importance-sampling}
In this subsection, we discuss methods to improve the efficiency of the policy gradient, focusing on the sampling process. Although we discussed in Section\ref{sec:policy-gradient} that Monte Carlo-based sampling can estimate the hypothesis space, it is awfully inefficient for one single update in scenarios like ours where the sampled output may consist of hundreds or thousands of tokens. A natural idea from this point is to consider whether it is possible to reuse these sampled outputs to update the LLMs multiple times. With importance sampling, this is feasible, and we can rewrite the loss function of training the LLM as
In this subsection, we discuss methods to improve the efficiency of the policy gradient, focusing on the sampling process. Although we discussed in Section~\ref{sec:policy-gradient} that Monte Carlo-based sampling can estimate the hypothesis space, it is awfully inefficient for one single update in scenarios like ours where the sampled output may consist of hundreds or thousands of tokens. A natural idea from this point is to consider whether it is possible to reuse these sampled outputs to update the LLMs multiple times. With importance sampling, this is feasible, and we can rewrite the loss function of training the LLM as
where $\theta_{\mathrm{ref}}$ denotes the parameters of the previously used LLM (also called reference policy or old policy). Here, we use $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ or $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot|\mathbf{x})$ to denote $\mathcal{D}$, distinguishing from which model the output is sampled. The ratio $\frac{\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}$, also called the ratio function, compares the probability of token $y_t$ under the current and reference policies. This ratio function is used to reweight the observed rewards, reflecting how much more or less likely a token is under the current policy compared to the reference policy. When this ratio is greater than 1, it indicates that the token $y_t$ is more favored by the current policy than the reference policy. Conversely, a ratio less than 1 indicates that $y_t$ is less favored by the current policy. However, when the current $\theta_{\mathrm{ref}}$ diverges significantly from $\theta$, the accuracy of the hypothesis space estimation decreases. Therefore, we do need to resync it regularly. As illustrated in Figure\ref{fig:policy-gradient-with-importance-sampling}, one simple way is to designate the reference policy as the LLM from which we initiate updates during a training step. Note that we can conserve computational time and memory by eliminating the need for model copying in Step 1. Specifically, rather than maintaining a separate copy of the model, we directly sample from the current policy, retain the output probabilities $\{\mathrm{Pr}_{\theta}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$, and later use these stored probabilities to act as $\{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$ when updating the policy with Eq. (\ref{eq:loss-function-training-llm-importance-sampling}).
where $\theta_{\mathrm{ref}}$ denotes the parameters of the previously used LLM (also called reference policy or old policy). Here, we use $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ or $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot|\mathbf{x})$ to denote $\mathcal{D}$, distinguishing from which model the output is sampled. The ratio $\frac{\mathrm{Pr}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}$, also called the ratio function, compares the probability of token $y_t$ under the current and reference policies. This ratio function is used to reweight the observed rewards, reflecting how much more or less likely a token is under the current policy compared to the reference policy. When this ratio is greater than 1, it indicates that the token $y_t$ is more favored by the current policy than the reference policy. Conversely, a ratio less than 1 indicates that $y_t$ is less favored by the current policy. However, when the current $\theta_{\mathrm{ref}}$ diverges significantly from $\theta$, the accuracy of the hypothesis space estimation decreases. Therefore, we do need to resync it regularly. As illustrated in Figure~\ref{fig:policy-gradient-with-importance-sampling}, one simple way is to designate the reference policy as the LLM from which we initiate updates during a training step. Note that we can conserve computational time and memory by eliminating the need for model copying in Step 1. Specifically, rather than maintaining a separate copy of the model, we directly sample from the current policy, retain the output probabilities $\{\mathrm{Pr}_{\theta}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$, and later use these stored probabilities to act as $\{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_1|\mathbf{x}), \cdots, \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_T|\mathbf{x}, \mathbf{y}_{<T})\}$ when updating the policy with Eq.~(\ref{eq:loss-function-training-llm-importance-sampling}).
However, an inherent issue arises when using importance sampling to update LLMs. As shown in Figure\ref{fig:terrible-step-importance-sampling}, when a particular optimization step results in poor updates (or a terrible step for short), the reference policy utilized in subsequent Step 1 of the next iteration inherits these poor characteristics. Consequently, the samples drawn in Step 2 are likely to be adversely affected, leading to a more terrible step. Unlike SFT\footnote{In supervised learning, a terrible step during a training step caused by bad samples can often be corrected in subsequent steps by good samples.}, this cycle does not correct the initial terrible step but potentially exacerbates it, creating a downward spiral in policy performance.
However, an inherent issue arises when using importance sampling to update LLMs. As shown in Figure~\ref{fig:terrible-step-importance-sampling}, when a particular optimization step results in poor updates (or a terrible step for short), the reference policy utilized in subsequent Step 1 of the next iteration inherits these poor characteristics. Consequently, the samples drawn in Step 2 are likely to be adversely affected, leading to a more terrible step. Unlike SFT\footnote{In supervised learning, a terrible step during a training step caused by bad samples can often be corrected in subsequent steps by good samples.}, this cycle does not correct the initial terrible step but potentially exacerbates it, creating a downward spiral in policy performance.
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
...
@@ -234,7 +234,7 @@ At the time step $t$, we can obtain the penalty as
...
@@ -234,7 +234,7 @@ At the time step $t$, we can obtain the penalty as
By including this penalty in the optimization objective, we encourage the current policy to remain close to the reference policy, limiting very large updates in a terrible step.
By including this penalty in the optimization objective, we encourage the current policy to remain close to the reference policy, limiting very large updates in a terrible step.
We can incorporate this penalty into the Eq.(\ref{eq:loss-function-training-llm-importance-sampling}), and obtain
We can incorporate this penalty into the Eq.~(\ref{eq:loss-function-training-llm-importance-sampling}), and obtain
@@ -247,8 +247,8 @@ where $\beta$ is the weight of the penalty. This training method is also called
...
@@ -247,8 +247,8 @@ where $\beta$ is the weight of the penalty. This training method is also called
\subsection{Proximal Policy Optimization}
\subsection{Proximal Policy Optimization}
A further improvement to TRPO involves addressing the gradient variance problem outlined in Section\ref{sec:reduce-gradient-variance}.
A further improvement to TRPO involves addressing the gradient variance problem outlined in Section~\ref{sec:reduce-gradient-variance}.
In Eq.(\ref{eq:importance-sampling-with-penalty}), the range of the ratio function is from $[0, +\infty)$, which could lead to high gradient variance in the learning process. To mitigate this problem, clipping is commonly employed to limit the magnitude of importance weights, thereby preventing excessively large updates and promoting stability in the learning process. A clipped version can be given by
In Eq.~(\ref{eq:importance-sampling-with-penalty}), the range of the ratio function is from $[0, +\infty)$, which could lead to high gradient variance in the learning process. To mitigate this problem, clipping is commonly employed to limit the magnitude of importance weights, thereby preventing excessively large updates and promoting stability in the learning process. A clipped version can be given by
@@ -260,7 +260,7 @@ In Eq. (\ref{eq:importance-sampling-with-penalty}), the range of the ratio funct
...
@@ -260,7 +260,7 @@ In Eq. (\ref{eq:importance-sampling-with-penalty}), the range of the ratio funct
\centering
\centering
\input{section3/Figures/figure-ppo-clip}
\input{section3/Figures/figure-ppo-clip}
\caption{
\caption{
PPO clipping at time step $t$. The basic idea can be represented with a hypothesis space as described in Figure\ref{fig:understand-policy-gradient}. The sub-figure (a) shows the case where the advantage $A_{t}$ is positive, suggesting that the output is preferred. Here, the probability of the sampled output is increased within a defined constraint (up to $1+\epsilon$) in the hypothesis space. The sub-figure (b) shows the case where $A_{t}$ is negative, indicating a dispreferred output. In this case, the probability of the sample output is reduced to a minimum (down to $1-\epsilon$) in the hypothesis space.
PPO clipping at time step $t$. The basic idea can be represented with a hypothesis space as described in Figure~\ref{fig:understand-policy-gradient}. The sub-figure (a) shows the case where the advantage $A_{t}$ is positive, suggesting that the output is preferred. Here, the probability of the sampled output is increased within a defined constraint (up to $1+\epsilon$) in the hypothesis space. The sub-figure (b) shows the case where $A_{t}$ is negative, indicating a dispreferred output. In this case, the probability of the sample output is reduced to a minimum (down to $1-\epsilon$) in the hypothesis space.
}
}
\label{fig:ppo-clip}
\label{fig:ppo-clip}
\end{figure*}
\end{figure*}
...
@@ -271,25 +271,25 @@ where the clipping function $\mathrm{Clip}(\cdot)$ can be defined by
...
@@ -271,25 +271,25 @@ where the clipping function $\mathrm{Clip}(\cdot)$ can be defined by
\label{eq:clipping-function}
\label{eq:clipping-function}
\end{eqnarray}
\end{eqnarray}
where the function $\mathrm{bound}(\cdot)$ constrains the ratio function to within the range $[1-\epsilon, 1+\epsilon]$. The clipping imposed by Eq. (\ref{eq:clipping-function}) is illustrated in Figure \ref{fig:ppo-clip}. This training method is also known as proximal policy optimization (PPO) \citep{schulman-etal:2017proximal}, currently the most widely used method for training LLMs with RL. Originally, PPO also introduced an adaptive $\beta$ to dynamically control this penalty. However, this adaptive approach has not been widely applied in training LLMs. This is because this process depends on the expectation of the penalty (i.e., $\mathbb{E}(\mathrm{Penalty})$), which would introduce additional computational overhead. Interested readers can refer to the original literature for more details on this adaptive approach.
where the function $\mathrm{bound}(\cdot)$ constrains the ratio function to within the range $[1-\epsilon, 1+\epsilon]$. The clipping imposed by Eq.~(\ref{eq:clipping-function}) is illustrated in Figure~\ref{fig:ppo-clip}. This training method is also known as proximal policy optimization (PPO) \citep{schulman-etal:2017proximal}, currently the most widely used method for training LLMs with RL. Originally, PPO also introduced an adaptive $\beta$ to dynamically control this penalty. However, this adaptive approach has not been widely applied in training LLMs. This is because this process depends on the expectation of the penalty (i.e., $\mathbb{E}(\mathrm{Penalty})$), which would introduce additional computational overhead. Interested readers can refer to the original literature for more details on this adaptive approach.
% introduce reward models
% introduce reward models
\subsection{Training Reward Models}
\subsection{Training Reward Models}
\label{sec:training-reward-models}
\label{sec:training-reward-models}
While our initial reward function based on output length, as described in Section\ref{sec:policy-gradient}, provides effective signals in the learning process, human preferences are considerably more complex and extend beyond merely preferring longer outputs. For example, in scenarios like a homework assistant, it is essential not only to generate longer outputs but also to avoid redundancy, ensure accuracy, and maintain fluency. This complexity underscores the need for a sophisticated reward modeling technique, moving beyond simple function-based methods to more effectively capture human preferences.
While our initial reward function based on output length, as described in Section~\ref{sec:policy-gradient}, provides effective signals in the learning process, human preferences are considerably more complex and extend beyond merely preferring longer outputs. For example, in scenarios like a homework assistant, it is essential not only to generate longer outputs but also to avoid redundancy, ensure accuracy, and maintain fluency. This complexity underscores the need for a sophisticated reward modeling technique, moving beyond simple function-based methods to more effectively capture human preferences.
Given these complexities, we typically train a reward model, a neural network that maps a pair of input and output token sequences to a scalar value, to capture human preferences. Given an input $\mathbf{x}$ and an output $\mathbf{y}$, the reward is expressed as $\mathrm{Reward}(\mathbf{x},\mathbf{y})$, where $\mathrm{Reward}(\cdot)$ denotes the reward model. There are many ways to implement the reward model. One simple approach is to build the reward model based on a pre-trained LLM, similar to the value model as presented in Section\ref{sec:reduce-gradient-variance}. More specifically, we employ the sequence $\mathrm{seq}_{\mathbf{x},\mathbf{y}}=[\mathbf{x}, \mathbf{y}]$ to serve as the input. We run the LLM on this sequence and obtain a representation from the top-most Transformer layer. Then, we take the representation at the last position of the output and map it to a scalar via linear transformation\footnote{In practice, when employing an LLM for training a reward model, we utilize it primarily as an encoder to encode the sequence $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$ into a representation. Here, the representation from the last position is selected as it encapsulates the semantic information of the entire sequence.}:
Given these complexities, we typically train a reward model, a neural network that maps a pair of input and output token sequences to a scalar value, to capture human preferences. Given an input $\mathbf{x}$ and an output $\mathbf{y}$, the reward is expressed as $\mathrm{Reward}(\mathbf{x},\mathbf{y})$, where $\mathrm{Reward}(\cdot)$ denotes the reward model. There are many ways to implement the reward model. One simple approach is to build the reward model based on a pre-trained LLM, similar to the value model as presented in Section~\ref{sec:reduce-gradient-variance}. More specifically, we employ the sequence $\mathrm{seq}_{\mathbf{x},\mathbf{y}}=[\mathbf{x}, \mathbf{y}]$ to serve as the input. We run the LLM on this sequence and obtain a representation from the top-most Transformer layer. Then, we take the representation at the last position of the output and map it to a scalar via linear transformation\footnote{In practice, when employing an LLM for training a reward model, we utilize it primarily as an encoder to encode the sequence $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$ into a representation. Here, the representation from the last position is selected as it encapsulates the semantic information of the entire sequence.}:
where $\mathbf{W}_{r}$ is a $d \times1$ linear mapping matrix, and $\phi$ represents the parameters of the reward model, which includes both the parameters of the LLM and the $\mathbf{W}_{r}$. This architecture of the reward model is illustrated in Figure\ref{fig:reward-model}.
where $\mathbf{W}_{r}$ is a $d \times1$ linear mapping matrix, and $\phi$ represents the parameters of the reward model, which includes both the parameters of the LLM and the $\mathbf{W}_{r}$. This architecture of the reward model is illustrated in Figure~\ref{fig:reward-model}.
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
\input{section3/Figures/figure-reward-model}
\input{section3/Figures/figure-reward-model}
\caption{
\caption{
Architecture of the reward model based on an LLM. Unlike the value model depicted in Figure\ref{fig:architecture-value-function}, this model extracts the representation from the last position of the output $\mathbf{y}$ to represent the entire sequence $[\mathbf{x}, \mathbf{y}]$. This representation is then mapped to a scalar through a linear transformation, which serves as the reward for the output.
Architecture of the reward model based on an LLM. Unlike the value model depicted in Figure~\ref{fig:architecture-value-function}, this model extracts the representation from the last position of the output $\mathbf{y}$ to represent the entire sequence $[\mathbf{x}, \mathbf{y}]$. This representation is then mapped to a scalar through a linear transformation, which serves as the reward for the output.
}
}
\label{fig:reward-model}
\label{fig:reward-model}
\end{figure*}
\end{figure*}
...
@@ -316,7 +316,7 @@ One simple and widely used model for describing such pairwise comparisons is the
...
@@ -316,7 +316,7 @@ One simple and widely used model for describing such pairwise comparisons is the
An example of loss computation using the Bradley-Terry model. Given an input $\mathbf{x}$ and two outputs $(\mathbf{y}_{a}, \mathbf{y}_{b})$, where $\mathbf{y}_a \succ\mathbf{y}_b$, we initially obtain rewards $R(\mathbf{x}, \mathbf{y}_a)$ and $R(\mathbf{x}, \mathbf{y}_b)$ for two sequences $\mathrm{seq}_{\mathbf{x},\mathbf{y}_{a}}$ and $\mathrm{seq}_{\mathbf{x},\mathbf{y}_{b}}$. These rewards are subsequently used to compute the loss as described in Eq.(\ref{eq:pairwise-reward-loss-expectation}). Note that, for efficiency, both sequences are typically processed in a single batch during one forward pass to simultaneously obtain $R(\mathbf{x}, \mathbf{y}_a)$ and $R(\mathbf{x}, \mathbf{y}_b)$. In fact, this training approach is also common in Siamese networks \citep{bromley-etal:1993signature}.
An example of loss computation using the Bradley-Terry model. Given an input $\mathbf{x}$ and two outputs $(\mathbf{y}_{a}, \mathbf{y}_{b})$, where $\mathbf{y}_a \succ\mathbf{y}_b$, we initially obtain rewards $R(\mathbf{x}, \mathbf{y}_a)$ and $R(\mathbf{x}, \mathbf{y}_b)$ for two sequences $\mathrm{seq}_{\mathbf{x},\mathbf{y}_{a}}$ and $\mathrm{seq}_{\mathbf{x},\mathbf{y}_{b}}$. These rewards are subsequently used to compute the loss as described in Eq.~(\ref{eq:pairwise-reward-loss-expectation}). Note that, for efficiency, both sequences are typically processed in a single batch during one forward pass to simultaneously obtain $R(\mathbf{x}, \mathbf{y}_a)$ and $R(\mathbf{x}, \mathbf{y}_b)$. In fact, this training approach is also common in Siamese networks \citep{bromley-etal:1993signature}.
}
}
\label{fig:training-reward-models}
\label{fig:training-reward-models}
\end{figure*}
\end{figure*}
...
@@ -333,9 +333,9 @@ where $(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)$ is drawn from a human-annotated d
...
@@ -333,9 +333,9 @@ where $(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)$ is drawn from a human-annotated d
\hat{\phi}& = &\argmin_{\phi}\mathcal{L}_r(\phi)
\hat{\phi}& = &\argmin_{\phi}\mathcal{L}_r(\phi)
\end{eqnarray}
\end{eqnarray}
Since the reward model itself is also an LLM, we can directly reuse the Transformer training procedure to optimize the reward model. The difference from training a standard LLM is that we only need to replace the cross-entropy loss with the pairwise comparison loss as illustrated in Figure\ref{fig:training-reward-models}. After the training of the reward model, we can apply the trained reward model $R_{\hat{\phi}}(\cdot)$ to supervise the target LLM for alignment.
Since the reward model itself is also an LLM, we can directly reuse the Transformer training procedure to optimize the reward model. The difference from training a standard LLM is that we only need to replace the cross-entropy loss with the pairwise comparison loss as illustrated in Figure~\ref{fig:training-reward-models}. After the training of the reward model, we can apply the trained reward model $R_{\hat{\phi}}(\cdot)$ to supervise the target LLM for alignment.
It is worth noting that although we train the reward model to perform pairwise ranking, we apply it to score each input-output pair independently during the alignment process. The pairwise ranking objective ensures that the reward model is sensitive to subtle differences between outputs, but we rely on the continuous scores produced by the reward model to guide the optimization of the LLM. An advantage of this approach is that we can choose from or combine various ranking loss functions and still apply the resulting reward models in the same way as we have done in Section\ref{sec:improved-reward-generalization}. However, a challenge arises with this method: the reward model can provide only sparse rewards; that is, it offers a delayed reward rather than an intermediate one. Hence, in this case, we can obtain
It is worth noting that although we train the reward model to perform pairwise ranking, we apply it to score each input-output pair independently during the alignment process. The pairwise ranking objective ensures that the reward model is sensitive to subtle differences between outputs, but we rely on the continuous scores produced by the reward model to guide the optimization of the LLM. An advantage of this approach is that we can choose from or combine various ranking loss functions and still apply the resulting reward models in the same way as we have done in Section~\ref{sec:improved-reward-generalization}. However, a challenge arises with this method: the reward model can provide only sparse rewards; that is, it offers a delayed reward rather than an intermediate one. Hence, in this case, we can obtain
\begin{eqnarray}
\begin{eqnarray}
r_{t}& = &
r_{t}& = &
\begin{cases}
\begin{cases}
...
@@ -344,7 +344,7 @@ It is worth noting that although we train the reward model to perform pairwise r
...
@@ -344,7 +344,7 @@ It is worth noting that although we train the reward model to perform pairwise r
\end{cases}
\end{cases}
\end{eqnarray}
\end{eqnarray}
To address this challenge, we can typically incorporate shaping rewards into the learning process \citep{wu-etal:2023fine} or train a process reward model by annotating process preference data \citep{lightman-etal:2023let}. Please refer to Section\ref{sec:step-by-step-verification} for more details.
To address this challenge, we can typically incorporate shaping rewards into the learning process \citep{wu-etal:2023fine} or train a process reward model by annotating process preference data \citep{lightman-etal:2023let}. Please refer to Section~\ref{sec:step-by-step-verification} for more details.
\begin{figure}[!t]
\begin{figure}[!t]
...
@@ -364,7 +364,7 @@ Up to this point, we have used numerous examples to discuss how to use RL to tra
...
@@ -364,7 +364,7 @@ Up to this point, we have used numerous examples to discuss how to use RL to tra
\begin{itemize}
\begin{itemize}
\item\vspace{0.3em}\textbf{Reward Model} (denoted by $R_{\phi}(\cdot)$ where $\phi$ denotes the parameters). The reward model learns from human preference data to predict the reward for each pair of input and output token sequences. It is typically initialized with an SFT LLM or a pre-trained LLM.
\item\vspace{0.3em}\textbf{Reward Model} (denoted by $R_{\phi}(\cdot)$ where $\phi$ denotes the parameters). The reward model learns from human preference data to predict the reward for each pair of input and output token sequences. It is typically initialized with an SFT LLM or a pre-trained LLM.
\item\vspace{0.3em}\textbf{Value Model} or \textbf{Value Function} (denoted by $V_{\omega}(\cdot)$ where $\omega$ denotes the parameters). The value model receives rewards from the reward model and is trained to predict the expected sum of rewards. It is generally initialized with a reward model or an SFT LLM.
\item\vspace{0.3em}\textbf{Value Model} or \textbf{Value Function} (denoted by $V_{\omega}(\cdot)$ where $\omega$ denotes the parameters). The value model receives rewards from the reward model and is trained to predict the expected sum of rewards. It is generally initialized with a reward model or an SFT LLM.
\item\vspace{0.3em}\textbf{Reference Model} or \textbf{Old Policy Model} (denoted by $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot)$ where $\theta_{\mathrm{ref}}$ denotes the parameters). The reference model is the baseline LLM that serves as a starting point for policy training. Unlike the discussion in Section\ref{sec:importance-sampling}, when computing the penalty in RLHF, we typically use the previous version of the model or a model trained without human feedback to serve as the reference policy model, making a more stable learning process.
\item\vspace{0.3em}\textbf{Reference Model} or \textbf{Old Policy Model} (denoted by $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot)$ where $\theta_{\mathrm{ref}}$ denotes the parameters). The reference model is the baseline LLM that serves as a starting point for policy training. Unlike the discussion in Section~\ref{sec:importance-sampling}, when computing the penalty in RLHF, we typically use the previous version of the model or a model trained without human feedback to serve as the reference policy model, making a more stable learning process.
\item\vspace{0.3em}\textbf{Policy Model} (denoted by $\mathrm{Pr}_{\theta}(\cdot)$ where $\theta$ denotes the parameters). Given its context, this policy governs how the LLM decides the most appropriate next token. It is trained under the supervision of both the reward model and the value model.
\item\vspace{0.3em}\textbf{Policy Model} (denoted by $\mathrm{Pr}_{\theta}(\cdot)$ where $\theta$ denotes the parameters). Given its context, this policy governs how the LLM decides the most appropriate next token. It is trained under the supervision of both the reward model and the value model.
@@ -80,9 +80,9 @@ While learning from AI feedback is highly scalable and generally objective, this
...
@@ -80,9 +80,9 @@ While learning from AI feedback is highly scalable and generally objective, this
\subsubsection{Reward Shaping}
\subsubsection{Reward Shaping}
\label{sec:reward-shaping}
\label{sec:reward-shaping}
As discussed in Section\ref{sec:training-reward-models}, while the reward model is effective in capturing human preferences, it provides sparse rewards. These rewards, known as delayed rewards, are received only at the end of the output generation process, as opposed to intermediate rewards that would be distributed continuously throughout it.
As discussed in Section~\ref{sec:training-reward-models}, while the reward model is effective in capturing human preferences, it provides sparse rewards. These rewards, known as delayed rewards, are received only at the end of the output generation process, as opposed to intermediate rewards that would be distributed continuously throughout it.
In fact, dealing with sparse rewards has long been a concern in RL, and has been one of the challenges in many practical applications. For example, robotics often needs to shape the reward function to ease optimization rather than relying solely on end-of-sequence rewards. Various methods have been developed to address this issue. One common approach is reward shaping, where the original function is modified to include intermediate rewards, thereby providing more immediate feedback. Here, the intermediate reward is often an indirect signal for improving the delayed reward. For example, as shown in Figure\ref{fig:example-shaping-rewards}, setting a length-based reward as an intermediate reward encourages the generation of more content, potentially increasing the overall quality of the final output as assessed by the delayed reward from the reward model. Additional examples of reward shaping in training LLMs can be found in \citet{kumar-etal:2024training}. In this way, we can obtain
In fact, dealing with sparse rewards has long been a concern in RL, and has been one of the challenges in many practical applications. For example, robotics often needs to shape the reward function to ease optimization rather than relying solely on end-of-sequence rewards. Various methods have been developed to address this issue. One common approach is reward shaping, where the original function is modified to include intermediate rewards, thereby providing more immediate feedback. Here, the intermediate reward is often an indirect signal for improving the delayed reward. For example, as shown in Figure~\ref{fig:example-shaping-rewards}, setting a length-based reward as an intermediate reward encourages the generation of more content, potentially increasing the overall quality of the final output as assessed by the delayed reward from the reward model. Additional examples of reward shaping in training LLMs can be found in \citet{kumar-etal:2024training}. In this way, we can obtain
where $\Phi(\cdot)$ is called the potential value function. If we define $\Phi(\cdot)$ as the common value function and substitute Eq. (\ref{eq:shaping-reward-function}) into Eq. (\ref{eq:transformed-reward-function}), we obtain
where $\Phi(\cdot)$ is called the potential value function. If we define $\Phi(\cdot)$ as the common value function and substitute Eq.~(\ref{eq:shaping-reward-function}) into Eq.~(\ref{eq:transformed-reward-function}), we obtain
\begin{eqnarray}
\begin{eqnarray}
r'_{t}& = & r_{t} + \gamma V_{t+1} - V_{t}
r'_{t}& = & r_{t} + \gamma V_{t+1} - V_{t}
\end{eqnarray}
\end{eqnarray}
...
@@ -123,9 +123,9 @@ In addition to reward shaping, another method to address the sparse reward issue
...
@@ -123,9 +123,9 @@ In addition to reward shaping, another method to address the sparse reward issue
\label{fig:improve-reward-generalization}
\label{fig:improve-reward-generalization}
\end{figure*}
\end{figure*}
As discussed in Section\ref{sec:training-reward-models}, reward models are trained using preference data and subsequently used to optimize LLMs. However, a problem arises: the data distribution during LLM optimization may differ from the distribution of the preference data, making it challenging for the reward model to generalize to unseen input-output pairs.
As discussed in Section~\ref{sec:training-reward-models}, reward models are trained using preference data and subsequently used to optimize LLMs. However, a problem arises: the data distribution during LLM optimization may differ from the distribution of the preference data, making it challenging for the reward model to generalize to unseen input-output pairs.
A well-known failure mode associated with this problem is commonly referred to as \textit{overoptimization} or \textit{reward hacking}, where the optimization stage improves the reward model score but deteriorates the alignment with true rewards \citep{gao-etal:2023scaling,eisenstein-etal:2023helping}. This mode occurs because the reward model may incorrectly assign high rewards to unseen input-output pairs, leading the LLM to learn and optimize for behaviors that do not truly align with the desired behaviors and objectives. For example, consider a scenario from Section\ref{sec:policy-gradient} where the reward model is designed to favor informative and accurate outputs for a homework assistant. However, if the reward model fails to generalize to an overly verbose output and assigns a high reward to it (perhaps due to its length or certain keywords), the LLM may learn to prioritize generating a long output, which is not actually more informative but receives a higher reward according to the reward model. Consequently, the LLM becomes misaligned with its true objectives, delivering concise and relevant information, because it optimizes for the incorrect reward signal from the reward model with weak generalization.
A well-known failure mode associated with this problem is commonly referred to as \textit{overoptimization} or \textit{reward hacking}, where the optimization stage improves the reward model score but deteriorates the alignment with true rewards \citep{gao-etal:2023scaling,eisenstein-etal:2023helping}. This mode occurs because the reward model may incorrectly assign high rewards to unseen input-output pairs, leading the LLM to learn and optimize for behaviors that do not truly align with the desired behaviors and objectives. For example, consider a scenario from Section~\ref{sec:policy-gradient} where the reward model is designed to favor informative and accurate outputs for a homework assistant. However, if the reward model fails to generalize to an overly verbose output and assigns a high reward to it (perhaps due to its length or certain keywords), the LLM may learn to prioritize generating a long output, which is not actually more informative but receives a higher reward according to the reward model. Consequently, the LLM becomes misaligned with its true objectives, delivering concise and relevant information, because it optimizes for the incorrect reward signal from the reward model with weak generalization.
Addressing this generalization problem is challenging, and no mature solution exists yet. The ideal approach would be to develop an oracle reward model that perfectly captures the true objectives of the task and generalizes across all input-output pairs during LLM optimization. However, creating such a model is extremely difficult due to the complexity of the real-world environment, as well as the challenge of collecting sufficient preference data. Instead, a more practical approach is to combine multiple reward models, improving generalization and providing more accurate rewards \citep{coste-etal:2024reward}.
Addressing this generalization problem is challenging, and no mature solution exists yet. The ideal approach would be to develop an oracle reward model that perfectly captures the true objectives of the task and generalizes across all input-output pairs during LLM optimization. However, creating such a model is extremely difficult due to the complexity of the real-world environment, as well as the challenge of collecting sufficient preference data. Instead, a more practical approach is to combine multiple reward models, improving generalization and providing more accurate rewards \citep{coste-etal:2024reward}.
where $R_k(\cdot)$ is the $k$-th reward model in the ensemble, $w_k$ is the weight of $R_k(\cdot)$, and $K$ is the number of reward models. This combined reward can then be used to supervise the training of a policy. In fact, there are many ways to combine different models; for example, one can make predictions using Bayesian model averaging or develop a fusion network to learn to combine the predictions from different models. Alternatively, one can frame this task as a multi-objective optimization problem and use multiple reward models to train the policy simultaneously. These methods have been intensively discussed in the literature on optimization and machine learning \citep{miettinen:1999nonlinear,Bishop:2006}.
where $R_k(\cdot)$ is the $k$-th reward model in the ensemble, $w_k$ is the weight of $R_k(\cdot)$, and $K$ is the number of reward models. This combined reward can then be used to supervise the training of a policy. In fact, there are many ways to combine different models; for example, one can make predictions using Bayesian model averaging or develop a fusion network to learn to combine the predictions from different models. Alternatively, one can frame this task as a multi-objective optimization problem and use multiple reward models to train the policy simultaneously. These methods have been intensively discussed in the literature on optimization and machine learning \citep{miettinen:1999nonlinear,Bishop:2006}.
On the other hand, to improve the generalization of reward models, it is important to prevent overfitting to preference data. Reward models are typically trained on an SFT LLM, which has a strong generalization capability. However, training the LLM with a large amount of preference data could lead to overfitting, which may ultimately reduce generalization performance. A common strategy to mitigate this issue is to employ a parameter freezing technique, which helps preserve the original features of the LLM while learning the reward model. Another practical approach is to add a regularization term to the preference learning loss function, which helps regulate the features of the LLM \citep{yang-etal:2024regularizing}. For example, we can use a simple SFT loss as regularization by adding a term that maximizes the probability of the preferred output $\mathbf{y}_a$ to Eq.(\ref{eq:pairwise-reward-loss-expectation}):
On the other hand, to improve the generalization of reward models, it is important to prevent overfitting to preference data. Reward models are typically trained on an SFT LLM, which has a strong generalization capability. However, training the LLM with a large amount of preference data could lead to overfitting, which may ultimately reduce generalization performance. A common strategy to mitigate this issue is to employ a parameter freezing technique, which helps preserve the original features of the LLM while learning the reward model. Another practical approach is to add a regularization term to the preference learning loss function, which helps regulate the features of the LLM \citep{yang-etal:2024regularizing}. For example, we can use a simple SFT loss as regularization by adding a term that maximizes the probability of the preferred output $\mathbf{y}_a$ to Eq.~(\ref{eq:pairwise-reward-loss-expectation}):
where $\alpha$ is a balancing factor. We further illustrate the parameter freezing and regularization methods in Figure\ref{fig:improve-reward-generalization}. Note that the regularization term applies only to the LLM. That is, when optimizing this term, only the parameters of the LLM are updated, while the parameters of the reward linear map remain fixed. Therefore, in this equation, the parameter associated with the regularization term is $\theta$, which specifically refers to the parameters of the LLM.
where $\alpha$ is a balancing factor. We further illustrate the parameter freezing and regularization methods in Figure~\ref{fig:improve-reward-generalization}. Note that the regularization term applies only to the LLM. That is, when optimizing this term, only the parameters of the LLM are updated, while the parameters of the reward linear map remain fixed. Therefore, in this equation, the parameter associated with the regularization term is $\theta$, which specifically refers to the parameters of the LLM.
\begin{figure}[!t]
\begin{figure}[!t]
\centering
\centering
...
@@ -152,7 +152,7 @@ where $\alpha$ is a balancing factor. We further illustrate the parameter freezi
...
@@ -152,7 +152,7 @@ where $\alpha$ is a balancing factor. We further illustrate the parameter freezi
\subsubsection{Generative Reward Models}
\subsubsection{Generative Reward Models}
\label{sec:generative-reward-models}
\label{sec:generative-reward-models}
Reward models are typically trained as discriminative models to assign numerical rewards to outputs and classify them as preferred or dispreferred. However, this method does not leverage the text-generation capabilities for which LLMs are fundamentally designed \citep{zhang-etal:zhang2024generative,wang-etal:wang2025gram}. For example, the discriminative reward model cannot perform CoT reasoning. To address this, an LLM can alternatively be employed as a reward model, thus endowing it with the ability to engage in text generation and reasoning, as depicted in Figure\ref{fig:generative-reward-model-architecture}. This model works as follows. First, we input a prompt $\mathbf{c}$, along with the tuple $(\mathbf{x},\mathbf{y}_{a},\mathbf{y}_{b})$, to the LLM. The prompt is a description of the task, as demonstrated in the example below.
Reward models are typically trained as discriminative models to assign numerical rewards to outputs and classify them as preferred or dispreferred. However, this method does not leverage the text-generation capabilities for which LLMs are fundamentally designed \citep{zhang-etal:zhang2024generative,wang-etal:wang2025gram}. For example, the discriminative reward model cannot perform CoT reasoning. To address this, an LLM can alternatively be employed as a reward model, thus endowing it with the ability to engage in text generation and reasoning, as depicted in Figure~\ref{fig:generative-reward-model-architecture}. This model works as follows. First, we input a prompt $\mathbf{c}$, along with the tuple $(\mathbf{x},\mathbf{y}_{a},\mathbf{y}_{b})$, to the LLM. The prompt is a description of the task, as demonstrated in the example below.
\vspace{0.5em}
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
\begin{tcolorbox}[frame empty]
...
@@ -168,7 +168,7 @@ You are given two outputs to an input. Evaluate which output is better based on
...
@@ -168,7 +168,7 @@ You are given two outputs to an input. Evaluate which output is better based on
\end{tcolorbox}
\end{tcolorbox}
\vspace{0.5em}
\vspace{0.5em}
Then, the LLM predicts subsequent tokens based on this input sequence. Let $w$ be the label token predicted by the LLM and $w^{*}$ be the annotated preference label. If $w^{*}=\text{A}$, it indicates a preference for $\mathbf{y}_a$ over $\mathbf{y}_b$; if $w^{*}=\text{B}$, then $\mathbf{y}_b$ is preferred. Note that here $\mathbf{y}_a$ and $\mathbf{y}_b$ do not have a pre-defined preference relationship as described in Section\ref{sec:training-reward-models}. Their relationship is instead represented by the label token.
Then, the LLM predicts subsequent tokens based on this input sequence. Let $w$ be the label token predicted by the LLM and $w^{*}$ be the annotated preference label. If $w^{*}=\text{A}$, it indicates a preference for $\mathbf{y}_a$ over $\mathbf{y}_b$; if $w^{*}=\text{B}$, then $\mathbf{y}_b$ is preferred. Note that here $\mathbf{y}_a$ and $\mathbf{y}_b$ do not have a pre-defined preference relationship as described in Section~\ref{sec:training-reward-models}. Their relationship is instead represented by the label token.
The loss function can be defined as the log-probability of predicting the annotated preference label:
The loss function can be defined as the log-probability of predicting the annotated preference label:
\begin{eqnarray}
\begin{eqnarray}
...
@@ -208,7 +208,7 @@ We can then define a new loss function for training the generative reward model
...
@@ -208,7 +208,7 @@ We can then define a new loss function for training the generative reward model
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (also known as LLM-as-a-judge), as demonstrated in Section\ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (also known as LLM-as-a-judge), as demonstrated in Section~\ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches. The basic idea is to use only the ranking capability of generative reward models to derive rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches. The basic idea is to use only the ranking capability of generative reward models to derive rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
...
@@ -450,7 +450,7 @@ where $N$ denotes the number of evaluation instances.
...
@@ -450,7 +450,7 @@ where $N$ denotes the number of evaluation instances.
Accurate advantage estimation is critical in RL, particularly when optimizing the policy model to truly reflect the potential benefits of different actions (or tokens in training LLMs). In this subsection, we will delve into methods for improving advantage estimation, providing more accurate advantages in the process of optimizing the policy model.
Accurate advantage estimation is critical in RL, particularly when optimizing the policy model to truly reflect the potential benefits of different actions (or tokens in training LLMs). In this subsection, we will delve into methods for improving advantage estimation, providing more accurate advantages in the process of optimizing the policy model.
Let us first review the Monte Carlo-based advantage estimation. As discussed in Section\ref{sec:reduce-gradient-variance}, outputs are sampled and their values computed. The advantage can be obtained by comparing the actual received rewards with the predicted values at time step $t$:
Let us first review the Monte Carlo-based advantage estimation. As discussed in Section~\ref{sec:reduce-gradient-variance}, outputs are sampled and their values computed. The advantage can be obtained by comparing the actual received rewards with the predicted values at time step $t$:
\begin{eqnarray}
\begin{eqnarray}
A_{t}& = &\sum_{k=t}^{T}r_{k} - V_{t}
A_{t}& = &\sum_{k=t}^{T}r_{k} - V_{t}
\label{eq:advantage-estimation-monte-carlo}
\label{eq:advantage-estimation-monte-carlo}
...
@@ -458,15 +458,15 @@ Let us first review the Monte Carlo-based advantage estimation. As discussed in
...
@@ -458,15 +458,15 @@ Let us first review the Monte Carlo-based advantage estimation. As discussed in
While the Monte Carlo-based estimation of advantage provides a relatively stable training process, there are two main challenges associated with it:
While the Monte Carlo-based estimation of advantage provides a relatively stable training process, there are two main challenges associated with it:
\begin{itemize}
\begin{itemize}
\item Sampling an entire output for each input is necessary. In some cases where immediate rewards are available, it could be more efficient to update the policy model with partial output. Unfortunately, when using Eq.(\ref{eq:advantage-estimation-monte-carlo}) for advantage estimation, this becomes unfeasible. This is because we must compute the term $\sum_{k=t}^T r_{k}$, which requires the rewards of the entire output.
\item Sampling an entire output for each input is necessary. In some cases where immediate rewards are available, it could be more efficient to update the policy model with partial output. Unfortunately, when using Eq.~(\ref{eq:advantage-estimation-monte-carlo}) for advantage estimation, this becomes unfeasible. This is because we must compute the term $\sum_{k=t}^T r_{k}$, which requires the rewards of the entire output.
\item This method still results in high variance. Since a single sample operation might be influenced by random factors, the estimated results may not be stable, as illustrated in Figure\ref{fig:monte-carlo-advantage-issue}.
\item This method still results in high variance. Since a single sample operation might be influenced by random factors, the estimated results may not be stable, as illustrated in Figure~\ref{fig:monte-carlo-advantage-issue}.
\end{itemize}
\end{itemize}
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
\input{section4/Figures/figure-issue-monte-carlo}
\input{section4/Figures/figure-issue-monte-carlo}
\caption{
\caption{
Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq.(\ref{eq:length-based-reward-function}).
Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq.~(\ref{eq:length-based-reward-function}).
Typically, if the value model is well-trained, it would predict an average $V_{t}$ of 6 at time step $t$, based on most outputs having around 60 tokens in length. However, for outlier outputs like $\mathbf{y}_{d,t:T}$, which might extend up to 300 tokens, the advantage estimation can exhibit high variance. For instance, the advantage for such an outlier could be computed as $A_{t}(\mathbf{x},\mathbf{y}_{d,<t},y_{d,t})=\sum_{k=t}^T r_{k}- V_t =30-6=24$, a stark contrast to another output, say $\mathbf{y}_{1,t:T}$, which is closer to the average length, where the advantage might be $A_{t}(\mathbf{x},\mathbf{y}_{1,<t},y_{1,t})=-1$. This discrepancy leads to high gradient variance, potentially destabilizing the learning process.
Typically, if the value model is well-trained, it would predict an average $V_{t}$ of 6 at time step $t$, based on most outputs having around 60 tokens in length. However, for outlier outputs like $\mathbf{y}_{d,t:T}$, which might extend up to 300 tokens, the advantage estimation can exhibit high variance. For instance, the advantage for such an outlier could be computed as $A_{t}(\mathbf{x},\mathbf{y}_{d,<t},y_{d,t})=\sum_{k=t}^T r_{k}- V_t =30-6=24$, a stark contrast to another output, say $\mathbf{y}_{1,t:T}$, which is closer to the average length, where the advantage might be $A_{t}(\mathbf{x},\mathbf{y}_{1,<t},y_{1,t})=-1$. This discrepancy leads to high gradient variance, potentially destabilizing the learning process.
}
}
\label{fig:monte-carlo-advantage-issue}
\label{fig:monte-carlo-advantage-issue}
...
@@ -518,14 +518,14 @@ In fact, this is entirely feasible, and one example of such an approach is \cite
...
@@ -518,14 +518,14 @@ In fact, this is entirely feasible, and one example of such an approach is \cite
\label{eq:advantage-grpo}
\label{eq:advantage-grpo}
\end{eqnarray}
\end{eqnarray}
where $\mathrm{Mean}(\cdot)$ and $\mathrm{Std}(\cdot)$ represent the mean and standard deviation functions, respectively. Note that the advantage computation used here differs slightly from that in PPO. In this case, this computed advantage is applied to each time step, whereas in the traditional PPO, we separately compute an advantage for each time step. At this point, we can also use the process reward model to supplement the advantage computation, allowing us to compute an advantage specific to each time step. A simple way to achieve this is by implementing Eq. (\ref{eq:advantage-grpo}) at each time step, where the rewards are computed using the process reward model. As a result, the objective for GRPO can be defined according to Eq. (\ref{eq:ppo-loss}):
where $\mathrm{Mean}(\cdot)$ and $\mathrm{Std}(\cdot)$ represent the mean and standard deviation functions, respectively. Note that the advantage computation used here differs slightly from that in PPO. In this case, this computed advantage is applied to each time step, whereas in the traditional PPO, we separately compute an advantage for each time step. At this point, we can also use the process reward model to supplement the advantage computation, allowing us to compute an advantage specific to each time step. A simple way to achieve this is by implementing Eq.~(\ref{eq:advantage-grpo}) at each time step, where the rewards are computed using the process reward model. As a result, the objective for GRPO can be defined according to Eq.~(\ref{eq:ppo-loss}):
In addition to optimizing the advantage computation, GRPO modifies the penalty term to enhance the optimization objective further. However, since our focus here is on advantage estimation, we will not delve into its details. Interested readers can refer to the GRPO paper for further details. The workflow of GRPO is illustrated in Figure\ref{fig:workflow-grpo}, showcasing how these modifications integrate into the training process.
In addition to optimizing the advantage computation, GRPO modifies the penalty term to enhance the optimization objective further. However, since our focus here is on advantage estimation, we will not delve into its details. Interested readers can refer to the GRPO paper for further details. The workflow of GRPO is illustrated in Figure~\ref{fig:workflow-grpo}, showcasing how these modifications integrate into the training process.
Moreover, there are other methods like GRPO that design advantage estimation without relying on a value model for training LLMs, such as those discussed in \citet{li-etal:2023remax} and \citet{hu:2025reinforce++}.
Moreover, there are other methods like GRPO that design advantage estimation without relying on a value model for training LLMs, such as those discussed in \citet{li-etal:2023remax} and \citet{hu:2025reinforce++}.
...
@@ -537,7 +537,7 @@ Efficiency is a critical consideration for many practical applications of RL, an
...
@@ -537,7 +537,7 @@ Efficiency is a critical consideration for many practical applications of RL, an
In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.
In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.
\subsubsection{Dynamic Sampling}
\subsubsection{Dynamic Sampling}
As mentioned in Section\ref{sec:policy-gradient}, training LLMs with RL typically requires sampling for each sample $\mathbf{x}$ in $\mathcal{S}_x$, which introduces significant computational time overhead. This becomes particularly challenging in large-scale RL scenarios, where thousands of training steps need to be conducted, such as in DeepSeek-R1 \citep{guo:2025deepseek}.
As mentioned in Section~\ref{sec:policy-gradient}, training LLMs with RL typically requires sampling for each sample $\mathbf{x}$ in $\mathcal{S}_x$, which introduces significant computational time overhead. This becomes particularly challenging in large-scale RL scenarios, where thousands of training steps need to be conducted, such as in DeepSeek-R1 \citep{guo:2025deepseek}.
From the perspective of LLM inference, there are many methods to reduce the time overhead of sampling: since the sampling process is implemented by LLM inference, we have reason to believe that any method that accelerates inference can also be applied to reduce the time required for sampling.
From the perspective of LLM inference, there are many methods to reduce the time overhead of sampling: since the sampling process is implemented by LLM inference, we have reason to believe that any method that accelerates inference can also be applied to reduce the time required for sampling.
Examples include KV-Cache \citep{pope-etal:2023efficiently}, quantization \citep{zhao-etal:2024atom}, and speculative decoding \citep{chen-etal:2023accelerating}. However, in this section, we will not discuss these methods in detail, as there are numerous approaches, each addressing different aspects of LLM inference. We refer the interested readers to these papers for more details. Instead, we will focus on one particular method that aims to reduce sampling overhead from the perspective of optimizing RL for training LLMs.
Examples include KV-Cache \citep{pope-etal:2023efficiently}, quantization \citep{zhao-etal:2024atom}, and speculative decoding \citep{chen-etal:2023accelerating}. However, in this section, we will not discuss these methods in detail, as there are numerous approaches, each addressing different aspects of LLM inference. We refer the interested readers to these papers for more details. Instead, we will focus on one particular method that aims to reduce sampling overhead from the perspective of optimizing RL for training LLMs.
...
@@ -560,7 +560,7 @@ A concept closely related to the discussion here is sample efficiency. This topi
...
@@ -560,7 +560,7 @@ A concept closely related to the discussion here is sample efficiency. This topi
An additional computational overhead in training LLMs with RL is the frequent need to call the reward model to compute rewards. Furthermore, to ensure the reliability and generalizability of the reward model, we often scale it to a large size \citep{gao-etal:2023scaling}, which increases the computational time for reward computations.
An additional computational overhead in training LLMs with RL is the frequent need to call the reward model to compute rewards. Furthermore, to ensure the reliability and generalizability of the reward model, we often scale it to a large size \citep{gao-etal:2023scaling}, which increases the computational time for reward computations.
One approach for mitigating this issue is to explore rule-based rewards, for example, as discussed in Section\ref{sec:policy-gradient}, the length of the outputs could be utilized as a reward. Instead of relying on large-scale, resource-intensive models, rule-based rewards assign rewards using predefined and task-specific rules. These rules, often derived from expert knowledge or task-specific requirements, are computationally inexpensive and provide fast, effective feedback to the policy model. By replacing or complementing traditional reward models with rule-based approaches, we can reduce both the time and computational cost associated with reward computations while still offering meaningful guidance for the learning process. This approach is also based on the understanding that, in some tasks, human preferences are simple to describe and do not require the complexity of training a reward model. Instead, we can use rules to express these preferences efficiently.
One approach for mitigating this issue is to explore rule-based rewards, for example, as discussed in Section~\ref{sec:policy-gradient}, the length of the outputs could be utilized as a reward. Instead of relying on large-scale, resource-intensive models, rule-based rewards assign rewards using predefined and task-specific rules. These rules, often derived from expert knowledge or task-specific requirements, are computationally inexpensive and provide fast, effective feedback to the policy model. By replacing or complementing traditional reward models with rule-based approaches, we can reduce both the time and computational cost associated with reward computations while still offering meaningful guidance for the learning process. This approach is also based on the understanding that, in some tasks, human preferences are simple to describe and do not require the complexity of training a reward model. Instead, we can use rules to express these preferences efficiently.
Here, in addition to the previously mentioned length-based rewards, we further demonstrate two examples of rule-based rewards. First, considering the output format, we can design rewards that encourage the policy model to generate outputs adhering to specific formats. For example, in response to the input ``Give me three tips to improve my accuracy in solving math problems,'' we can assign rewards based on whether the output can be parsed as JSON correctly and include ``tip1'', ``tip2'', and ``tip3'' as the keys.
Here, in addition to the previously mentioned length-based rewards, we further demonstrate two examples of rule-based rewards. First, considering the output format, we can design rewards that encourage the policy model to generate outputs adhering to specific formats. For example, in response to the input ``Give me three tips to improve my accuracy in solving math problems,'' we can assign rewards based on whether the output can be parsed as JSON correctly and include ``tip1'', ``tip2'', and ``tip3'' as the keys.
% Example 1
% Example 1
...
@@ -584,7 +584,7 @@ Another lightweight reward method is to use a reward model with fewer parameters
...
@@ -584,7 +584,7 @@ Another lightweight reward method is to use a reward model with fewer parameters
\label{fig:comparison-rl-dpo}
\label{fig:comparison-rl-dpo}
\end{figure*}
\end{figure*}
Although learning reward models is a standard step in RL, it makes the entire training process much more complex than supervised training. Training a reliable reward model is itself not an easy task and a poorly trained reward model can greatly affect the outcome of policy learning. We now consider an alternative method, called direct preference optimization (DPO), which simplifies the training framework by eliminating the need to explicitly model rewards \citep{rafailov:2023direct}. This method directly optimizes the policy model based on user preferences rather than developing a separate reward model. As a result, we can achieve human preference alignment in a supervised learning-like fashion. Figure\ref{fig:comparison-rl-dpo} shows a comparison of the standard PPO method and the DPO method.
Although learning reward models is a standard step in RL, it makes the entire training process much more complex than supervised training. Training a reliable reward model is itself not an easy task and a poorly trained reward model can greatly affect the outcome of policy learning. We now consider an alternative method, called direct preference optimization (DPO), which simplifies the training framework by eliminating the need to explicitly model rewards \citep{rafailov:2023direct}. This method directly optimizes the policy model based on user preferences rather than developing a separate reward model. As a result, we can achieve human preference alignment in a supervised learning-like fashion. Figure~\ref{fig:comparison-rl-dpo} shows a comparison of the standard PPO method and the DPO method.
DPO simplifies the RL process for LLMs by utilizing a straightforward cross-entropy loss, which streamlines the learning process and enhances its stability. Rather than delving into the detailed derivation of the DPO loss function, we will discuss its optimization objective from the perspective of optimizing an LLM. The loss function derived from a Bradley-Terry reward model can be given by
DPO simplifies the RL process for LLMs by utilizing a straightforward cross-entropy loss, which streamlines the learning process and enhances its stability. Rather than delving into the detailed derivation of the DPO loss function, we will discuss its optimization objective from the perspective of optimizing an LLM. The loss function derived from a Bradley-Terry reward model can be given by
\begin{eqnarray}
\begin{eqnarray}
...
@@ -606,7 +606,7 @@ This loss function utilizes the implicit reward for DPO training, which replaces
...
@@ -606,7 +606,7 @@ This loss function utilizes the implicit reward for DPO training, which replaces
\label{fig:understand-dpo-optimization-objective}
\label{fig:understand-dpo-optimization-objective}
\end{figure*}
\end{figure*}
Here, we can consider that DPO directly models human feedback through the relative likelihood of outputs under the policy and reference models. This method provides a more straightforward approach than traditional methods that use a separate reward model and then apply RL algorithms like PPO to adjust the probabilities of sampled outputs. In this way, we can further understand the optimization objective of DPO by leveraging the hypothesis space as mentioned in Section\ref{sec:policy-gradient}. In practice, as illustrated in Figure \ref{fig:understand-dpo-optimization-objective}, we can see DPO as reranking in the hypothesis space through the Bradley-Terry model approach. Specifically, this method increases the probabilities of preferred outputs in preference data while decreasing those of dispreferred ones. Similar to TRPO, DPO also incorporates a penalty derived from the reference model, which ensures that updates remain within a trusted behavior space for the policy model.
Here, we can consider that DPO directly models human feedback through the relative likelihood of outputs under the policy and reference models. This method provides a more straightforward approach than traditional methods that use a separate reward model and then apply RL algorithms like PPO to adjust the probabilities of sampled outputs. In this way, we can further understand the optimization objective of DPO by leveraging the hypothesis space as mentioned in Section~\ref{sec:policy-gradient}. In practice, as illustrated in Figure~\ref{fig:understand-dpo-optimization-objective}, we can see DPO as reranking in the hypothesis space through the Bradley-Terry model approach. Specifically, this method increases the probabilities of preferred outputs in preference data while decreasing those of dispreferred ones. Similar to TRPO, DPO also incorporates a penalty derived from the reference model, which ensures that updates remain within a trusted behavior space for the policy model.
However, there are two sides to every coin. While DPO simplifies the RL process, it also introduces limitations. Notably, since DPO eliminates the exploration phase during training, its potential peak performance is generally considered lower compared to RL-based methods like PPO, which incorporate exploration to discover more effective policies. In other words, the performance of DPO is inherently bounded by the labeled preferred outputs in the preference data. To address this limitation, current approaches focus on ensuring high-quality preferred outputs, either by employing advanced LLMs like GPT-4 or through human labeling \citep{cui-etal:2023ultrafeedback,morimura-etal:2024filtered}.
However, there are two sides to every coin. While DPO simplifies the RL process, it also introduces limitations. Notably, since DPO eliminates the exploration phase during training, its potential peak performance is generally considered lower compared to RL-based methods like PPO, which incorporate exploration to discover more effective policies. In other words, the performance of DPO is inherently bounded by the labeled preferred outputs in the preference data. To address this limitation, current approaches focus on ensuring high-quality preferred outputs, either by employing advanced LLMs like GPT-4 or through human labeling \citep{cui-etal:2023ultrafeedback,morimura-etal:2024filtered}.
...
@@ -618,5 +618,5 @@ However, there are two sides to every coin. While DPO simplifies the RL process,
...
@@ -618,5 +618,5 @@ However, there are two sides to every coin. While DPO simplifies the RL process,
\label{tab:dpo_variants}
\label{tab:dpo_variants}
\end{table*}
\end{table*}
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table\ref{tab:dpo_variants}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table~\ref{tab:dpo_variants}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
@@ -24,7 +24,7 @@ Initially, we review the fundamental objective of RL: to maximize the rewards ob
...
@@ -24,7 +24,7 @@ Initially, we review the fundamental objective of RL: to maximize the rewards ob
One approach to test-time scaling using a reward model involves sampling multiple reasoning paths given input and then using the reward model to select the best one from $N$ alternative outputs generated by the LLM, called best-of-$N$ sampling (BoN sampling). We can consider BoN sampling a reranking technique. In fact, reranking methods have been a prevalent technique in NLP, particularly in machine translation, where they have been employed for a long time to enhance output quality by selecting the most appropriate translation from a set of candidates. Additionally, this method often functions as a simple model ensemble approach. In such cases, different outputs generated by various models can be reranked according to a specified metric.
One approach to test-time scaling using a reward model involves sampling multiple reasoning paths given input and then using the reward model to select the best one from $N$ alternative outputs generated by the LLM, called best-of-$N$ sampling (BoN sampling). We can consider BoN sampling a reranking technique. In fact, reranking methods have been a prevalent technique in NLP, particularly in machine translation, where they have been employed for a long time to enhance output quality by selecting the most appropriate translation from a set of candidates. Additionally, this method often functions as a simple model ensemble approach. In such cases, different outputs generated by various models can be reranked according to a specified metric.
As illustrated in Figure\ref{fig:bon-sampling}, in the BoN sampling, we first sample $N$ different outputs $\{\mathbf{y}_{1}, \mathbf{y}_{2}, \cdots, \mathbf{y}_{N}\}$ for the input $\mathbf{x}$:
As illustrated in Figure~\ref{fig:bon-sampling}, in the BoN sampling, we first sample $N$ different outputs $\{\mathbf{y}_{1}, \mathbf{y}_{2}, \cdots, \mathbf{y}_{N}\}$ for the input $\mathbf{x}$:
@@ -47,12 +47,12 @@ While BoN sampling is effective for scaling test-time computation, it is ineffic
...
@@ -47,12 +47,12 @@ While BoN sampling is effective for scaling test-time computation, it is ineffic
We can collect or generate reasoning paths corresponding to problems from existing datasets to train a process reward model. Human experts then annotate each step in these paths for correctness. These annotations can be used to train LLMs or as rewards in reward modeling directly. However, in practice, richer annotations are often introduced \citep{lightman-etal:2024lets}. In addition to the \textit{correct} and \textit{incorrect} labels, a step can also be labeled as \textit{neutral} to indicate that while the step may be technically correct, it might still be problematic within the overall reasoning process. Additionally, an automatic process annotation framework can be utilized, which relies on generating multiple entire reasoning paths based on the current step and using the accuracy of these paths to serve as the quality annotation of the step \citep{wang-etal:2023math}.
We can collect or generate reasoning paths corresponding to problems from existing datasets to train a process reward model. Human experts then annotate each step in these paths for correctness. These annotations can be used to train LLMs or as rewards in reward modeling directly. However, in practice, richer annotations are often introduced \citep{lightman-etal:2024lets}. In addition to the \textit{correct} and \textit{incorrect} labels, a step can also be labeled as \textit{neutral} to indicate that while the step may be technically correct, it might still be problematic within the overall reasoning process. Additionally, an automatic process annotation framework can be utilized, which relies on generating multiple entire reasoning paths based on the current step and using the accuracy of these paths to serve as the quality annotation of the step \citep{wang-etal:2023math}.
Given a set of step-level annotated reasoning paths and corresponding inputs, we can train a reward model to provide a reward for each step in the reasoning process. The reward model can be treated as a classification model. Thus, its architecture can be an LLM with a Softmax layer stacked on top, akin to the architecture depicted in Figure\ref{fig:reward-model}. Here, consider a reasoning path including $n_{s}$ steps, represented as $\mathbf{y}=\{\bar{\mathbf{y}}_{1},\cdots ,\bar{\mathbf{y}}_{n_{s}}\}$. At each step $k$, the process reward model takes both the problem description, denoted by $\mathbf{x}$, and the reasoning steps generated so far, denoted by $\bar{\mathbf{y}}$, as inputs. It then outputs a probability distribution over the set of labels ${\text{\textit{correct}}, \text{\textit{incorrect}}}$, or ${\text{\textit{correct}}, \text{\textit{incorrect}}, \text{\textit{neutral}}}$, to evaluate the reasoning at that point. This model can be trained with a standard classification loss, e.g., cross-entropy. Once trained, the reward model can be used to evaluate reasoning paths by assessing the correctness of each step. A simple method is to use classification log-probabilities to define the reward of each reasoning step. For example, the reward of the $k$-th reasoning step can be given by
Given a set of step-level annotated reasoning paths and corresponding inputs, we can train a reward model to provide a reward for each step in the reasoning process. The reward model can be treated as a classification model. Thus, its architecture can be an LLM with a Softmax layer stacked on top, akin to the architecture depicted in Figure~\ref{fig:reward-model}. Here, consider a reasoning path including $n_{s}$ steps, represented as $\mathbf{y}=\{\bar{\mathbf{y}}_{1},\cdots ,\bar{\mathbf{y}}_{n_{s}}\}$. At each step $k$, the process reward model takes both the problem description, denoted by $\mathbf{x}$, and the reasoning steps generated so far, denoted by $\bar{\mathbf{y}}$, as inputs. It then outputs a probability distribution over the set of labels ${\text{\textit{correct}}, \text{\textit{incorrect}}}$, or ${\text{\textit{correct}}, \text{\textit{incorrect}}, \text{\textit{neutral}}}$, to evaluate the reasoning at that point. This model can be trained with a standard classification loss, e.g., cross-entropy. Once trained, the reward model can be used to evaluate reasoning paths by assessing the correctness of each step. A simple method is to use classification log-probabilities to define the reward of each reasoning step. For example, the reward of the $k$-th reasoning step can be given by
where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes the probability of the \textit{correct} label generated by the reward model. The reward score $R_{\phi}(\mathbf{x},\mathbf{y})$ can then be used to select the best step while generating a reasoning path. Additionally, as discussed in Section\ref{sec:generative-reward-models}, there is the option to train a generative process reward model, also called a generative verifier in the literature, to further improve reasoning-step evaluation \citep{zhang-etal:2024generative}. Note that in practice, the process reward model serves not only to provide rewards for test-time scaling but also to train the model using RL, e.g., the rewards from this model can be employed as shaping rewards.
where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes the probability of the \textit{correct} label generated by the reward model. The reward score $R_{\phi}(\mathbf{x},\mathbf{y})$ can then be used to select the best step while generating a reasoning path. Additionally, as discussed in Section~\ref{sec:generative-reward-models}, there is the option to train a generative process reward model, also called a generative verifier in the literature, to further improve reasoning-step evaluation \citep{zhang-etal:2024generative}. Note that in practice, the process reward model serves not only to provide rewards for test-time scaling but also to train the model using RL, e.g., the rewards from this model can be employed as shaping rewards.
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
...
@@ -63,11 +63,11 @@ where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes
...
@@ -63,11 +63,11 @@ where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes
\label{fig:step-by-step-with-prm}
\label{fig:step-by-step-with-prm}
\end{figure*}
\end{figure*}
As illustrated in Figure\ref{fig:step-by-step-with-prm}, step-by-step verification can be conducted using a simple greedy search method. Specifically, multiple candidate reasoning steps are sampled at each step, and the process reward model is employed to select the best one. This selected step then serves as the foundation for generating the subsequent step, continuing this process until the entire reasoning path is obtained. In this way, any search method can be applied to improve test-time scaling. For example, we can expand the search space with beam search, retaining multiple promising reasoning steps at each step. Furthermore, the reasoning steps can be refined through the self-refinement technique at each step using verification feedback, such as rewards, to enhance accuracy \citep{yao-etal:2023tree}.
As illustrated in Figure~\ref{fig:step-by-step-with-prm}, step-by-step verification can be conducted using a simple greedy search method. Specifically, multiple candidate reasoning steps are sampled at each step, and the process reward model is employed to select the best one. This selected step then serves as the foundation for generating the subsequent step, continuing this process until the entire reasoning path is obtained. In this way, any search method can be applied to improve test-time scaling. For example, we can expand the search space with beam search, retaining multiple promising reasoning steps at each step. Furthermore, the reasoning steps can be refined through the self-refinement technique at each step using verification feedback, such as rewards, to enhance accuracy \citep{yao-etal:2023tree}.
\subsubsection{Monte Carlo Tree Search}
\subsubsection{Monte Carlo Tree Search}
In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popular search method in step-by-step verification. In practice, MCTS is not a new method but is well-established across various domains. Notably, it typically serves as an effective alternative to traditional RL methods, particularly in environments characterized by large or intricate state spaces, where conventional RL algorithms may falter. For example, within an RL framework, MCTS can aid in planning by simulating diverse actions to maximize rewards, as demonstrated by its success in complex game environments \citep{silver-etal:2016mastering}. In LLM reasoning, MCTS leverages randomness and structured tree search to probe potential reasoning paths, thereby expanding the search space in an efficient way. More specifically, as illustrated in Figure\ref{fig:monte-carlo-tree-search}, MCTS repeatedly cycles through the following four stages to explore potential reasoning paths:
In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popular search method in step-by-step verification. In practice, MCTS is not a new method but is well-established across various domains. Notably, it typically serves as an effective alternative to traditional RL methods, particularly in environments characterized by large or intricate state spaces, where conventional RL algorithms may falter. For example, within an RL framework, MCTS can aid in planning by simulating diverse actions to maximize rewards, as demonstrated by its success in complex game environments \citep{silver-etal:2016mastering}. In LLM reasoning, MCTS leverages randomness and structured tree search to probe potential reasoning paths, thereby expanding the search space in an efficient way. More specifically, as illustrated in Figure~\ref{fig:monte-carlo-tree-search}, MCTS repeatedly cycles through the following four stages to explore potential reasoning paths:
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
...
@@ -92,12 +92,12 @@ In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popu
...
@@ -92,12 +92,12 @@ In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popu
\item\textbf{Simulation}. The process performs simulations (or rollouts) for each new node added during the expansion stage. These simulated paths help evaluate the effectiveness of the reasoning steps initiated from the node. For example, the final answer obtained from the simulation can be compared with the correct answer to derive a delayed reward.
\item\textbf{Simulation}. The process performs simulations (or rollouts) for each new node added during the expansion stage. These simulated paths help evaluate the effectiveness of the reasoning steps initiated from the node. For example, the final answer obtained from the simulation can be compared with the correct answer to derive a delayed reward.
\item\textbf{Backpropagation}. The process updates the UCT values of prior nodes using the results of these simulations. Key updates include the visitation counts, $N_p$ and $N_{\bar{\mathbf{y}}_{k}}$ in Eq.(\ref{eq:uct}), reflecting how frequently each node has been explored. Additionally, this stage allows for the incorporation of delayed rewards to adjust the $\bar{R}(\bar{\mathbf{y}}_k)$ values.
\item\textbf{Backpropagation}. The process updates the UCT values of prior nodes using the results of these simulations. Key updates include the visitation counts, $N_p$ and $N_{\bar{\mathbf{y}}_{k}}$ in Eq.~(\ref{eq:uct}), reflecting how frequently each node has been explored. Additionally, this stage allows for the incorporation of delayed rewards to adjust the $\bar{R}(\bar{\mathbf{y}}_k)$ values.
\end{itemize}
\end{itemize}
\subsection{Iterative RL}
\subsection{Iterative RL}
Training LLMs with RL usually follows a two-phase approach: training a pre-trained LLM with SFT and further training with an RL algorithm applied to the SFT LLM. However, using large-scale RL in such a two-phase approach to train LLMs in reasoning may lead to significant knowledge forgetting as the model continuously adjusts to fit the reward model. For example, as mentioned in DeepSeek-R1 \citep{guo:2025deepseek}, directly applying large-scale RL can achieve the desired reasoning outcomes, but often at the cost of reduced readability in the reasoning process. To address these challenges, we can utilize an iterative RL approach to continuously enhance the various capabilities of the LLM. Here we consider DeepSeek-R1 as an example to illustrate how to perform an iterative RL. The idea is to split the RL process into multiple phases, each designed to enhance different capabilities using varied rewards, to develop a robust reasoning model that is able to generate clearer and more comprehensible reasoning paths. Figure\ref{fig:iterative-rl} provides a schematic illustration of the iterative RL process. Here we give a brief outline of each phase involved.
Training LLMs with RL usually follows a two-phase approach: training a pre-trained LLM with SFT and further training with an RL algorithm applied to the SFT LLM. However, using large-scale RL in such a two-phase approach to train LLMs in reasoning may lead to significant knowledge forgetting as the model continuously adjusts to fit the reward model. For example, as mentioned in DeepSeek-R1 \citep{guo:2025deepseek}, directly applying large-scale RL can achieve the desired reasoning outcomes, but often at the cost of reduced readability in the reasoning process. To address these challenges, we can utilize an iterative RL approach to continuously enhance the various capabilities of the LLM. Here we consider DeepSeek-R1 as an example to illustrate how to perform an iterative RL. The idea is to split the RL process into multiple phases, each designed to enhance different capabilities using varied rewards, to develop a robust reasoning model that is able to generate clearer and more comprehensible reasoning paths. Figure~\ref{fig:iterative-rl} provides a schematic illustration of the iterative RL process. Here we give a brief outline of each phase involved.
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
...
@@ -111,12 +111,12 @@ Training LLMs with RL usually follows a two-phase approach: training a pre-train
...
@@ -111,12 +111,12 @@ Training LLMs with RL usually follows a two-phase approach: training a pre-train
\begin{itemize}
\begin{itemize}
\item The iterative RL aims to train an LLM that generates a human-like reasoning process. Initially, it involves collecting high-quality data from various reasoning tasks, such as mathematical problem-solving and code generation. This data is then used to fine-tune a pre-trained LLM as a cold start. This phase is designed to equip the LLM with basic reasoning capabilities, preventing linguistically confused or nonsensical reasoning paths from being sampled during the following RL phase.
\item The iterative RL aims to train an LLM that generates a human-like reasoning process. Initially, it involves collecting high-quality data from various reasoning tasks, such as mathematical problem-solving and code generation. This data is then used to fine-tune a pre-trained LLM as a cold start. This phase is designed to equip the LLM with basic reasoning capabilities, preventing linguistically confused or nonsensical reasoning paths from being sampled during the following RL phase.
\item After the cold start phase, large-scale RL is employed on the reasoning task. During this phase, rule-based rewards (i.e., format checking and answer verification) are utilized to optimize the reasoning process continuously. By doing so, the model learns longer and more reasonable reasoning paths, progressively enhancing its capacity to tackle complex reasoning tasks.
\item After the cold start phase, large-scale RL is employed on the reasoning task. During this phase, rule-based rewards (i.e., format checking and answer verification) are utilized to optimize the reasoning process continuously. By doing so, the model learns longer and more reasonable reasoning paths, progressively enhancing its capacity to tackle complex reasoning tasks.
\item While the model has learned how to generate reasoning paths through large-scale RL, it often produces outputs with poor readability. This can mainly be attributed to the large-scale RL phase, which does not focus on optimizing the readability of the reasoning paths but instead solely on format correctness and answer accuracy. To address this issue, the third phase involves using rejection sampling to further enhance the readability of outputs. More specifically, multiple outputs are sampled from the model trained in the previous phase across various tasks such as mathematical reasoning, code generation, and question answering. A reward model then selects the best outputs, as discussed in Section\ref{sec:BoN-sampling}. These selected outputs are used to train the model using SFT, which aims to refine its performance by aligning it more closely with human-like reasoning.
\item While the model has learned how to generate reasoning paths through large-scale RL, it often produces outputs with poor readability. This can mainly be attributed to the large-scale RL phase, which does not focus on optimizing the readability of the reasoning paths but instead solely on format correctness and answer accuracy. To address this issue, the third phase involves using rejection sampling to further enhance the readability of outputs. More specifically, multiple outputs are sampled from the model trained in the previous phase across various tasks such as mathematical reasoning, code generation, and question answering. A reward model then selects the best outputs, as discussed in Section~\ref{sec:BoN-sampling}. These selected outputs are used to train the model using SFT, which aims to refine its performance by aligning it more closely with human-like reasoning.
\item In the final phase, multi-task RL is implemented to ensure broader generalization and maintain high performance across tasks beyond reasoning. The process involves training the model using RL on various tasks against a general reward model.
\item In the final phase, multi-task RL is implemented to ensure broader generalization and maintain high performance across tasks beyond reasoning. The process involves training the model using RL on various tasks against a general reward model.
\end{itemize}
\end{itemize}
An interesting issue arises with this design of iterative RL: why is RL aimed at enhancing reasoning capabilities placed at the initial phase rather than at the end? This is partly because the rewards for the reasoning task, which are inherently more stable as described in Section\ref{sec:lightweight-reward-methods}, are well-suited for large-scale RL compared to traditional reward models. Furthermore, the reasoning capabilities could be applicable across many tasks, allowing for large-scale learning to boost performance in subsequent multi-task learning. We consider that this foundational enhancement in reasoning capabilities may contribute to the observed robust performance across various tasks, even without the use of labeled data in later phases. In this case, large-scale RL could be viewed as an approach to continued pre-training.
An interesting issue arises with this design of iterative RL: why is RL aimed at enhancing reasoning capabilities placed at the initial phase rather than at the end? This is partly because the rewards for the reasoning task, which are inherently more stable as described in Section~\ref{sec:lightweight-reward-methods}, are well-suited for large-scale RL compared to traditional reward models. Furthermore, the reasoning capabilities could be applicable across many tasks, allowing for large-scale learning to boost performance in subsequent multi-task learning. We consider that this foundational enhancement in reasoning capabilities may contribute to the observed robust performance across various tasks, even without the use of labeled data in later phases. In this case, large-scale RL could be viewed as an approach to continued pre-training.
Since the optimization objective of each phase is different, the design of iterative RL can also be analyzed and understood from the perspective of multi-objective optimization \citep{wang-etal:2024hybrid}. In the context of multi-objective optimization, iterative RL for LLMs can be likened to interactive methods where the solution process is iterative, and preferences are actively defined and refined by the decision-maker during the search for the most preferred solutions \citep{miettinen-etal:2008introduction,deb-etal:2016multi}. More specifically, in this scenario, RL can be seen as a decision-maker, iteratively refining and enhancing the different capabilities of the LLM across different phases.
Since the optimization objective of each phase is different, the design of iterative RL can also be analyzed and understood from the perspective of multi-objective optimization \citep{wang-etal:2024hybrid}. In the context of multi-objective optimization, iterative RL for LLMs can be likened to interactive methods where the solution process is iterative, and preferences are actively defined and refined by the decision-maker during the search for the most preferred solutions \citep{miettinen-etal:2008introduction,deb-etal:2016multi}. More specifically, in this scenario, RL can be seen as a decision-maker, iteratively refining and enhancing the different capabilities of the LLM across different phases.
...
@@ -125,9 +125,9 @@ Since the optimization objective of each phase is different, the design of itera
...
@@ -125,9 +125,9 @@ Since the optimization objective of each phase is different, the design of itera
While test-time scaling is instrumental in exploring more effective reasoning paths, its potential is restricted if the model has not learned to generate high-quality reasoning paths. In other words, in practice, test-time scaling struggles to achieve desired outcomes without foundational reasoning ability, no matter how extensive it is \citep{yuan-etal:2023scaling,zeng-etal:2025revisiting}. Consequently, there has been a growing research interest in training LLMs specifically to develop reasoning abilities that mimic human-like processes. A straightforward approach is to use annotated reasoning data to train the LLM through SFT. However, a significant challenge in this approach is the high cost of annotations, especially since annotating detailed step-by-step reasoning paths is significantly more cost-intensive than annotating SFT data in other tasks like machine translation and summarization.
While test-time scaling is instrumental in exploring more effective reasoning paths, its potential is restricted if the model has not learned to generate high-quality reasoning paths. In other words, in practice, test-time scaling struggles to achieve desired outcomes without foundational reasoning ability, no matter how extensive it is \citep{yuan-etal:2023scaling,zeng-etal:2025revisiting}. Consequently, there has been a growing research interest in training LLMs specifically to develop reasoning abilities that mimic human-like processes. A straightforward approach is to use annotated reasoning data to train the LLM through SFT. However, a significant challenge in this approach is the high cost of annotations, especially since annotating detailed step-by-step reasoning paths is significantly more cost-intensive than annotating SFT data in other tasks like machine translation and summarization.
Another more advanced approach is to utilize large-scale RL, transitioning from human-annotated to self-reinforced learning processes. Unlike the RL used as a fine-tuning method discussed in Section\ref{sec:example-using-rl-training-llms}, which typically involves training for a few dozen or hundreds of steps at a small scale, this approach applies RL at a larger scale with rewards to break free from the limitation of supervised reasoning data. This approach has enabled the development of robust reasoning models, such as OpenAI-o1 \citep{openai:2024learning}, DeepSeek-R1 \citep{guo:2025deepseek}, and Kimi-1.5 \citep{kimi-team:2025kimi}. Notably, DeepSeek-R1-Zero was able to develop a robust reasoning model by applying large-scale RL to a pre-trained LLM without the need for any annotated reasoning data. However, despite its successes, large-scale RL is not easy and introduces unique challenges that are not present at smaller scales, as follows.
Another more advanced approach is to utilize large-scale RL, transitioning from human-annotated to self-reinforced learning processes. Unlike the RL used as a fine-tuning method discussed in Section~\ref{sec:example-using-rl-training-llms}, which typically involves training for a few dozen or hundreds of steps at a small scale, this approach applies RL at a larger scale with rewards to break free from the limitation of supervised reasoning data. This approach has enabled the development of robust reasoning models, such as OpenAI-o1 \citep{openai:2024learning}, DeepSeek-R1 \citep{guo:2025deepseek}, and Kimi-1.5 \citep{kimi-team:2025kimi}. Notably, DeepSeek-R1-Zero was able to develop a robust reasoning model by applying large-scale RL to a pre-trained LLM without the need for any annotated reasoning data. However, despite its successes, large-scale RL is not easy and introduces unique challenges that are not present at smaller scales, as follows.
One is that large-scale RL requires a highly generalizable reward model. As the scale of training increases, the model is trained on broader data, necessitating a reward model capable of effectively generalizing across this varied data. On the other hand, the extensive scope of training introduces significant variability in the model, leading to considerable changes in sampling behaviors. Consequently, it is crucial to ensure that the reward model possesses robust generalization capabilities to prevent overfitting and maintain the effectiveness of the learning process. There are several methods to achieve this. For example, \citet{guo:2025deepseek} incorporated rule-based rewards, as discussed in Section\ref{sec:lightweight-reward-methods}, such as format checking and answer verification in reasoning scenarios, which can provide a stable reward throughout the learning process. This suggests that in certain RL scenarios, prioritizing rule-based rewards may be beneficial if they can effectively describe human preferences. \citet{yuan-etal:2024selfrewarding} introduced a self-rewarding framework that dynamically updates the reward model based on the currently optimized policy model so that this reward model can effectively evaluate the behaviors from the current policy model.
One is that large-scale RL requires a highly generalizable reward model. As the scale of training increases, the model is trained on broader data, necessitating a reward model capable of effectively generalizing across this varied data. On the other hand, the extensive scope of training introduces significant variability in the model, leading to considerable changes in sampling behaviors. Consequently, it is crucial to ensure that the reward model possesses robust generalization capabilities to prevent overfitting and maintain the effectiveness of the learning process. There are several methods to achieve this. For example, \citet{guo:2025deepseek} incorporated rule-based rewards, as discussed in Section~\ref{sec:lightweight-reward-methods}, such as format checking and answer verification in reasoning scenarios, which can provide a stable reward throughout the learning process. This suggests that in certain RL scenarios, prioritizing rule-based rewards may be beneficial if they can effectively describe human preferences. \citet{yuan-etal:2024selfrewarding} introduced a self-rewarding framework that dynamically updates the reward model based on the currently optimized policy model so that this reward model can effectively evaluate the behaviors from the current policy model.
Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behavior region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between stability and adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behavior region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between stability and adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
@@ -60,13 +60,13 @@ Unlike single-turn response generation, agent planning requires a sequence of in
...
@@ -60,13 +60,13 @@ Unlike single-turn response generation, agent planning requires a sequence of in
\label{fig:agent-planning-sft}
\label{fig:agent-planning-sft}
\end{figure*}
\end{figure*}
Before applying RL, SFT is commonly adopted to provide a cold start for LLM-based agents by teaching them basic planning and interaction behaviors \citep{chen-etal:fireact,zhang-etal:agentohana,zeng-etal:agenttuning}. As illustrated in Figure\ref{fig:agent-planning-sft}, constructing SFT data for agent planning typically involves three stages: (1) preparing diverse environments and planning tasks; (2) synthesizing expert-level trajectories, consisting of action-observation sequences, through agent-environment interactions; and (3) selecting high-quality trajectories based on predefined evaluation criteria. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as expert agents and retaining reliable demonstrations according to reward signals or task success criteria. Through this process, LLMs can learn planning behaviors from demonstrations and improve their ability to generate structured execution plans.
Before applying RL, SFT is commonly adopted to provide a cold start for LLM-based agents by teaching them basic planning and interaction behaviors \citep{chen-etal:fireact,zhang-etal:agentohana,zeng-etal:agenttuning}. As illustrated in Figure~\ref{fig:agent-planning-sft}, constructing SFT data for agent planning typically involves three stages: (1) preparing diverse environments and planning tasks; (2) synthesizing expert-level trajectories, consisting of action-observation sequences, through agent-environment interactions; and (3) selecting high-quality trajectories based on predefined evaluation criteria. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as expert agents and retaining reliable demonstrations according to reward signals or task success criteria. Through this process, LLMs can learn planning behaviors from demonstrations and improve their ability to generate structured execution plans.
\begin{eqnarray}
\begin{eqnarray}
\mathbf{x}&=& [e,q] \\
\mathbf{x}&=& [e,q] \\
\mathbf{y}&=& [p,u_1,o_1,\cdots,u_T,o_T]
\mathbf{y}&=& [p,u_1,o_1,\cdots,u_T,o_T]
\end{eqnarray}
\end{eqnarray}
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section\ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section~\ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
Similar to LLM instruction tuning \citep{longpre-etal:flan,zhou-etal:lima}, the scale and quality of trajectory data significantly influence the effectiveness of agent training. High-quality trajectories provide explicit supervision signals for agents to learn how to decompose complex tasks, select appropriate tools, and interact with dynamic environments. Therefore, recent studies have explored constructing trajectory-based instruction tuning datasets to enhance the planning capabilities of LLM-based agents.
Similar to LLM instruction tuning \citep{longpre-etal:flan,zhou-etal:lima}, the scale and quality of trajectory data significantly influence the effectiveness of agent training. High-quality trajectories provide explicit supervision signals for agents to learn how to decompose complex tasks, select appropriate tools, and interact with dynamic environments. Therefore, recent studies have explored constructing trajectory-based instruction tuning datasets to enhance the planning capabilities of LLM-based agents.
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
The mechanism of credit assignment is illustrated in Figure\ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
The mechanism of credit assignment is illustrated in Figure~\ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. These step-level rewards are diagnostic signals rather than a conservative numerical decomposition, so they need not sum to the trajectory-level reward. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. These step-level rewards are diagnostic signals rather than a conservative numerical decomposition, so they need not sum to the trajectory-level reward. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
@@ -23,7 +23,7 @@ A representative example is the multimodal language model\footnote{A multimodal
...
@@ -23,7 +23,7 @@ A representative example is the multimodal language model\footnote{A multimodal
Although the overall RL formulation can be reused, training VLMs with RL is still challenging in practice. The central difficulty is not the policy optimization algorithm itself, but the scarcity of high-quality multimodal preference data. Compared with textual preference data, visual preference data is more expensive to collect, harder to verify, and often more task-dependent. This makes it difficult to train a reliable visual reward model, which is a key component for applying RLHF-style methods to VLMs.
Although the overall RL formulation can be reused, training VLMs with RL is still challenging in practice. The central difficulty is not the policy optimization algorithm itself, but the scarcity of high-quality multimodal preference data. Compared with textual preference data, visual preference data is more expensive to collect, harder to verify, and often more task-dependent. This makes it difficult to train a reliable visual reward model, which is a key component for applying RLHF-style methods to VLMs.
One straightforward way to alleviate this problem is to generate visual preference data through the automatic preference data generation method described in Section\ref{sec:automatic-preference-data-generation}\citep{yu-etal:2024rlaif}. Another alternative is a multi-stage training approach for the visual reward model, motivated by a simple idea: human preferences are already well captured in text, and these preferences can be transferred across modalities \citep{wang-etal:2024rovrm}. By leveraging textual preference data, this approach reduces the dependence on large-scale visual preference data when training a visual reward model. More specifically, as illustrated in Figure \ref{fig:preference-transfer}, we can train a visual reward model in the following three stages:
One straightforward way to alleviate this problem is to generate visual preference data through the automatic preference data generation method described in Section~\ref{sec:automatic-preference-data-generation}\citep{yu-etal:2024rlaif}. Another alternative is a multi-stage training approach for the visual reward model, motivated by a simple idea: human preferences are already well captured in text, and these preferences can be transferred across modalities \citep{wang-etal:2024rovrm}. By leveraging textual preference data, this approach reduces the dependence on large-scale visual preference data when training a visual reward model. More specifically, as illustrated in Figure~\ref{fig:preference-transfer}, we can train a visual reward model in the following three stages:
\begin{itemize}
\begin{itemize}
\item Stage 1: pre-training with large-scale textual preference data. Given the transferability of human preferences across different modalities, we can begin by using large-scale textual preference data to pre-train the visual reward model. Note that the projector parameters are frozen without images. This stage can be considered as providing a stronger starting point for training the visual reward model, as it enables the model to pre-learn general human preferences.
\item Stage 1: pre-training with large-scale textual preference data. Given the transferability of human preferences across different modalities, we can begin by using large-scale textual preference data to pre-train the visual reward model. Note that the projector parameters are frozen without images. This stage can be considered as providing a stronger starting point for training the visual reward model, as it enables the model to pre-learn general human preferences.
\item Stage 2: fine-tuning with image caption-based preference data. Pre-learned human preferences cannot be directly applied to vision tasks due to both \textit{task gap} and \textit{modality gap}. To bridge the task gap, we fine-tune the reward model using image caption-based preference data in this stage. The rationale behind this approach is that general textual preference data does not cover vision-specific tasks, such as ``Please describe the content in this image''. We use such data to fine-tune the model, adapting it to vision-specific tasks. The projector parameters are frozen at this stage.
\item Stage 2: fine-tuning with image caption-based preference data. Pre-learned human preferences cannot be directly applied to vision tasks due to both \textit{task gap} and \textit{modality gap}. To bridge the task gap, we fine-tune the reward model using image caption-based preference data in this stage. The rationale behind this approach is that general textual preference data does not cover vision-specific tasks, such as ``Please describe the content in this image''. We use such data to fine-tune the model, adapting it to vision-specific tasks. The projector parameters are frozen at this stage.
