\section{Improved Reinforcement Learning for LLMs}
In the previous section, we introduced some improvements to RL, such as importance sampling and reward baseline techniques. However, directly applying them to train LLMs still presents numerous challenges. In this section, we will delve deeper into the improvements for using RL to train LLMs.

\subsection{Advanced Reward Models}
Reward models are a fundamental component in RL, as they define what a policy model optimizes for. Therefore, the quality of the reward model training directly influences the effectiveness of the LLM optimization during RLHF. A poorly trained reward model can lead to suboptimal LLM, while a well-optimized reward model ensures that the LLM is effectively aligned with the desired behaviours and objectives. Here we will introduce several methods for obtaining advanced reward models. Our discussion will be relatively general, and since the reward model is widely used in many RL problems, such as robot planning and control. This broad applicability makes it straightforward to adapt the methods discussed here to other related applications.


\subsubsection{Automatic Preference Data Generation}
\label{sec:automatic-preference-data-generation}
Although learning from human preferences is an effective and popular method for aligning LLMs, annotating preference data is costly. Relying on human feedback not only faces scalability limitations but may also introduce bias, as human feedback is inherently subjective. Consequently, AI-based feedback methods offer a promising solution to address these scalability and consistency issues, avoiding the limitations associated with human annotators.

One simple method is to generate preference data using LLM. Given a set of inputs, we first use an LLM to generate pairs of outputs. Then, we prompt the LLM to label the preference between each pair of outputs, along with its corresponding input. Below is an example of prompting the LLM to generate a preference label for a pair of outputs.

\vspace{0.5em}
\begin{tcolorbox}[frame empty]

\begingroup
\setlength{\leftskip}{2em}
\setlength{\rightskip}{2em}

Consider a homework assistant scenario in which a student types an input. You will review two outputs to this input. Please indicate which output you prefer. A good output should be informative, accurate, and well-articulated. It should directly address the student's question, provide thorough explanations or solutions, and maintain an encouraging tone.

\vspace{0.3cm}

Input:

\vspace{0.1cm}

\textit{Give me three tips to improve my accuracy in solving math problems.}

\vspace{0.3cm}

Output A:

\vspace{0.1cm}

\textit{Improving math accuracy requires careful attention to detail, strategic problem-solving techniques, and consistent practice. Here are three tips: \\
1.Practice regularly: \\
- Consistent practice builds fluency and familiarity with different problem types... \\
... \\
By combining your advice with these additional tips, anyone can significantly enhance their math problem-solving abilities.}

\vspace{0.3cm}

Output B:

\vspace{0.1cm}

\textit{There are three tips for improving accuracy in solving math problems: \\
1.Practice regularly. \\
2.Understand the concepts. \\
3.Double-check your work.}

\vspace{0.3cm}

\underline{Output A is preferred.}

\endgroup

\end{tcolorbox}
\vspace{0.5em}

Once we collect such preference labels, we can use them, along with the output pair and input, to train the reward model. Of course, these labels are not entirely accurate either. Recent studies have shown that AI-based feedback often exhibits a location bias problem, making it more likely to prefer the output at the front \citep{zheng-etal:2023judging}.  We can consider demonstrating a few examples or using advanced prompting techniques, such as Chain-of-Though (CoT), to improve the labeling performance \citep{liu-etal:2023GEval}.


For data generation, although it is easy to scale up, it is often necessary to ensure the data is accurate and diverse. Here, the data quality and diversity issues involve not only the labeling of preferences but also the inputs and outputs of the model. Therefore, we often need to use a variety of techniques to obtain large-scale, high-quality data. For example, one can generate diverse model outputs and annotations by using different LLMs, prompts, in-context demonstrations, and so on \citep{cui-etal:2024ultra}. Furthermore, \citet{dubois-etal:2024alpacafarm} report that the variability in pairwise preference data is important for training LLMs from either human or AI feedback.

While learning from AI feedback is highly scalable and generally objective, this method is more suited to well-defined tasks where objective performance metrics are available. By contrast, learning from human feedback is more advantageous when aligning AI systems with human values, preferences, and complex real-world tasks that require an understanding of subtle or subjective context. These methods can be combined to train LLMs that benefit from both human insights and the scalability of AI feedback.

\begin{figure*}[!t]
    \centering
    \input{section4/Figures/figure-an-overlap-example-dense-reward-model}
    \caption[Issue of high variance in Monte Carlo-based advantage estimation]{
    An example of reward shaping. Here, length-based shaping rewards are implemented, e.g., awarding a +1 intermediate reward for every 10 tokens generated. Additionally, a delayed reward from the reward model evaluates the overall quality of the output at the end.
    }
    \label{fig:example-shaping-rewards}
\end{figure*}

\subsubsection{Reward Shaping}
\label{sec:reward-shaping}

As discussed in Section \ref{sec:training-reward-models}, while the reward model is effective in capturing human preferences, it provides sparse rewards. These rewards, known as delayed rewards, are only at the end of the output generation process, as opposed to intermediate rewards that would be distributed continuously throughout it. 

In fact, dealing with sparse rewards has long been a concern in RL, and has been one of the challenges in many practical applications. For example, robotics often needs to shape the reward function to ease optimization rather than relying solely on end-of-sequence rewards. Various methods have been developed to address this issue. One common approach is reward shaping, where the original function is modified to include intermediate rewards, thereby providing more immediate feedback. Here, the intermediate reward is often an indirect way to be able to improve this delayed reward. For example, as shown in Figure \ref{fig:example-shaping-rewards}, setting a length-based reward as an intermediate reward encourages the generation of more content, potentially increasing the overall quality of the final output as assessed by the delayed reward from the reward model. Additional examples of reward shaping in training LLMs can be found in \citet{kumar-etal:2024training}. In this way, we can obtain
\begin{eqnarray}
    r'_{t} & = & r_{t} + f(\mathbf{x}, \mathbf{y}_{<t}, y_{t})
    \label{eq:transformed-reward-function}
\end{eqnarray}

where $r'(\cdot)$ is the transformed reward, $r(\cdot)$ is the original delayed reward function, and $f(\cdot)$ is the shaping reward function that provides an intermediate reward. To ensure the optimality of the policy under the transformed reward function, the shaping reward function can be given in the form
\begin{eqnarray}
f(\mathbf{x}, \mathbf{y}_{<t}, y_{t}) & = & \gamma \Phi(\mathbf{x}, \mathbf{y}_{<t+1}, y_{t+1}) - \Phi(\mathbf{x}, \mathbf{y}_{<t}, y_{t}) \label{eq:shaping-reward-function}
\end{eqnarray}

where $\Phi(\cdot)$ is called the potential value function. If we define $\Phi(\cdot)$ as the common value function and substitute Eq. (\ref{eq:shaping-reward-function}) into Eq. (\ref{eq:transformed-reward-function}), we obtain
\begin{eqnarray}
r'_{t} & = & r_{t} + \gamma V_{t+1} - V_{t}
\end{eqnarray}

It is interesting to see that this function is exactly the same as the advantage function used in PPO. This relates advantage-based methods to reward shaping: the advantage is essentially a shaped reward.

In practice, an alternative method for implementing reward shaping in training LLMs involves utilizing the reward model to provide intermediate rewards based on the differences between output rewards \citep{bahdanau-etal:2016actor,wu-etal:2023fine}. For example, at time step $t$, the intermediate reward $r_t$ can be obtained by
\begin{eqnarray}
    r_{t} & = & R(\mathbf{x}, \mathbf{y}_{<t+1}) - R(\mathbf{x}, \mathbf{y}_{<t})
\end{eqnarray}

However, in this case, it is crucial that the reward model is capable of accurately assigning rewards for partial outputs, i.e., $\{\mathbf{y}_{<2},\mathbf{y}_{<3},\cdots,\mathbf{y}_{<T}\}$.

In addition to reward shaping, another method to address the sparse reward issue is adopting curriculum learning, where tasks are structured sequentially with gradually increasing complexity. This can help models master simpler tasks first, which prepares them for more complex challenges as their skills develop. There are many methods that can mitigate the impact of sparse rewards, such as Monte Carlo methods and intrinsic motivation. Most of these methods are general, and the discussion of them can be found in the broader literature on RL, such as \citet{Sutton-and-Barto:2018RL}'s book.



\subsubsection{Improved Reward Generalization}
\label{sec:improved-reward-generalization}

\begin{figure*}[!t]
  \centering
  \input{section4/Figures/figure-methods-improving-reward-generalization}
  \caption{
    When training reward models, we can use parameter freezing and regularization methods to preserve the features of the LLM and thereby improve reward generalization.
  }
  \label{fig:improve-reward-generalization}
\end{figure*}

As discussed in Section \ref{sec:training-reward-models}, reward models are trained using preference data and subsequently used to optimize LLMs. However, a problem arises: the data distribution during LLM optimization may differ from the distribution of the preference data, making it challenging for the reward model to generalize to unseen input-output pairs. 

A well-known failure mode associated with this problem is commonly referred to as \textit{overoptimization} or \textit{reward hacking}, where the optimization stage improves the reward model but deteriorates the alignment with true rewards \citep{gao-etal:2023scaling,eisenstein-etal:2023helping}. This mode occurs because the reward model may incorrectly assign high rewards to unseen input-output pairs, leading the LLM to learn and optimize for behaviours that do not truly align with the desired behaviours and objectives. For example, consider a scenario from Section \ref{sec:policy-gradient} where the reward model is designed to favor informative and accurate outputs for a homework assistant. However, if the reward model fails to generalize to an overly verbose output and assigns a high reward to it (perhaps due to its length or certain keywords), the LLM may learn to prioritize generating a long output, which is not actually more informative but reward better according to the reward model. Consequently, the LLM becomes misaligned with its true objectives—delivering concise and relevant information—because it optimizes for the incorrect reward signal from the reward model with weak generalization. 

Addressing this generalization problem is challenging, and no mature solution exists yet. The ideal approach would be to develop an oracle reward model that perfectly captures the true objectives of the task and generalizes across all input-output pairs during LLM optimization. However, creating such a model is extremely difficult due to the complexity of the real-world environment, as well as the challenge of collecting sufficient preference data. Instead, a more practical approach is to combine multiple reward models, improving generalization and proving more correct rewards \citep{coste-etal:2024reward}.

Given a set of reward models, combining them is straightforward, and in some cases, we can simply treat this problem as an ensemble learning problem. A simple yet common approach is to average the outputs of these models to obtain a more precise reward estimation:
\begin{eqnarray}
R_{\mathrm{combine}}(\mathbf{x}, \mathbf{y}) & = & \frac{1}{K} \sum_{k=1}^{K} w_k \cdot R_k(\mathbf{x},\mathbf{y})
\end{eqnarray}

where $R_k(\cdot)$ is the $k$-th reward model in the ensemble, $w_k$ is the weight of $r_k(\cdot)$, and $K$ is the number of reward models. This combined reward can then be used to supervise the training of a policy. In fact, there are many ways to combine different models; for example, one can make predictions using Bayesian model averaging or develop a fusion network to learn to combine the predictions from different models. Alternatively, one can frame this task as a multi-objective optimization problem and use multiple reward models to train the policy simultaneously. These methods have been intensively discussed in the literature on optimization and machine learning \citep{miettinen:1999nonlinear,Bishop:2006}.

On the other hand, to improve the generalization of reward models, it is important to prevent overfitting to preference data. Reward models are typically trained on an SFT LLM, which has a strong generalization capability. However, training the LLM with a large amount of preference data could lead to overfitting, which may ultimately reduce generalization performance. A common strategy to mitigate this issue is to employ a parameter freezing technique, which helps preserve the original features of the LLM while learning the reward model. Another practical approach is to add a regularization term to the preference learning loss function, which helps regulate the features of the LLM \citep{yang-etal:2024regularizing}. For example, we can use a simple SFT loss as regularization by adding a term that maximizes the probability of the preferred output $\mathbf{y}_a$ to Eq. (\ref{eq:pairwise-reward-loss-expectation}): 
\begin{eqnarray}
\mathcal{L}_\mathrm{reg}(\phi) & = & -\mathbb{E}_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b) \sim \mathcal{D}_r} \big[ \log \mathrm{Pr}_{\phi}(\mathbf{y}_a \succ \mathbf{y}_b | \mathbf{x}) + \alpha \log(\mathrm{Pr}_{\theta}(\mathbf{y}_{a}|\mathbf{x})) \big]
\end{eqnarray}

where $\alpha$ is a balancing factor, we further illustrate the freezing parameter and regularization methods in Figure \ref{fig:improve-reward-generalization}. Note that the regularization term applies only to the LLM. That is, when optimizing this term, only the parameters of the LLM are updated, while the parameters of the reward linear map remain fixed. Therefore, in this equation, the parameter associated with the regularization term is $\theta$, which specifically refers to the parameters of the LLM.

\begin{figure}[!t]
  \centering
  \input{section4/Figures/figure-generative-reward-model-architecture}
  \caption{In generative reward models, we use an LLM to predict the label token given a prompt, an input, and a pair of outputs. This model can be trained in the same way as standard LLMs.}
\label{fig:generative-reward-model-architecture}
\end{figure}

\subsubsection{Generative Reward Models}
\label{sec:generative-reward-models}
Reward models are typically trained as discriminative models to assign numerical rewards to outputs and classify them as preferred or dispreferred. However, this method does not leverage the text-generation capabilities for which LLMs are fundamentally designed \citep{zhang-etal:zhang2024generative,wang-etal:wang2025gram}. For example, the discriminative reward model can not perform CoT reasoning. To address this, an LLM can alternatively be employed as a reward model, thus endowing it with the ability to engage in text generation and reasoning, as depicted in Figure \ref{fig:generative-reward-model-architecture}. This model works as follows. First, we input a prompt $\mathbf{c}$, along with the tuple $(\mathbf{x},\mathbf{y}_{a},\mathbf{y}_{b})$, to the LLM. The prompt is a description of the task, as demonstrated in the example below. 

\vspace{0.5em}
\begin{tcolorbox}[frame empty]

\begingroup
\setlength{\leftskip}{2em}
\setlength{\rightskip}{2em}

You are given two outputs to an input. Evaluate which output is better based on quality, relevance, and clarity. If the first output is better, return `A'. If the second output is better, return `B'.

\endgroup

\end{tcolorbox}
\vspace{0.5em}

Then, the LLM predicts subsequent tokens based on this input sequence. Let $w$ be the label token predicted by the LLM. If $w=\text{A}$, it indicates a preference for $\mathbf{y}_a$ over $\mathbf{y}_b$; if $w=\text{B}$, then $\mathbf{y}_b$ is preferred. Note that here $\mathbf{y}_a$ and $\mathbf{y}_b$  do not have a pre-defined preference relationship as described in Section \ref{sec:training-reward-models}. Their relationship is instead represented by the label token.

The loss function can be defined as the log-probability of predicting `A':
\begin{eqnarray}
\mathcal{L}_\mathrm{gen}(\theta) & = & -\mathbb{E}_{(\mathbf{c}, \mathbf{x},\mathbf{y}_a,\mathbf{y}_b) \sim \mathcal{D}_r} \big[ \log \mathrm{Pr}_{\theta}(w=\text{A}|\mathbf{s}) \big]
\label{eq:gen-reward-modeling}
\end{eqnarray}

where $\mathbf{s}$ denotes the string $[\mathbf{c},\mathbf{x},\mathbf{y}_a,\mathbf{y}_b]$\footnote{In this work, we will use $\mathbf{s}$ interchangeably to refer to either the tuple of a training sample or a string representing that tuple.}, and $\pi_{\phi}(\cdot)$ denotes the probability of token prediction by the LLM. 

Although we discuss methods for training a generative reward model here, an interesting question arises: how can we use the generative reward model in RLHF? We provide some guidance as follows. Specifically, when applying this generative reward model to provide a reward for an input-output pair $(\mathbf{x}',\mathbf{y}')$, we can generate a reference output $\mathbf{y}_{\mathrm{ref}}$ by using the LLM, for example, through greedy search, and concatenate $\mathbf{x}'$, $\mathbf{y}'$ and $\mathbf{y}_{\mathrm{ref}}$ into $\mathbf{s}' = [\mathbf{c}, \mathbf{x}', \mathbf{y}', \mathbf{y}_{\mathrm{ref}}]$. 
Additionally, to mitigate the positional bias problem \citep{wang-etal:2023large}, we can introduce an alternative input order by transposing the positions of output, i.e., presenting $\mathbf{y}_{\mathrm{ref}}$ before $\mathbf{y}'$, to construct a secondary input string $\mathbf{s}'_{T} = [\mathbf{c}, \mathbf{x}', \mathbf{y}_{\mathrm{ref}}, \mathbf{y}']$.  
The reward for $(\mathbf{x}', \mathbf{y}')$ is thus defined as the log-probability that $\mathbf{y}'$ is preferred over $\mathbf{y}_{\mathrm{ref}}$:
\begin{eqnarray}
r_{\phi}(\mathbf{x}', \mathbf{y}') & = & \frac{\mathrm{Pr}_{\theta}(w=\text{A}|\mathbf{s}')+\mathrm{Pr}_{\theta}(w=\text{B}|\mathbf{s}'_{T})}{2}
\label{eq:apply-generative-rm}
\end{eqnarray}

where the reward ranges from 0 to 1. 

To further improve the generative reward model, we can label the relevant explanation to enable the generative reward model to produce a CoT rationale \citep{zhang-etal:2024generative,wang-etal:wang2026gramrr}. In this case, we use a prompt $\mathbf{c}$ with a CoT rationale generation instruction, as shown below.

\vspace{0.5em}
\begin{tcolorbox}[frame empty]

\begingroup
\setlength{\leftskip}{2em}
\setlength{\rightskip}{2em}

You are given two outputs to a user input. Evaluate which output is better based on quality, relevance, and clarity. If the first output is better, return `A'. If the second output is better, return `B'. \uline{Note that, before presenting the evaluation results, you should provide a detailed analytical process and reasoning steps.}

\endgroup

\end{tcolorbox}
\vspace{0.5em}

We can then define a new loss function for training the generative reward model as follows:
\begin{eqnarray}
\mathcal{L}_\mathrm{gen}(\theta) & = & -\mathbb{E}_{(\mathbf{c},\mathbf{x},\mathbf{y}_a,\mathbf{y}_b,\mathbf{rat}) \sim \mathcal{D}_r} \big[ \log \mathrm{Pr}_{\theta}(\mathbf{rat}|\mathbf{s}) + \log \mathrm{Pr}_{\theta}(w=\text{A}|[\mathbf{s},\mathbf{rat}]) \big]
\end{eqnarray}

where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.

\subsubsection{Reward Model Evaluation}

After training a reward model, \textit{how we to evaluate the reward model?}  a common practice for evaluating
the reward is directly assessing the performance of the aligned LLM. While this practice can respond to final metrics, it incurs significant computational costs. Additionally, this approach 



1. Using RLHF to evaluate it. \\ 
2. Using pair ranking to evaluate it (RM-Bench, Reward-Bench, so on.). \\ 
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\

\subsection{Better Advantage Estimation}
Accurate advantage estimation is critical in RL, particularly when optimizing the policy model to truly reflect the potential benefits of different actions (or tokens in training LLMs). In this subsection, we will delve into methods for improving advantage estimation, providing more accurate advantages in the process of optimizing the policy model.

\subsubsection{Temporal Difference-based Advantage Estimation}
Let us first review the Monte Carlo-based advantage estimation. As discussed in Section \ref{sec:reduce-gradient-variance}, outputs are sampled and their values computed. The advantage can be obtained by comparing the actual received rewards with the predicted values at time step $t$:
\begin{eqnarray}
    A_{t} & = & \sum_{t=k}^{T}r_{k} - V_{t}
    \label{eq:advantage-estimation-monte-carlo}
\end{eqnarray}

While the Monte Carlo-based estimation of advantage provides a relatively stable training process, there are two main challenges associated with it:
\begin{itemize}
    \item Sampling an entire output for each input is necessary. In some cases where immediate rewards are available, it could be more efficient to update the policy model with partial output. Unfortunately, when using Eq. (\ref{eq:advantage-estimation-monte-carlo}) for advantage estimation, this becomes unfeasible. This is because we must compute the term $\sum_{t=k}^T r_{k}$, which requires the rewards of the entire output.
    \item This method still results in high variance. Since a single sample operation might be influenced by random factors, the estimated results may not be stable, as illustrated in Figure \ref{fig:monte-carlo-advantage-issue}.
\end{itemize}

\begin{figure*}[!t]
    \centering
    \input{section4/Figures/figure-issue-monte-carlo}
    \caption{
    Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq. (\ref{eq:langth-based-reward-function}).
    Typically, if the value model is well-trained, it would predict an average $V_{t}$ of 6 at time step $t$, based on most outputs having around 60 tokens in length. However, for outlier outputs like $\mathbf{y}_{d,t:T}$, which might extend up to 300 tokens, the advantage estimation can exhibit high variance. For instance, the advantage for such an outlier could be computed as $V_{t}(\mathbf{x},\mathbf{y}_{d,<t},y_{d,t}) = \sum_{k=t}^T r_{k} - V_t = 30 - 6 = 24$, a stark contrast to another output, say $\mathbf{y}_{1,t:T}$, which is closer to the average length, where the advantage might be $V_{t}(\mathbf{x},\mathbf{y}_{1,<t},y_{1,t}) = -1$. This discrepancy leads to high gradient variance, potentially destabilizing the learning process.
    }
    \label{fig:monte-carlo-advantage-issue}
\end{figure*}

To address these challenges, an alternative approach is the temporal difference-based advantage estimation \citep{sutton-and-richard:1988learning}, which allows for estimating the advantage using incomplete outputs. More specifically, this method computes the difference between the predicted values at consecutive time steps (i.e., current and next stapes), adjusting the advantage estimate based on new information as it becomes available, thus reducing the dependency on the entire output. In a practical implementation, the temporal difference-based advantage estimation can be given by
\begin{eqnarray} 
A_{t}^{\mathrm{TD}} & = & r_{t} + \gamma V_{t+1} - V_{t} 
\label{eq:td-advantage-estimation} 
\end{eqnarray}

By focusing on the current and next steps, temporal difference-based advantage estimation minimizes the influence of distant future events. This leads to lower variance in optimizing the policy model and improves the stability of policy training.

\subsubsection{Generalized Advantage Estimation}
While temporal difference-based estimation effectively reduces variance, it can also increase estimation bias due to its heavy reliance on the predictions of the value model. By contrast, one effective method is generalized advantage estimation (GAE), which is one of the most popular advantage estimation methods used in LLMs and other fields \citep{schulman-etal:2015high}. This method mainly refines the advantage estimation by using exponentially weighted averages of temporal difference advantages across multiple future steps. More specifically, consider a bias $\delta_t$ in the temporal difference at time step $t$, defined as
\begin{eqnarray}
    \delta_t & = & r_t + \gamma V_{t+1} - V_{t}
\end{eqnarray}

GAE introduces parameters that dynamically balance the trade-off between bias and variance. This basic idea is to use the weighted sum of multi-step temporal difference bias as an estimation of the advantage, combining the advantages of Monte Carlo (high variance and low bias) and temporal difference methods (low variance and high bias). This combination can be expressed as
\begin{eqnarray}
    A_t^{\mathrm{gae}} & = & \sum_{n=1}^{T-t} (\gamma \lambda)^{n} \delta_{t+n}
\end{eqnarray}

We can recursively develop the above model:
\begin{eqnarray}
    A_t^{\mathrm{gae}} & = & \delta_{t} + \gamma \lambda A_{t+1}^{\mathrm{gae}}
    \label{eq:GAE-advantage-estimation}
\end{eqnarray}

This recursive formulation shows that the current advantage estimate incorporates not only the immediate temporal difference error at time $t$ but also the estimated advantages of future time steps, weighted and adjusted by the parameters $\gamma$ and $\lambda$. This method effectively combines the strengths of both Monte Carlo and temporal difference methods, leading to a more stable and efficient training process.

\subsubsection{Group Relative Policy Optimization}

Although the methods discussed in the previous subsections effectively estimate advantage during the learning process, they all rely on a value model that is typically trained concurrently with the policy model. This dependence not only complicates the training process but also requires significant computational resources when applied to training LLMs. Given this, a natural idea arises: could we use more lightweight methods to compute the advantage without relying on the value model? By doing so, we could forego the value model component, thereby improving efficiency.

\begin{figure*}[!t]
    \centering
    \input{section4/Figures/figure-grpo}
    \caption{
    Group relative policy optimization. Compared to PPO, it foregoes the value model by estimating advantages directly from the rewards of a group of outputs. This method can simplify the training process and reduce the computational resources.
    }
    \label{fig:workflow-grpo}
\end{figure*} 

In fact, this is entirely feasible, and one example of such an approach is \citet{shao-etal:2024deepseekmath}'s approach, called group relative policy optimization or GRPO for short. In the GRPO approach, the advantage is computed based on the relative rewards of the outputs within each group. More specifically, given an input $\mathbf{x}$, GRPO samples a group of outputs $\mathbf{Y}=\{\mathbf{y}_1, \mathbf{y}_2, \cdots, \mathbf{y}_{G}\}$. Then, the rewards for each output are computed using a reward model or a reward function, denoted as $\mathbf{R} = \{R(\mathbf{x}, \mathbf{y}_{1}), R(\mathbf{x}, \mathbf{y}_{2}), \dots, R(\mathbf{x}, \mathbf{y}_{G})\}$. The advantage of the $i$-th output can be computed through a relative reward in comparison to the other outputs:
\begin{eqnarray}
    A^{\mathrm{grpo}}_{i,t} = \frac{R(\mathbf{x}, \mathbf{y}_{i})-\mathrm{Mean}(\mathbf{R})}{\mathrm{Std}(\mathbf{R})}
    \label{eq:advantage-grpo}
\end{eqnarray}

where the $\mathrm{Mean}(\cdot)$ and $\mathrm{Std}(\cdot)$ represent the mean and standard deviation functions, respectively. Note that the advantage computation used here differs slightly from that in PPO. In this case, this computed advantage is applied to each time step, whereas in the traditional PPO, we separately compute an advantage for each time step. At this point, we can also use the process reward model to supplement the advantage computation, allowing us to compute an advantage specific to each time step. A simple way to achieve this is by implementing Eq. (\ref{eq:advantage-grpo}) at each time step, where the rewards are computed using the process reward model. As a result, the objective for GRPO can be defined according to Eq. (\ref{eq:ppo-loss}):
\begin{eqnarray}
    \mathcal{L}_{g}(\theta) & = & - \mathbb{E}_{\mathbf{x} \sim \mathcal{S}_{x}} \frac{1}{G} \sum_{i=1}^{G} \left[
    \sum_{t=1}^{T} \mathrm{Clip}\big(\frac{\mathrm{Pr}_{\theta}(y_{i,t}|\mathbf{x},\mathbf{y}_{i,<t})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_{i,t}|\mathbf{x},\mathbf{y}_{i,<t})}A_{i,t}^{\mathrm{grpo}}\big) - \beta \mathrm{Penalty}
    \right] 
    \label{eq:grpo-loss}
\end{eqnarray}
In addition to optimizing the advantage computation, GRPO modifies the penalty term to enhance the optimization objective further. However, since our focus here is on advantage estimation, we will not delve into its details. Interested readers can refer to the GRPO paper for further details. The workflow of GRPO is illustrated in Figure \ref{fig:workflow-grpo}, showcasing how these modifications integrate into the training process.

Moreover, there are other methods like GRPO that design advantage estimation without relying on a value model for training LLMs, such as those discussed in \citet{li-etal:2023remax} and \citet{hu:2025reinforce++}.



\subsection{Efficient Reinforcement Learning Methods}
Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in the context of LLMs due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.

In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.

\subsubsection{Dynamic Sampling}
As mentioned in Section \ref{sec:policy-gradient}, training LLMs with RL typically requires sampling for each sample $\mathbf{x}$ in $\mathcal{S}_x$, which introduces significant computational time overhead. This becomes particularly challenging in large-scale RL scenarios, where thousands of training steps need to be conducted, such as in DeepSeek-R1 \citep{guo:2025deepseek}. 

From the perspective of LLM inference, there are many methods to reduce the time overhead of sampling: since the sampling process is implemented by LLM inference, we have reason to believe that any method that accelerates inference can also be applied to reduce the time required for sampling. 
Examples include KV-Cache \citep{pope-etal:2023efficiently}, quantization \citep{zhao-etal:2024atom}, and speculative decoding \citep{chen-etal:2023accelerating}. However, in this section, we will not discuss these methods in detail, as there are numerous approaches, each addressing different aspects of LLM inference. We refer the interested readers to these papers for more details. Instead, we will focus on one particular method that aims to reduce sampling overhead from the perspective of optimizing RL for training LLMs.

Before discussing specific methods, let us first review the purpose of sampling. Given a sample $\mathbf{x}$, the goal of sampling is to explore a better output $\mathbf{y}$ from the policy model. This process allows us to evaluate different possible outputs in order to find those that lead to improved outcomes, enabling the model to make more informed decisions and optimize its performance on the state. Unlike typical RL scenarios, where policy models may start with limited information or random initialization, the policy model in the context of LLMs is usually quite advanced. These LLMs often begin as pre-trained models equipped with substantial knowledge acquired through pre-training and SFT. As a result, for certain samples, we can consider that the policy model may already perform well without the need for further exploration and optimization. This can be viewed through the lens of the classic RL dilemma of exploration versus exploitation \citep{Sutton-and-Barto:2018RL}. In such cases, exploration may be less necessary, and exploiting the knowledge already embedded in the pre-trained model could be more effective. Thus, a balance must be struck between exploring new possibilities and leveraging the pre-existing knowledge of the policy model to maximize efficiency. 

To achieve this balance between exploration and exploitation, one simple and straightforward approach to improving the efficiency of sampling is to identify the samples that are required to explore and perform sampling only for those samples. Given an input-only dataset $\mathcal{S}_x = \{\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N\}$, where $N$ denotes the size of the dataset, we can use a reward model to evaluate the need for exploration by the policy model. First, we generate $\hat{\mathbf{y}}$ for each $\mathbf{x}$ using greedy search. Then, by taking $\mathbf{x}$ and $\hat{\mathbf{y}}$ as input to the reward model, the reward for the $k$-th sample $\mathbf{x}_k$ is computed as $R_{\phi}(\mathbf{x}_{k}, \hat{\mathbf{y}}_{k})$.

At this point, we can directly use the reward value to determine whether the policy model requires further exploration and optimization for the sample. Specifically, we set a threshold $r_{\mathrm{upper}}$. If the reward for a sample exceeds $r_{\mathrm{upper}}$, we can consider that no further sampling and optimization is required. Otherwise, exploration and optimization are required. This approach enables selective sampling and optimization, improving the efficiency of the overall process by focusing resources on the most promising samples that are likely to benefit from further exploration \citep{wang-etal:2024esrl}. A challenge in implementation is determining the value of $r_{\mathrm{upper}}$. Typically, we set it as the average reward:
\begin{eqnarray}
    r_\mathrm{upper} & = & \frac{1}{N} \sum_{i=1}^{N} R_{\phi}(\mathbf{x}_{i}, \hat{\mathbf{y}}_{i})
\end{eqnarray}

A similar dynamic sampling strategy, known as prioritized sampling, has proven effective in large-scale RL scenarios \citep{kimi-team:2025kimi}. However, we must consider another important factor in dynamic sampling. For samples with very low rewards, the policy model may lack the capacity to learn and optimize effectively. Excessive exploration of such samples can be inefficient. Consequently, we can introduce an additional threshold $r_{\mathrm{lower}}$ to eliminate unnecessary exploration of these low-reward samples, thereby enhancing efficiency. In this case, the policy model can concentrate its exploration and optimization efforts on a select group of samples, avoiding wasted sampling time on those that are unlikely to explore a more optimal output \citep{li-etal:2025limr}.

A concept closely related to the discussion here is sample efficiency. This topic has been widely discussed in recent literature. For example, some research on knowledge distillation seeks to select a smaller subset of samples to achieve more optimal performance \citep{wang-etal:2021selective}. More recently, much work on instruction sample selection in SFT aims to improve SFT efficiency by learning from a small number of instruction samples \citep{chen-etal:2023alpagasus}.

\subsubsection{Lightweight Reward Methods}
\label{sec:lightweight-reward-methods}

An additional computational overhead in training LLMs with RL is the frequent need to call the reward model to compute rewards. Furthermore, to ensure the reliability and generalizability of the reward model, we often scale it to a large size \citep{gao-etal:2023scaling}, which increases the computational time for reward computations.

One approach for mitigating this issue is to explore rule-based rewards, for example, as discussed in Section \ref{sec:policy-gradient}, the length of the outputs could be utilized as a reward. Instead of relying on large-scale, resource-intensive models, rule-based rewards assign rewards using predefined and task-specific rules. These rules, often derived from expert knowledge or task-specific requirements, are computationally inexpensive and provide fast, effective feedback to the policy model. By replacing or complementing traditional reward models with rule-based approaches, we can reduce both the time and computational cost associated with reward computations while still offering meaningful guidance for the learning process. This approach is also based on the understanding that, in some tasks, human preferences are simple to describe and do not require the complexity of training a reward model. Instead, we can use rules to express these preferences efficiently.

Here, in addition to the previously mentioned length-based rewards, we further demonstrate two examples of rule-based rewards. First, considering the output format, we can design rewards that encourage the policy model to generate outputs adhering to specific formats. For example, in response to the input "Give me three tips to improve my accuracy in solving math problems," we can assign rewards based on whether the output can be parsed as JSON correctly and include ``tip1'', ``tip2'', and ``tip3'' as the keys.
% Example 1
\input{section4/Figures/figure-examples-of-rule-based-reward-length}

Similarly, in mathematical reasoning tasks, where the desired outcome is a correct final answer, we can design a rule to verify the correctness of the answer and provide rewards based on whether the result is correct \citep{havrilla-etal:2024teaching,shao-etal:2024deepseekmath}.
% Example 2
\input{section4/Figures/figure-examples-of-rule-based-reward-math}

In addition to improving computational efficiency, another benefit of using rule-based rewards is their stability. Compared to reward models, rule-based rewards are less prone to issues such as reward hacking. As the rules are predefined and not subject to the same complex training process as traditional models, the risk of misalignment between the optimization objectives of the policy model and the oracle rewards is reduced. This stability makes rule-based rewards a more predictable and reliable approach, ensuring that the learning process of the policy model is less influenced by prediction errors or biases that may arise from the reward model. Furthermore, recent research has shown that employing rule-based rewards can significantly enhance the reasoning capabilities of LLMs through RL, underscoring its practical effectiveness \citep{guo:2025deepseek,xie-etal:2025logic}.

Another lightweight reward method is to consider achieving a lightweight reward model with fewer parameters, which would reduce both computational time and resource usage. There are several methods to achieve this. For example, given that reward models fundamentally utilize an LLM, many efficient methods, such as Mixture of Experts (MoE) \citep{masoudnia-etal:2014mixture} and model pruning \citep{sun-etal:2023simple}, originally developed for LLMs can be adapted to the reward model to improve its efficiency. Furthermore, knowledge distillation techniques offer a pathway to construct a smaller reward model that learns to emulate the behaviour of a larger one \citep{wang-etal:2023learning}. These approaches not only improve the efficiency of the reward model but also maintain its ability to deliver accurate rewards.  


\subsection{Direct Preference Optimization}

\begin{figure*}[!t]
    \centering
    \input{section4/Figures/figure-rlhf-vs-dpo}
    \caption{Standard PPO vs. DPO in training LLMs. In PPO, the human preference data is used to train a reward model, which is then employed to train the policy and the value function. In DPO, the use of human preference data is more direct, and the policy is trained on this data without the need for reward model training.}
    \label{fig:comparsion-rl-dpo}
\end{figure*}

Although learning reward models is a standard step in RL, it makes the entire training process much more complex than supervised training. Training a reliable reward model is itself not an easy task and a poorly trained reward model can greatly affect the outcome of policy learning. We now consider an alternative method, called direct preference optimization (DPO), which simplifies the training framework by eliminating the need to explicitly model rewards \citep{rafailov:2023direct}. This method directly optimizes the policy model based on user preferences rather than developing a separate reward model. As a result, we can achieve human preference alignment in a supervised learning-like fashion. Figure \ref{fig:comparsion-rl-dpo} shows a comparison of the standard PPO method and the DPO method.

DPO simplifies the RL process for LLMs by utilizing a straightforward cross-entropy loss, which streamlines the learning process and enhances its stability. Rather than delving into the detailed derivation of the DPO loss function, we will discuss its optimization objective from the perspective of optimizing an LLM. The loss function derived from a Bradley-Terry reward model can be given by
\begin{eqnarray}
    \mathcal{L}_{\mathrm{dpo}}(\theta) & = & - \mathbb{E}_{(\mathbf{x}, \mathbf{y}_{a}, \mathbf{y}_{b}) \sim \mathcal{D}_{r}} \big[ \log\mathrm{Sigmoid} \big( \beta \log \frac{\mathrm{Pr}_{\theta}(\mathbf{y}_{a}|\mathbf{\mathbf{x}})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_{a}|\mathbf{\mathbf{x}})} - \beta \log \frac{\mathrm{Pr}_{\theta}(\mathbf{y}_{b}|\mathbf{\mathbf{x}})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_{b}|\mathbf{\mathbf{x}})} \big) \big]
    \label{eq:dpo-loss}
\end{eqnarray}

This loss function utilizes the implicit reward for DPO training, which replaces the need for an external reward model. The implicit reward is computed as the log-ratio of probabilities:
\begin{eqnarray}
    R(\mathbf{x}, \mathbf{y}) = \beta \frac{\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{\mathbf{x}})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}|\mathbf{\mathbf{x}})}
\end{eqnarray}

\begin{figure*}[!t]
    \centering
    \input{section4/Figures/figure-dpo}
    \caption{
    Illustration of the optimization objective of DPO from a hypothesis space perspective. The right-top chart shows the probability distributions over different hypotheses before optimization, where $\mathbf{y}_{a}$ is the preferred output and $\mathbf{y}_{b}$ is the dispreferred output. The right-bottom chart shows the probabilities after applying DPO, where the likelihood of the preferred output increases and that of dispreferred outputs decreases. Note that during this optimization process, outputs similar to the preferred output $\mathbf{y}_{a}$ (such as $\mathbf{y}_{1}$) will obtain an increased probability, while those similar to the less preferred output $\mathbf{y}_{b}$ (such as $\mathbf{y}_{2}$) will experience a decreased probability, due to the generalization of the model. 
    }
    \label{fig:understand-dpo-optimization-objective}
\end{figure*}

Here, we can consider that the DPO directly models human feedback through the relative likelihood of outputs under the policy and reference models. This method provides a more straightforward approach than traditional methods that use a separate reward model and then apply RL algorithms like PPO to adjust the probabilities of sampled outputs. In this way, we can further understand the optimization objective of the DPO by leveraging the hypothesis space as mentioned in Section \ref{sec:policy-gradient}. In practice, as illustrated in Figure \ref{fig:understand-dpo-optimization-objective}, we can see the DPO as reranking in the hypothesis space through the Bradley-Terry model approach. Specifically, this method increases the probabilities of preferred outputs in preference data while decreasing those of dispreferred ones. Similar to TRPO, the DPO also incorporates a penalty derived from the reference model, which ensures that updates remain within a trusted behaviour space for the policy model.

However, there are two sides to every coin. While DPO simplifies the RL process, it also introduces limitations. Notably, since DPO eliminates the exploration phase during training, its potential peak performance is generally considered lower compared to RL-based methods like PPO, which incorporate exploration to discover more effective policies. In other words, the performance of DPO is inherently bounded by the labeled preferred outputs in the preference data. To address this limitation, current approaches focus on ensuring high-quality preferred outputs, either by employing advanced LLMs like GPT-4 or through human labeling \citep{cui-etal:2023ultrafeedback,morimura-etal:2024filtered}.

\begin{table*}[!t]
  \centering
  \resizebox{\textwidth}{!}{
  \input{section4/Tables/dpo_variants}}
  \caption{DPO variants and their optimization objectives.}
  \label{tab:dpo_varients}
\end{table*} 

Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table \ref{tab:dpo_varients}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.

