@@ -9,7 +9,7 @@ In this subsection, we introduce the fundamentals of supervised fine-tuning for
...
@@ -9,7 +9,7 @@ In this subsection, we introduce the fundamentals of supervised fine-tuning for
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_0...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
Specifically, let $\mathbf{x}=x_1...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
\INPUT the optimized reward model $r_{\phi}(\cdot)$, the reference model $\pi_{\theta_{\mathrm{ref}}}(\cdot)$, the initialized value model $V_{\omega}(\cdot)$, the policy model $\pi_{\theta}(\cdot)$, the input-only dataset $S_{x}$;
\INPUT the optimized reward model $R_{\phi}(\cdot)$, the reference model $\pi_{\theta_{\mathrm{ref}}}(\cdot)$, the initialized value model $V_{\omega}(\cdot)$, the policy model $\pi_{\theta}(\cdot)$, the input-only dataset $\mathcal{S}_{x}$;
\State sample an output from $\pi_{\theta_\mathrm{old}}$: $\mathbf{y}\sim\pi_{\theta_\mathrm{old}}(\cdot|\mathbf{x})$
\State sample an output from $\pi_{\theta_\mathrm{old}}$: $\mathbf{y}\sim\pi_{\theta_\mathrm{old}}(\cdot|\mathbf{x})$
\State compute the reward for $[\mathbf{x},\mathbf{y}]$: $r_{\phi}(\mathbf{x},\mathbf{y})$
\State compute the reward for $[\mathbf{x},\mathbf{y}]$: $R_{\phi}(\mathbf{x},\mathbf{y})$
\State\%\%\% roll out
\State\%\%\% roll out
\For{ppo\_epoch\_idx=1 to $\mathrm{PPO\_EPOCH}$}
\For{ppo\_epoch\_idx=1 to $\mathrm{PPO\_EPOCH}$}
\State\%\%\%\textit{compute policy model loss and value model loss through advantages and returns}
\State\%\%\%\textit{compute policy model loss and value model loss through advantages and returns}
...
@@ -15,7 +15,7 @@
...
@@ -15,7 +15,7 @@
\For{t=1 to T}
\For{t=1 to T}
\State compute the penalty through Eq. (\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log\mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})-\log\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State compute the penalty through Eq. (\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log\mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})-\log\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State\hspace{0.5cm} predict the next time step value $V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})$
\State\hspace{0.5cm} predict the next time step value $V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})$
\State\textbf{end if}
\State\textbf{end if}
\State\%\%\%\textit{compute policy model loss}
\State\%\%\%\textit{compute policy model loss}
\State compute the advantage $A(\mathbf{x},\mathbf{y}_{<t},y_{t})$ through Eq. (\ref{eq:advantage}): $A(\mathbf{x},\mathbf{y}_{<t},y_{t})=r_{t}+\gamma V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the advantage $A(\mathbf{x},\mathbf{y}_{<t},y_{t})$: $A(\mathbf{x},\mathbf{y}_{<t},y_{t})=r_{t}+\gamma V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the policy model loss through Eq. (\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p}=\mathrm{loss}_{p}+\mathrm{Clip}\Big(\frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})}\Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State compute the policy model loss through Eq. (\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p}=\mathrm{loss}_{p}+\mathrm{Clip}\Big(\frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})}\Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State\%\%\%\textit{compute value model loss}
\State\%\%\%\textit{compute value model loss}
\State compute the return $\mathrm{Return}_{t}$: $\mathrm{Return}_{t}=A(\mathbf{x},\mathbf{y}_{<t},y_t)+V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the return $\mathrm{Return}_{t}$: $\mathrm{Return}_{t}=A(\mathbf{x},\mathbf{y}_{<t},y_t)+V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
@@ -173,7 +173,7 @@ In practice, there are many ways to implement the value function. One simple app
...
@@ -173,7 +173,7 @@ In practice, there are many ways to implement the value function. One simple app
V_{t}& = &\mathbf{h}_{y_{t}}\mathbf{W}_{v}
V_{t}& = &\mathbf{h}_{y_{t}}\mathbf{W}_{v}
\end{eqnarray}
\end{eqnarray}
where $\mathbf{h}_{y_{t}}$ is a $d$-dimensional vector, and $\mathbf{W}{v}$ is a $d \times1$ linear mapping matrix. This value model is typically trained using the Temporal Difference (TD) error. This training method relies on the principle that the difference between the values at adjacent time steps should correspond to the reward received at the former time step, i.e., $V_t - V_{t+1}= r_t$. Suppose the value model is parameterized by $\omega$. Given a sequence $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$, the loss function is given by
where $\mathbf{h}_{y_{t}}$ is a $d$-dimensional vector, and $\mathbf{W}_{v}$ is a $d \times1$ linear mapping matrix. This value model is typically trained using the Temporal Difference (TD) error. This training method relies on the principle that the difference between the values at adjacent time steps should correspond to the reward received at the former time step, i.e., $V_t - V_{t+1}= r_t$. Suppose the value model is parameterized by $\omega$. Given a sequence $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$, the loss function is given by
where $\gamma\in[0,1]$ is the discount factor that adjusts the importance of future rewards. When $\gamma$ is set to less than 1, it signifies that early rewards are considered more important than future rewards. This basic idea is also applied in other fields. For example, in LLMs, it has been demonstrated that early token generation plays a crucial role, as it can influence the style and accuracy of the entire output \citep{wang-and-zhou:2024chain}.
where $\gamma\in[0,1]$ is the discount factor that adjusts the importance of future rewards. When $\gamma$ is set to less than 1, it signifies that early rewards are considered more important than future rewards. This basic idea is also applied in other fields. For example, in LLMs, it has been demonstrated that early token generation plays a crucial role, as it can influence the style and accuracy of the entire output \citep{wang-and-zhou:2024chain}.
...
@@ -227,6 +228,7 @@ Addressing this issue involves considering trust regions in optimization \citep{
...
@@ -227,6 +228,7 @@ Addressing this issue involves considering trust regions in optimization \citep{
At the time step $t$, we can obtain the penalty as
At the time step $t$, we can obtain the penalty as
@@ -249,7 +251,7 @@ A further improvement to TRPO involves addressing the gradient variance problem
...
@@ -249,7 +251,7 @@ A further improvement to TRPO involves addressing the gradient variance problem
In Eq. (\ref{eq:importance-sampling-with-penalty}), the range of the ratio function is from $[0, +\infty)$, which could lead to high gradient variance in the learning process. To mitigate this problem, clipping is commonly employed to limit the magnitude of importance weights, thereby preventing excessively large updates and promoting stability in the learning process. A clipped version can be given by
In Eq. (\ref{eq:importance-sampling-with-penalty}), the range of the ratio function is from $[0, +\infty)$, which could lead to high gradient variance in the learning process. To mitigate this problem, clipping is commonly employed to limit the magnitude of importance weights, thereby preventing excessively large updates and promoting stability in the learning process. A clipped version can be given by
where $\mathbf{W}{r}$ is a $d \times1$ linear mapping matrix, and $\phi$ represents the parameters of the reward model, which includes both the parameters of the LLM and the $\mathbf{W}{r}$. This architecture of the reward model is illustrated in Figure \ref{fig:reward-model}.
where $\mathbf{W}_{r}$ is a $d \times1$ linear mapping matrix, and $\phi$ represents the parameters of the reward model, which includes both the parameters of the LLM and the $\mathbf{W}_{r}$. This architecture of the reward model is illustrated in Figure \ref{fig:reward-model}.
\begin{figure*}[!t]
\begin{figure*}[!t]
\centering
\centering
...
@@ -331,7 +333,7 @@ where $(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)$ is drawn from a human-annotated d
...
@@ -331,7 +333,7 @@ where $(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)$ is drawn from a human-annotated d
\hat{\phi}& = &\argmin_{\phi}\mathcal{L}_r(\phi)
\hat{\phi}& = &\argmin_{\phi}\mathcal{L}_r(\phi)
\end{eqnarray}
\end{eqnarray}
Since the reward model itself is also an LLM, we can directly reuse the Transformer training procedure to optimize the reward model. The difference from training a standard LLM is that we only need to replace the cross-entropy loss with the pairwise comparison loss as illustrated in Figure \ref{fig:training-reward-models}. After the training of the reward model, we can apply the trained reward model $r_{\hat{\phi}}(\cdot)$ to supervise the target LLM for alignment.
Since the reward model itself is also an LLM, we can directly reuse the Transformer training procedure to optimize the reward model. The difference from training a standard LLM is that we only need to replace the cross-entropy loss with the pairwise comparison loss as illustrated in Figure \ref{fig:training-reward-models}. After the training of the reward model, we can apply the trained reward model $R_{\hat{\phi}}(\cdot)$ to supervise the target LLM for alignment.
It is worth noting that although we train the reward model to perform pairwise ranking, we apply it to score each input-output pair independently during the alignment process. The pairwise ranking objective ensures that the reward model is sensitive to subtle differences between outputs, but we rely on the continuous scores produced by the reward model to guide the optimization of the LLM. An advantage of this approach is that we can choose from or combine various ranking loss functions and still apply the resulting reward models in the same way as we have done in Section \ref{sec:improved-reward-generalization}. However, a challenge arises with this method: the reward model can provide only sparse rewards; that is, it offers a delayed reward rather than an intermediate one. Hence, in this case, we can obtain
It is worth noting that although we train the reward model to perform pairwise ranking, we apply it to score each input-output pair independently during the alignment process. The pairwise ranking objective ensures that the reward model is sensitive to subtle differences between outputs, but we rely on the continuous scores produced by the reward model to guide the optimization of the LLM. An advantage of this approach is that we can choose from or combine various ranking loss functions and still apply the resulting reward models in the same way as we have done in Section \ref{sec:improved-reward-generalization}. However, a challenge arises with this method: the reward model can provide only sparse rewards; that is, it offers a delayed reward rather than an intermediate one. Hence, in this case, we can obtain
where $R_k(\cdot)$ is the $k$-th reward model in the ensemble, $w_k$ is the weight of $r_k(\cdot)$, and $K$ is the number of reward models. This combined reward can then be used to supervise the training of a policy. In fact, there are many ways to combine different models; for example, one can make predictions using Bayesian model averaging or develop a fusion network to learn to combine the predictions from different models. Alternatively, one can frame this task as a multi-objective optimization problem and use multiple reward models to train the policy simultaneously. These methods have been intensively discussed in the literature on optimization and machine learning \citep{miettinen:1999nonlinear,Bishop:2006}.
where $R_k(\cdot)$ is the $k$-th reward model in the ensemble, $w_k$ is the weight of $R_k(\cdot)$, and $K$ is the number of reward models. This combined reward can then be used to supervise the training of a policy. In fact, there are many ways to combine different models; for example, one can make predictions using Bayesian model averaging or develop a fusion network to learn to combine the predictions from different models. Alternatively, one can frame this task as a multi-objective optimization problem and use multiple reward models to train the policy simultaneously. These methods have been intensively discussed in the literature on optimization and machine learning \citep{miettinen:1999nonlinear,Bishop:2006}.
On the other hand, to improve the generalization of reward models, it is important to prevent overfitting to preference data. Reward models are typically trained on an SFT LLM, which has a strong generalization capability. However, training the LLM with a large amount of preference data could lead to overfitting, which may ultimately reduce generalization performance. A common strategy to mitigate this issue is to employ a parameter freezing technique, which helps preserve the original features of the LLM while learning the reward model. Another practical approach is to add a regularization term to the preference learning loss function, which helps regulate the features of the LLM \citep{yang-etal:2024regularizing}. For example, we can use a simple SFT loss as regularization by adding a term that maximizes the probability of the preferred output $\mathbf{y}_a$ to Eq. (\ref{eq:pairwise-reward-loss-expectation}):
On the other hand, to improve the generalization of reward models, it is important to prevent overfitting to preference data. Reward models are typically trained on an SFT LLM, which has a strong generalization capability. However, training the LLM with a large amount of preference data could lead to overfitting, which may ultimately reduce generalization performance. A common strategy to mitigate this issue is to employ a parameter freezing technique, which helps preserve the original features of the LLM while learning the reward model. Another practical approach is to add a regularization term to the preference learning loss function, which helps regulate the features of the LLM \citep{yang-etal:2024regularizing}. For example, we can use a simple SFT loss as regularization by adding a term that maximizes the probability of the preferred output $\mathbf{y}_a$ to Eq. (\ref{eq:pairwise-reward-loss-expectation}):
\begin{eqnarray}
\begin{eqnarray}
...
@@ -176,7 +176,7 @@ The loss function can be defined as the log-probability of predicting `A':
...
@@ -176,7 +176,7 @@ The loss function can be defined as the log-probability of predicting `A':
\label{eq:gen-reward-modeling}
\label{eq:gen-reward-modeling}
\end{eqnarray}
\end{eqnarray}
where $\mathbf{s}$ denotes the string $[\mathbf{c},\mathbf{x},\mathbf{y}_a,\mathbf{y}_b]$\footnote{In this work, we will use $\mathbf{s}$ interchangeably to refer to either the tuple of a training sample or a string representing that tuple.}, and $\pi_{\phi}(\cdot)$ denotes the probability of token prediction by the LLM.
where $\mathbf{s}$ denotes the string $[\mathbf{c},\mathbf{x},\mathbf{y}_a,\mathbf{y}_b]$\footnote{In this work, we will use $\mathbf{s}$ interchangeably to refer to either the tuple of a training sample or a string representing that tuple.}, and $\mathrm{Pr}_{\theta}(\cdot)$ denotes the probability of token prediction by the LLM.
Although we discuss methods for training a generative reward model here, an interesting question arises: how can we use the generative reward model in RLHF? We provide some guidance as follows. Specifically, when applying this generative reward model to provide a reward for an input-output pair $(\mathbf{x}',\mathbf{y}')$, we can generate a reference output $\mathbf{y}_{\mathrm{ref}}$ by using the LLM, for example, through greedy search, and concatenate $\mathbf{x}'$, $\mathbf{y}'$ and $\mathbf{y}_{\mathrm{ref}}$ into $\mathbf{s}' =[\mathbf{c}, \mathbf{x}', \mathbf{y}', \mathbf{y}_{\mathrm{ref}}]$.
Although we discuss methods for training a generative reward model here, an interesting question arises: how can we use the generative reward model in RLHF? We provide some guidance as follows. Specifically, when applying this generative reward model to provide a reward for an input-output pair $(\mathbf{x}',\mathbf{y}')$, we can generate a reference output $\mathbf{y}_{\mathrm{ref}}$ by using the LLM, for example, through greedy search, and concatenate $\mathbf{x}'$, $\mathbf{y}'$ and $\mathbf{y}_{\mathrm{ref}}$ into $\mathbf{s}' =[\mathbf{c}, \mathbf{x}', \mathbf{y}', \mathbf{y}_{\mathrm{ref}}]$.
Additionally, to mitigate the positional bias problem \citep{wang-etal:2023large}, we can introduce an alternative input order by transposing the positions of output, i.e., presenting $\mathbf{y}_{\mathrm{ref}}$ before $\mathbf{y}'$, to construct a secondary input string $\mathbf{s}'_{T}=[\mathbf{c}, \mathbf{x}', \mathbf{y}_{\mathrm{ref}}, \mathbf{y}']$.
Additionally, to mitigate the positional bias problem \citep{wang-etal:2023large}, we can introduce an alternative input order by transposing the positions of output, i.e., presenting $\mathbf{y}_{\mathrm{ref}}$ before $\mathbf{y}'$, to construct a secondary input string $\mathbf{s}'_{T}=[\mathbf{c}, \mathbf{x}', \mathbf{y}_{\mathrm{ref}}, \mathbf{y}']$.
...
@@ -452,13 +452,13 @@ Accurate advantage estimation is critical in RL, particularly when optimizing th
...
@@ -452,13 +452,13 @@ Accurate advantage estimation is critical in RL, particularly when optimizing th
Let us first review the Monte Carlo-based advantage estimation. As discussed in Section \ref{sec:reduce-gradient-variance}, outputs are sampled and their values computed. The advantage can be obtained by comparing the actual received rewards with the predicted values at time step $t$:
Let us first review the Monte Carlo-based advantage estimation. As discussed in Section \ref{sec:reduce-gradient-variance}, outputs are sampled and their values computed. The advantage can be obtained by comparing the actual received rewards with the predicted values at time step $t$:
\begin{eqnarray}
\begin{eqnarray}
A_{t}& = &\sum_{t=k}^{T}r_{k} - V_{t}
A_{t}& = &\sum_{k=t}^{T}r_{k} - V_{t}
\label{eq:advantage-estimation-monte-carlo}
\label{eq:advantage-estimation-monte-carlo}
\end{eqnarray}
\end{eqnarray}
While the Monte Carlo-based estimation of advantage provides a relatively stable training process, there are two main challenges associated with it:
While the Monte Carlo-based estimation of advantage provides a relatively stable training process, there are two main challenges associated with it:
\begin{itemize}
\begin{itemize}
\item Sampling an entire output for each input is necessary. In some cases where immediate rewards are available, it could be more efficient to update the policy model with partial output. Unfortunately, when using Eq. (\ref{eq:advantage-estimation-monte-carlo}) for advantage estimation, this becomes unfeasible. This is because we must compute the term $\sum_{t=k}^T r_{k}$, which requires the rewards of the entire output.
\item Sampling an entire output for each input is necessary. In some cases where immediate rewards are available, it could be more efficient to update the policy model with partial output. Unfortunately, when using Eq. (\ref{eq:advantage-estimation-monte-carlo}) for advantage estimation, this becomes unfeasible. This is because we must compute the term $\sum_{k=t}^T r_{k}$, which requires the rewards of the entire output.
\item This method still results in high variance. Since a single sample operation might be influenced by random factors, the estimated results may not be stable, as illustrated in Figure \ref{fig:monte-carlo-advantage-issue}.
\item This method still results in high variance. Since a single sample operation might be influenced by random factors, the estimated results may not be stable, as illustrated in Figure \ref{fig:monte-carlo-advantage-issue}.
\end{itemize}
\end{itemize}
...
@@ -467,7 +467,7 @@ While the Monte Carlo-based estimation of advantage provides a relatively stable
...
@@ -467,7 +467,7 @@ While the Monte Carlo-based estimation of advantage provides a relatively stable
\input{section4/Figures/figure-issue-monte-carlo}
\input{section4/Figures/figure-issue-monte-carlo}
\caption{
\caption{
Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq. (\ref{eq:langth-based-reward-function}).
Illustration of the issue of high variance in Monte Carlo-based advantage estimation, using a length-based reward function as described in Eq. (\ref{eq:langth-based-reward-function}).
Typically, if the value model is well-trained, it would predict an average $V_{t}$ of 6 at time step $t$, based on most outputs having around 60 tokens in length. However, for outlier outputs like $\mathbf{y}_{d,t:T}$, which might extend up to 300 tokens, the advantage estimation can exhibit high variance. For instance, the advantage for such an outlier could be computed as $V_{t}(\mathbf{x},\mathbf{y}_{d,<t},y_{d,t})=\sum_{k=t}^T r_{k}- V_t =30-6=24$, a stark contrast to another output, say $\mathbf{y}_{1,t:T}$, which is closer to the average length, where the advantage might be $V_{t}(\mathbf{x},\mathbf{y}_{1,<t},y_{1,t})=-1$. This discrepancy leads to high gradient variance, potentially destabilizing the learning process.
Typically, if the value model is well-trained, it would predict an average $V_{t}$ of 6 at time step $t$, based on most outputs having around 60 tokens in length. However, for outlier outputs like $\mathbf{y}_{d,t:T}$, which might extend up to 300 tokens, the advantage estimation can exhibit high variance. For instance, the advantage for such an outlier could be computed as $A_{t}(\mathbf{x},\mathbf{y}_{d,<t},y_{d,t})=\sum_{k=t}^T r_{k}- V_t =30-6=24$, a stark contrast to another output, say $\mathbf{y}_{1,t:T}$, which is closer to the average length, where the advantage might be $A_{t}(\mathbf{x},\mathbf{y}_{1,<t},y_{1,t})=-1$. This discrepancy leads to high gradient variance, potentially destabilizing the learning process.
}
}
\label{fig:monte-carlo-advantage-issue}
\label{fig:monte-carlo-advantage-issue}
\end{figure*}
\end{figure*}
...
@@ -488,7 +488,7 @@ While temporal difference-based estimation effectively reduces variance, it can
...
@@ -488,7 +488,7 @@ While temporal difference-based estimation effectively reduces variance, it can
GAE introduces parameters that dynamically balance the trade-off between bias and variance. This basic idea is to use the weighted sum of multi-step temporal difference bias as an estimation of the advantage, combining the advantages of Monte Carlo (high variance and low bias) and temporal difference methods (low variance and high bias). This combination can be expressed as
GAE introduces parameters that dynamically balance the trade-off between bias and variance. This basic idea is to use the weighted sum of multi-step temporal difference bias as an estimation of the advantage, combining the advantages of Monte Carlo (high variance and low bias) and temporal difference methods (low variance and high bias). This combination can be expressed as
@@ -588,13 +588,13 @@ Although learning reward models is a standard step in RL, it makes the entire tr
...
@@ -588,13 +588,13 @@ Although learning reward models is a standard step in RL, it makes the entire tr
DPO simplifies the RL process for LLMs by utilizing a straightforward cross-entropy loss, which streamlines the learning process and enhances its stability. Rather than delving into the detailed derivation of the DPO loss function, we will discuss its optimization objective from the perspective of optimizing an LLM. The loss function derived from a Bradley-Terry reward model can be given by
DPO simplifies the RL process for LLMs by utilizing a straightforward cross-entropy loss, which streamlines the learning process and enhances its stability. Rather than delving into the detailed derivation of the DPO loss function, we will discuss its optimization objective from the perspective of optimizing an LLM. The loss function derived from a Bradley-Terry reward model can be given by
This loss function utilizes the implicit reward for DPO training, which replaces the need for an external reward model. The implicit reward is computed as the log-ratio of probabilities:
This loss function utilizes the implicit reward for DPO training, which replaces the need for an external reward model. The implicit reward is computed as the log-ratio of probabilities:
@@ -28,7 +28,7 @@ As illustrated in Figure \ref{fig:bon-sampling}, in the BoN sampling, we first s
...
@@ -28,7 +28,7 @@ As illustrated in Figure \ref{fig:bon-sampling}, in the BoN sampling, we first s
where the $\mathrm{argTopN}$ operation returns the top-$N$ outputs that maximize the function $\Pr(\mathbf{y}|\mathbf{x})$. These outputs can be sampled in various ways, depending on the search algorithm used by the model (e.g., nucleus sampling or beam search). Once the $N$-best output candidates are sampled, the reward model is used to evaluate and select the best one:
where the $\mathrm{argTopN}$ operation returns the top-$N$ outputs that maximize the function $\Pr(\mathbf{y}|\mathbf{x})$. These outputs can be sampled in various ways, depending on the search algorithm used by the model (e.g., nucleus sampling or beam search). Once the $N$-best output candidates are sampled, the reward model is used to evaluate and select the best one:
This process not only identifies the output with the highest reward but also makes it a direct evaluation of the effectiveness of the reward model. For example, the agreement between the reward model and human judgments can be assessed by judging whether $\mathbf{y}_{\mathrm{best}}$ matches the best output selected by humans. Given its speed and cost-efficiency compared to more complex evaluation methods such as those used in PPO, this approach is widely used for evaluating the performance of reward models \citep{rafailov:2023direct,gao-etal:2023scaling}. Nevertheless, in scenarios without a reward model, we can also employ the majority vote approach to determine the final output of the reasoning task. The basic steps involve first identifying the answer that most reasoning paths converge upon and then considering any of the outputs leading to this answer as the final output.
This process not only identifies the output with the highest reward but also makes it a direct evaluation of the effectiveness of the reward model. For example, the agreement between the reward model and human judgments can be assessed by judging whether $\mathbf{y}_{\mathrm{best}}$ matches the best output selected by humans. Given its speed and cost-efficiency compared to more complex evaluation methods such as those used in PPO, this approach is widely used for evaluating the performance of reward models \citep{rafailov:2023direct,gao-etal:2023scaling}. Nevertheless, in scenarios without a reward model, we can also employ the majority vote approach to determine the final output of the reasoning task. The basic steps involve first identifying the answer that most reasoning paths converge upon and then considering any of the outputs leading to this answer as the final output.
...
@@ -46,7 +46,7 @@ We can collect or generate reasoning paths corresponding to problems from existi
...
@@ -46,7 +46,7 @@ We can collect or generate reasoning paths corresponding to problems from existi
Given a set of step-level annotated reasoning paths and corresponding inputs, we can train a reward model to provide a reward for each step in the reasoning process. The reward model can be treated as a classification model. So its architecture can be an LLM with a Softmax layer stacked on top, akin to the architecture depicted in Figure \ref{fig:reward-model}. Here, consider an reasoning path including $n_{s}$ steps, represented as $\mathbf{y}=\{\bar{\mathbf{y}}_{1},\cdots ,\bar{\mathbf{y}}_{n_{s}}\}$. At each step $k$, the process reward model takes both the problem description, denoted by $\mathbf{x}$, and the reasoning steps generated so far, denoted by $\bar{\mathbf{y}}$, as inputs. It then outputs a probability distribution over the set of labels ${\text{\textit{correct}}, \text{\textit{incorrect}}}$, or ${\text{\textit{correct}}, \text{\textit{incorrect}}, \text{\textit{neutral}}}$, to evaluate the reasoning at that point. This model can trained by a casual classification loss, e.g., Sigmoid Cross-entropy. Once trained, the reward model can be used to evaluate reasoning paths by assessing the correctness of each step. A simple method to use log-probabilities of classification to define the reward of each reasoning step, for example, the reward of the $k$-th reasoning step can be given by
Given a set of step-level annotated reasoning paths and corresponding inputs, we can train a reward model to provide a reward for each step in the reasoning process. The reward model can be treated as a classification model. So its architecture can be an LLM with a Softmax layer stacked on top, akin to the architecture depicted in Figure \ref{fig:reward-model}. Here, consider an reasoning path including $n_{s}$ steps, represented as $\mathbf{y}=\{\bar{\mathbf{y}}_{1},\cdots ,\bar{\mathbf{y}}_{n_{s}}\}$. At each step $k$, the process reward model takes both the problem description, denoted by $\mathbf{x}$, and the reasoning steps generated so far, denoted by $\bar{\mathbf{y}}$, as inputs. It then outputs a probability distribution over the set of labels ${\text{\textit{correct}}, \text{\textit{incorrect}}}$, or ${\text{\textit{correct}}, \text{\textit{incorrect}}, \text{\textit{neutral}}}$, to evaluate the reasoning at that point. This model can trained by a casual classification loss, e.g., Sigmoid Cross-entropy. Once trained, the reward model can be used to evaluate reasoning paths by assessing the correctness of each step. A simple method to use log-probabilities of classification to define the reward of each reasoning step, for example, the reward of the $k$-th reasoning step can be given by
where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes the probability of the \textit{correct} label generated by the reward model. The reward score $R_{\phi}(\mathbf{x},\mathbf{y})$ can then be used to select the best step while generating a reasoning path. Additionally, as discussed in Section \ref{sec:generative-reward-models}, there is the option to train a generative process reward model that further improves this performance on evaluating the reasoning step, also called generative verifier in the literature \citep{zhang-etal:2024generative}. Note that in practice, the process reward model serves not only to provide rewards for test-time scaling but also to train the model using RL, e.g., the rewards from this model can be employed as shaping rewards.
where $\Pr_{\phi}(\textit{correct}|\mathbf{x},\bar{\mathbf{y}}_{\le k})$ denotes the probability of the \textit{correct} label generated by the reward model. The reward score $R_{\phi}(\mathbf{x},\mathbf{y})$ can then be used to select the best step while generating a reasoning path. Additionally, as discussed in Section \ref{sec:generative-reward-models}, there is the option to train a generative process reward model that further improves this performance on evaluating the reasoning step, also called generative verifier in the literature \citep{zhang-etal:2024generative}. Note that in practice, the process reward model serves not only to provide rewards for test-time scaling but also to train the model using RL, e.g., the rewards from this model can be employed as shaping rewards.
...
@@ -149,7 +149,7 @@ Compared with conventional offline distillation, OPD adopts a simple but importa
...
@@ -149,7 +149,7 @@ Compared with conventional offline distillation, OPD adopts a simple but importa
This implementation of OPD is relatively simple, as illustrated in Figure~\ref{fig:opd-implementation}. Given an input $\mathbf{x}$, we first sample an output $\mathbf{y}=\{y_1,\ldots,y_n\}$ from the current student model $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$. For each token position $i$, the prefix $(\mathbf{x},\mathbf{y}_{<i})$ denotes a state actually visited by the student model. We then feed the same state into the teacher model $\mathrm{Pr}_{\mathrm{tea}}$ and obtain its probability distribution over the next token. The student model is optimized to match this teacher distribution. This OPD objective can defined as:
This implementation of OPD is relatively simple, as illustrated in Figure~\ref{fig:opd-implementation}. Given an input $\mathbf{x}$, we first sample an output $\mathbf{y}=\{y_1,\ldots,y_n\}$ from the current student model $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$. For each token position $i$, the prefix $(\mathbf{x},\mathbf{y}_{<i})$ denotes a state actually visited by the student model. We then feed the same state into the teacher model $\mathrm{Pr}_{\mathrm{tea}}$ and obtain its probability distribution over the next token. The student model is optimized to match this teacher distribution. This OPD objective can defined as:
