@@ -210,6 +210,12 @@ We can then define a new loss function for training the generative reward model
...
@@ -210,6 +210,12 @@ We can then define a new loss function for training the generative reward model
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
\subsubsection{Reward Model Evaluation}
How to evaluate a reward model? \\
1. Using RLHF to evaluate it. \\
2. Using pair ranking to evaluate it (RM-Bench, Reward-Bench, so on.). \\
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\
\subsection{Better Advantage Estimation}
\subsection{Better Advantage Estimation}
Accurate advantage estimation is critical in RL, particularly when optimizing the policy model to truly reflect the potential benefits of different actions (or tokens in training LLMs). In this subsection, we will delve into methods for improving advantage estimation, providing more accurate advantages in the process of optimizing the policy model.
Accurate advantage estimation is critical in RL, particularly when optimizing the policy model to truly reflect the potential benefits of different actions (or tokens in training LLMs). In this subsection, we will delve into methods for improving advantage estimation, providing more accurate advantages in the process of optimizing the policy model.
...
@@ -383,3 +389,4 @@ However, there are two sides to every coin. While DPO simplifies the RL process,
...
@@ -383,3 +389,4 @@ However, there are two sides to every coin. While DPO simplifies the RL process,
\end{table*}
\end{table*}
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table \ref{tab:dpo_varients}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table \ref{tab:dpo_varients}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
