@@ -210,6 +210,12 @@ We can then define a new loss function for training the generative reward model
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
\subsubsection{Reward Model Evaluation}
How to evaluate a reward model? \\
1. Using RLHF to evaluate it. \\
2. Using pair ranking to evaluate it (RM-Bench, Reward-Bench, so on.). \\
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\
\subsection{Better Advantage Estimation}
Accurate advantage estimation is critical in RL, particularly when optimizing the policy model to truly reflect the potential benefits of different actions (or tokens in training LLMs). In this subsection, we will delve into methods for improving advantage estimation, providing more accurate advantages in the process of optimizing the policy model.
...
...
@@ -383,3 +389,4 @@ However, there are two sides to every coin. While DPO simplifies the RL process,
\end{table*}
Another limitation associated with DPO is over-optimization. Since DPO employs the Bradley-Terry model for modeling preferences, it can suffer from over-optimization, such as length exploitation, where a longer output might be mistakenly deemed as more aligned with human preferences \citep{singhal-etal:2023long,wang-etal:2023far}. Many efforts have been made to address this issue and propose different variants of DPO, as detailed in Table \ref{tab:dpo_varients}. Note that here we only provide an introduction to their optimization objectives. For more discussions on these variants, interested readers can refer to the related papers.
@@ -29,7 +29,7 @@ However, training VLMs with RL is not a low-hanging fruit in practical applicati
In this process, not all preference data may align with the preferences used in subsequent phases, potentially leading to preference conflicts. We can enhance the visual reward model through data selection techniques, such as LESS \citep{xia-etal:2024less} to address this. In fact, preference transfer is effective across modalities and has also been shown to work across different tasks and languages \citep{cheng-etal:2023everyone,wu-etal:2024reuse}. Interested readers can refer to these papers for more detailed discussions of these topics.
As discussed in Section~\ref{sec:generative-reward-models}, reward reasoning models have demonstrated superior performance in reward prediction. A natural question then arises: \textit{can reward reasoning capabilities also be transferred from text to multimodal settings?} Recent work by \cite{wang2026msrl} provides empirical evidence supporting this hypothesis. By following a similar multi-stage training paradigm, they show that reward reasoning capabilities can indeed be effectively transferred across modalities, further enhancing the performance of visual reward models.
As discussed in Section~\ref{sec:generative-reward-models}, reward reasoning models have demonstrated superior performance in reward prediction. A natural question then arises: \textit{can reward reasoning capabilities also be transferred from text to multimodal settings?} Recent work by \cite{wang2026msrl} provides empirical evidence supporting this hypothesis. By following a similar multi-stage training paradigm, they show that reward reasoning capabilities can indeed be effectively transferred across modalities, further enhancing the performance of visual reward models.
