uupdate.

section4/section4.tex

@@ -210,7 +210,7 @@ We can then define a new loss function for training the generative reward model
 \end{eqnarray}
 \end{eqnarray}
 where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
 where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
 
 
-Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
+Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches. The basic idea is to use only the ranking capability of generative reward models to derive rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
 
 
 
 
 \subsubsection{Rubric-based Reward Models}
 \subsubsection{Rubric-based Reward Models}
@@ -238,7 +238,7 @@ After training a reward model, an important question is \textit{how to evaluate
 \begin{itemize}
 \begin{itemize}
 
 
 \item \textbf{RL-based Evaluation.}
 \item \textbf{RL-based Evaluation.}
-A straightforward approach to evaluate a reward model is to measure its effectiveness in downstream RL training. Specifically, given multiple reward models $\{\mathcal{M}_r^1,\cdots,\mathcal{M}_r^k\}$, we use each reward model to provide reward signals for training a policy model while keeping the training data, RL algorithm, and hyperparameters identical \citep{wang-etal:probing,frick-etal:ppe}. After RL training, we obtain a set of policies $\{\mathcal{P}_1,\cdots,\mathcal{P}_k\}$, which are evaluated on human preference benchmarks or downstream tasks. The performance of each optimized policy is then used as an indirect measure of the corresponding reward model quality. In this way, we consider that a high-quality reward model should provide more reliable reward signals, enabling the policy to achieve better final performance.
+A straightforward approach to evaluate a reward model is to measure its effectiveness in downstream RL training. Specifically, given multiple reward models $\{\mathcal{M}_r^1,\cdots,\mathcal{M}_r^k\}$, we use each reward model to provide reward signals for training a policy model while keeping the training data, RL algorithm, and hyperparameters identical \citep{wang-etal:probing,frick-etal:ppe}. After RL training, we obtain a set of optimized policies $\{\mathcal{P}_1,\cdots,\mathcal{P}_k\}$ and evaluate them on human preference benchmarks or downstream tasks. The performance of each policy then serves as an indirect measure of the quality of its corresponding reward model. In this way, we consider that a high-quality reward model should provide more reliable reward signals, enabling the policy to achieve better final performance.
 
 
 \item \textbf{Pairwise Ranking Evaluation.}
 \item \textbf{Pairwise Ranking Evaluation.}
 Although RL-based evaluation directly measures whether a reward model can improve downstream policy optimization, it suffers from two limitations. First, it introduces significant evaluation costs. Since RL training requires substantial computational resources and time, it is difficult to efficiently compare different reward models. Second, the evaluation results can be sensitive to other factors in the RL pipeline, such as the choice of RL algorithms, which may introduce additional variations beyond the quality of the reward model itself. To address these limitations, a more efficient approach is to directly evaluate the preference ranking ability of reward models. Given a prompt $\mathbf{x}$ and two candidate outputs $\mathbf{y}^{+}$ and $\mathbf{y}^{-}$, where $\mathbf{y}^{+}$ is preferred over $\mathbf{y}^{-}$ according to human preference annotations, the reward model predicts their relative preference. Specifically, for discriminative reward models, we compare the predicted scores of the two outputs. If the reward model assigns a higher score to $\mathbf{y}^{+}$, its prediction is consistent with human preference. For generative reward models, the model directly selects $\mathbf{y}^{+}$ as the preferred output based on its generated preference. Otherwise, the prediction is considered incorrect. By constructing a large number of pairwise ranking samples, we can evaluate the reward model using ranking accuracy:
 Although RL-based evaluation directly measures whether a reward model can improve downstream policy optimization, it suffers from two limitations. First, it introduces significant evaluation costs. Since RL training requires substantial computational resources and time, it is difficult to efficiently compare different reward models. Second, the evaluation results can be sensitive to other factors in the RL pipeline, such as the choice of RL algorithms, which may introduce additional variations beyond the quality of the reward model itself. To address these limitations, a more efficient approach is to directly evaluate the preference ranking ability of reward models. Given a prompt $\mathbf{x}$ and two candidate outputs $\mathbf{y}^{+}$ and $\mathbf{y}^{-}$, where $\mathbf{y}^{+}$ is preferred over $\mathbf{y}^{-}$ according to human preference annotations, the reward model predicts their relative preference. Specifically, for discriminative reward models, we compare the predicted scores of the two outputs. If the reward model assigns a higher score to $\mathbf{y}^{+}$, its prediction is consistent with human preference. For generative reward models, the model directly selects $\mathbf{y}^{+}$ as the preferred output based on its generated preference. Otherwise, the prediction is considered incorrect. By constructing a large number of pairwise ranking samples, we can evaluate the reward model using ranking accuracy:

section5/section5.tex

@@ -129,8 +129,9 @@ One is that large-scale RL requires a highly generalizable reward model. As the
 Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behaviour region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between keeping stability and fostering adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
 Another is that large-scale RL might be constrained by the reference model. To prevent the policy model from deviating too far from its initial state, an SFT LLM is typically employed as the reference model, adding a penalty that restricts updates to ensure the policy model operates within a desirable behaviour region. However, reliance on a static reference model can limit the adaptability and innovation potential of the policy model in large-scale RL. A common strategy to mitigate this issue is to periodically update the reference model to better align with the improving capabilities and knowledge of the policy model, thus striking a balance between keeping stability and fostering adaptiveness during the training process \citep{gorbatovski-etal:2024learn}.
 
 
 
 
+
 \subsection{On-Policy Distillation}
 \subsection{On-Policy Distillation}
-While large-scale RL has shown strong potential for improving model capabilities, it often relies on large amounts of verifiable training data, where the correctness of model outputs can be reliably evaluated. Several approaches have been explored to relax this requirement. A straightforward solution is to construct proxy rewards from the model's own outputs. For example, majority voting can be used to derive pseudo-gold answers for computing accuracy-based rewards, while uncertainty-related signals such as output entropy can also serve as reward signals. However, these approaches largely depend on the model's own capabilities and may provide unreliable supervision. A more promising direction is \textit{on-policy distillation} (OPD), which introduces supervision from a stronger teacher model while preserving on-policy exploration of the student model\footnote{In standard knowledge distillation settings, we typically refer to the ``smaller'' model as the \textit{student model} and the ``larger'' model as the \textit{teacher model}.}. The key idea is to let the student model generate reasoning trajectories on-policy and then use the teacher model to provide token-level distributional supervision at the states actually visited by the student. 
+While large-scale RL has shown strong potential for improving model capabilities, it often relies on large amounts of verifiable training data. Several approaches have been explored to relax this requirement. A straightforward solution is to construct proxy rewards from the model's own outputs. A straightforward solution is to construct proxy rewards from the model's own outputs. For example, we can use majority voting to derive pseudo-gold answers and compute accuracy-based rewards. A more promising direction is \textit{on-policy distillation} (OPD), which introduces supervision from a stronger teacher model while preserving on-policy exploration of the student model\footnote{In standard knowledge distillation settings, we typically refer to the ``smaller'' model as the \textit{student model} and the ``larger'' model as the \textit{teacher model}.}. The key idea is to let the student model generate reasoning trajectories on-policy and then use the teacher model to provide token-level distributional supervision at the states actually visited by the student. 
 
 
 Compared with conventional offline distillation, OPD adopts a simple but important design: the training samples are generated online by the current student policy rather than collected offline from the teacher model. This design mainly addresses the distribution mismatch in offline distillation, where the student is trained on teacher-generated trajectories that may differ from the states it encounters during its own inference. In contrast, OPD provides teacher supervision directly on the states visited by the student, allowing the teacher to correct the student's actual behaviors. For example, consider a mathematical reasoning problem where the student has already generated the partial trajectory ``3x+7=22 $\rightarrow$ 3x=15''. At this student-generated state, the teacher model can directly provide token-level supervision that assigns a higher probability to generating ``x=5'' next, rather than an incorrect continuation such as ``x=4''. In this way, OPD teaches the student how to continue correctly from the states it actually visits, rather than only imitating complete trajectories generated by the teacher model.
 Compared with conventional offline distillation, OPD adopts a simple but important design: the training samples are generated online by the current student policy rather than collected offline from the teacher model. This design mainly addresses the distribution mismatch in offline distillation, where the student is trained on teacher-generated trajectories that may differ from the states it encounters during its own inference. In contrast, OPD provides teacher supervision directly on the states visited by the student, allowing the teacher to correct the student's actual behaviors. For example, consider a mathematical reasoning problem where the student has already generated the partial trajectory ``3x+7=22 $\rightarrow$ 3x=15''. At this student-generated state, the teacher model can directly provide token-level supervision that assigns a higher probability to generating ``x=5'' next, rather than an incorrect continuation such as ``x=4''. In this way, OPD teaches the student how to continue correctly from the states it actually visits, rather than only imitating complete trajectories generated by the teacher model.