@@ -215,8 +215,149 @@ Although incorporating CoT rationales into generative reward models improves pre
...
@@ -215,8 +215,149 @@ Although incorporating CoT rationales into generative reward models improves pre
\subsubsection{Rubric-based Reward Models}
\subsubsection{Rubric-based Reward Models}
While generative reward models have shown strong performance in preference prediction, they typically produce an overall judgment for each response or response pair. For complex open-ended tasks, such holistic evaluation may be insufficient because response quality often depends on multiple instruction-specific aspects. To address this issue, we can employ rubric-based reward modeling. The basic idea is to introduce a set of task-specific criteria that explicitly guide the reward model on \emph{what aspects to evaluate} when predicting preferences. In practice, these criteria can be organized into different rubric formats, as follows:
While generative reward models have shown strong performance in preference prediction, they typically produce an overall judgment for each response or response pair. For complex open-ended tasks, such holistic evaluation may be insufficient because response quality often depends on multiple instruction-specific aspects. To address this issue, we can employ rubric-based reward modeling. The basic idea is to introduce a set of task-specific criteria that explicitly guide the reward model on \emph{what aspects to evaluate} when predicting preferences. In practice, these criteria can be organized into different rubric formats, as follows:
\begin{itemize}
\item\textbf{Holistic Rubric.}
A holistic rubric provides multiple evaluation criteria but asks the reward model to produce a single overall judgment. For example,
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{1em}
\setlength{\rightskip}{1em}
\textbf{Reward Model Input:}
\textit{Input:} Give me three tips to improve my accuracy in solving math problems.
\textit{Output A:} Improving math accuracy requires careful attention to detail and consistent practice. Here are three tips: practice regularly, understand the concepts, and \ldots
\textit{Output B:} Practice more, read carefully, and check your answers.
\ul{Please compare the two outputs based on accuracy, informativeness, clarity, and helpfulness, and give an overall preference.}
\vspace{2mm}
\textbf{Reward Model Output:}
Output A is better overall because it provides more informative and helpful suggestions. \ldots
Therefore, the reward model assigns a higher reward to Output A and selects Output A as the preferred response.
\endgroup
\end{tcolorbox}
\vspace{0.5em}
\item\textbf{Analytic Rubric.}
An analytic rubric evaluates each criterion separately. The criterion-level scores can then be averaged or weighted to obtain the final reward.
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{1em}
\setlength{\rightskip}{1em}
\textbf{Reward Model Input:}
\textit{Input:} Give me three tips to improve my accuracy in solving math problems.
\textit{Output A:} Improving math accuracy requires careful attention to detail and consistent practice. Here are three tips: practice regularly, understand the concepts, and \ldots
\textit{Output B:} Practice more, read carefully, and check your answers.
\ul{Please score each output from 1 to 5 on the following criteria: accuracy, informativeness, clarity, and helpfulness.}
After aggregating the criterion-level scores, Output A receives the higher overall score and is preferred over Output B.
\endgroup
\end{tcolorbox}
\vspace{0.5em}
\item\textbf{Checklist Rubric.}
A checklist rubric decomposes evaluation into atomic and verifiable conditions. Each condition is evaluated independently, typically using binary labels such as \textit{Pass} or \textit{Fail}.
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{1em}
\setlength{\rightskip}{1em}
\textbf{Reward Model Input:}
\textit{Input:} Give me three tips to improve my accuracy in solving math problems.
\textit{Output A:} Improving math accuracy requires careful attention to detail and consistent practice. Here are three tips: practice regularly, understand the concepts, and \ldots
\textit{Output B:} Practice more, read carefully, and check your answers.
\ul{Please check whether each output satisfies the following requirements:}
\ul{1. Does the response provide exactly three tips?}
\ul{2. Are the tips relevant to improving math accuracy?}
\ul{3. Does the response explain the suggested tips?}
\ul{4. Is the response clear and easy to understand?}
\vspace{2mm}
\textbf{Reward Model Output:}
Output A: Pass, Pass, Pass, Pass.
Output B: Pass, Pass, Fail, Pass. \ldots
Since Output A satisfies all checklist items while Output B does not explain the tips, Output A is assigned the higher reward.
\endgroup
\end{tcolorbox}
\vspace{0.5em}
\item\textbf{Hierarchical Rubric.}
A hierarchical rubric organizes evaluation criteria into multiple levels. The reward model first evaluates candidate outputs using lower-level criteria and then aggregates these results into higher-level dimensions to obtain the final reward.
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{1em}
\setlength{\rightskip}{1em}
\textbf{Reward Model Input:}
\textit{Input:} Give me three tips to improve my accuracy in solving math problems.
\textit{Output A:} Improving math accuracy requires careful attention to detail and consistent practice. Here are three tips: practice regularly, understand the concepts, and \ldots
\textit{Output B:} Practice more, read carefully, and check your answers.
\ul{Please evaluate the outputs using the following hierarchical criteria:}
By aggregating the lower-level evaluations into the two higher-level dimensions, Output A obtains the stronger final reward and is selected as the better response.
\endgroup
\end{tcolorbox}
\vspace{0.5em}
\end{itemize}
It is worth noting that reasoning can also be incorporated into rubric-based evaluation. For example, in a checklist rubric, we can ask the reward model to reason about each criterion before producing the corresponding Pass/Fail judgment.
