update.

section4/section4.tex

@@ -239,17 +239,26 @@ R_\theta(\mathbf{x}_i,\mathbf{y}_i^{-})
 where $N$ denotes the number of evaluation samples. Examples of this evaluation approach include RewardBench \citep{lambert-etal:Rewardbench,malik-etal:rewardbench}, RM-Bench \citep{liu-etal:rm-bench}, RMB \citep{zhou-etal:rmb}, and JudgeBench \citep{tan-etal:judgebench}.
 
 \item \textbf{Listwise Ranking Evaluation.}
-In practical alignment scenarios, the reward model often needs to select the best output from multiple candidates, such as in best-of-$n$ sampling or reranking. Therefore, listwise ranking evaluation measures whether a reward model can produce a consistent ranking over a set of candidate outputs. Specifically, given a prompt $\mathbf{x}$ and a candidate output set:
+In practical alignment scenarios, the reward model often needs to select the best output from multiple candidates, such as in best-of-$n$ sampling or reranking. Therefore, listwise ranking evaluation measures whether a reward model can produce a consistent ranking over a set of candidate outputs. Specifically, given a prompt $\mathbf{x}$ and a candidate output set: $\mathcal{Y}=\{\mathbf{y}_1,\mathbf{y}_2,\cdots,\mathbf{y}_n\}$, the reward model assigns scores to all candidates. Based on the computed rewards, the reward model selects the highest-scoring output:
 \begin{eqnarray}
-\mathcal{Y}=\{\mathbf{y}_1,\mathbf{y}_2,\cdots,\mathbf{y}_n\}
+\mathbf{y}_\mathrm{best}=\arg\max_{\mathbf{y}_i\in\mathcal{Y}}R_\theta(\mathbf{x},\mathbf{y}_i)
 \end{eqnarray}
 
-the reward model assigns scores to all candidates. Based on the predicted rewards, the reward model selects the highest-scoring output:
+The selected output is then compared with the human-preferred output to evaluate how well the reward model captures human preferences. We take PPE \citep{frick-etal:ppe} as an example to illustrate this evaluation approach. Specifically, we can construct each evaluation sample with multiple candidate outputs, where the quality of each output can be automatically verified. For example, in mathematical reasoning tasks, we can use an answer verifier to determine whether each generated solution is correct. Given a set of candidates $\mathcal{Y}$, the verification result of each output can be represented as:
 \begin{eqnarray}
-\hat{\mathbf{y}}=\arg\max_{\mathbf{y}_i\in\mathcal{Y}}R_\theta(\mathbf{x},\mathbf{y}_i)
+v_i=\mathrm{Verify}(\mathbf{x},\mathbf{y}_i),\quad v_i\in\{0,1\}
+\end{eqnarray}
+where $v_i=1$ indicates that the output is correct. If the output selected by the reward model is verified as correct: $\mathrm{Verify}(\mathbf{x},\mathbf{y}_{\mathrm{best}})=1 $, the reward model is considered successful on this instance. The final performance is measured by the selection accuracy over all evaluation instances:
+\begin{eqnarray}
+\mathrm{Acc}
+=
+\frac{1}{N}
+\sum_{j=1}^{N}
+\mathbb{I}
+(\mathrm{Verify}(\mathbf{x}_j,\mathbf{y}_{\mathrm{best}}^j)=1),
 \end{eqnarray}
 
-The predicted ranking is then compared with human preference rankings to evaluate the consistency between the reward model and human judgments. Examples of this evaluation approach include PPE \citep{frick-etal:ppe}. 
+where $N$ denotes the number of evaluation instances.
 
 \end{itemize}