\newblock Reinforcement learning for self-improving agent with skill library.
\newblock Reinforcement learning for self-improving agent with skill library.
\newblock In \emph{Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 1529--1550, 2026{\natexlab{d}}.
\newblock In \emph{Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 1529--1550, 2026{\natexlab{e}}.
author={Frick, Evan and Li, Tianle and Chen, Connor and Chiang, Wei-Lin and Angelopoulos, Anastasios and Jiao, Jiantao and Zhu, Banghua and Gonzalez, Joseph E and Stoica, Ion},
booktitle={International Conference on Learning Representations},
volume={2025},
pages={18128--18163},
year={2025}
}
@inproceedings{tan-etal:judgebench,
title={Judgebench: A benchmark for evaluating llm-based judges},
author={Tan, Sijun and Zhuang, Siyuan and Montgomery, Kyle and Tang, William and Cuadron, Alejandro and Wang, Chenguang and Popa, Raluca and Stoica, Ion},
booktitle={International Conference on Learning Representations},
volume={2025},
pages={63277--63303},
year={2025}
}
@inproceedings{zhou-etal:rmb,
title={Rmb: Comprehensively benchmarking reward models in llm alignment},
author={Zhou, Enyu and Zheng, Guodong and Wang, Binghai and Xi, Zhiheng and Dou, Shihan and Bao, Rong and Shen, Wei and Xiong, Limao and Fan, Jessica and Mou, Yurong and others},
booktitle={International Conference on Learning Representations},
volume={2025},
pages={26543--26589},
year={2025}
}
@inproceedings{liu-etal:rm-bench,
title={Rm-bench: Benchmarking reward models of language models with subtlety and style},
author={Liu, Yantao and Yao, Zijun and Min, Rui and Cao, Yixin and Hou, Lei and Li, Juanzi},
booktitle={International Conference on Learning Representations},
volume={2025},
pages={44323--44355},
year={2025}
}
@inproceedings{malik-etal:rewardbench,
title={Rewardbench 2: Advancing reward model evaluation},
author={Malik, Saumya and Pyatkin, Valentina and Land, Sander and Morrison, Jacob and Smith, Noah and Hajishirzi, Hanna and Lambert, Nathan},
booktitle={International Conference on Learning Representations},
volume={2026},
pages={144839--144866},
year={2026}
}
@inproceedings{lambert-etal:Rewardbench,
title={Rewardbench: Evaluating reward models for language modeling},
author={Lambert, Nathan and Pyatkin, Valentina and Morrison, Jacob and Miranda, LJ and Lin, Bill Yuchen and Chandu, Khyathi and Dziri, Nouha and Kumar, Sachin and Zick, Tom and Choi, Yejin and others},
booktitle={Findings of the Association for Computational Linguistics: NAACL 2025},
pages={1755--1797},
year={2025}
}
@misc{wang-etal:rrc,
title={RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction},
author={Chenglong Wang and Ziming Zhu and Yifu Huo and Bei Li and Qiaozhi He and Yan Ding and Xiaoyang Hao and Yuxin Gao and Tianhua Zhou and Xiaojia Chang and Tongran Liu and Jingbo Zhu},
year={2026},
eprint={2608.06310},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2608.06310},
}
@article{monea-etal:llms,
@article{monea-etal:llms,
title={Llms are in-context reinforcement learners},
title={Llms are in-context reinforcement learners},
author={Monea, Giovanni and Bosselut, Antoine and Brantley, Kiant{\'e} and Artzi, Yoav},
author={Monea, Giovanni and Bosselut, Antoine and Brantley, Kiant{\'e} and Artzi, Yoav},
@@ -181,11 +181,11 @@ Although we discuss methods for training a generative reward model here, an inte
...
@@ -181,11 +181,11 @@ Although we discuss methods for training a generative reward model here, an inte
Additionally, to mitigate the positional bias problem \citep{wang-etal:2023large}, we can introduce an alternative input order by transposing the positions of output, i.e., presenting $\mathbf{y}_{\mathrm{ref}}$ before $\mathbf{y}'$, to construct a secondary input string $\mathbf{s}'_{T}=[\mathbf{c}, \mathbf{x}', \mathbf{y}_{\mathrm{ref}}, \mathbf{y}']$.
Additionally, to mitigate the positional bias problem \citep{wang-etal:2023large}, we can introduce an alternative input order by transposing the positions of output, i.e., presenting $\mathbf{y}_{\mathrm{ref}}$ before $\mathbf{y}'$, to construct a secondary input string $\mathbf{s}'_{T}=[\mathbf{c}, \mathbf{x}', \mathbf{y}_{\mathrm{ref}}, \mathbf{y}']$.
The reward for $(\mathbf{x}', \mathbf{y}')$ is thus defined as the log-probability that $\mathbf{y}'$ is preferred over $\mathbf{y}_{\mathrm{ref}}$:
The reward for $(\mathbf{x}', \mathbf{y}')$ is thus defined as the log-probability that $\mathbf{y}'$ is preferred over $\mathbf{y}_{\mathrm{ref}}$:
To further improve the generative reward model, we can label the relevant explanation to enable the generative reward model to produce a CoT rationale \citep{zhang-etal:2024generative,wang-etal:wang2026gramrr}. In this case, we use a prompt $\mathbf{c}$ with a CoT rationale generation instruction, as shown below.
To further improve the generative reward model, we can label the relevant explanation to enable the generative reward model to produce a CoT rationale \citep{zhang-etal:2024generative,wang-etal:wang2026gramrr}. In this case, we use a prompt $\mathbf{c}$ with a CoT rationale generation instruction, as shown below.
...
@@ -210,15 +210,52 @@ We can then define a new loss function for training the generative reward model
...
@@ -210,15 +210,52 @@ We can then define a new loss function for training the generative reward model
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
\subsubsection{Reward Model Evaluation}
\subsubsection{Reward Model Evaluation}
After training a reward model, \textit{how we to evaluate the reward model?} a common practice for evaluating
After training a reward model, an important question is \textit{how to evaluate whether the learned reward function can accurately capture human preferences}. Unlike conventional supervised models, reward models do not directly predict explicit labels, but instead learn to assign scores that reflect the relative quality of different responses. As a result, existing evaluation methods mainly focus on measuring the preference modeling ability of reward models or their effectiveness in RL training. We summarize three commonly used evaluation methods as follows.
the reward is directly assessing the performance of the aligned LLM. While this practice can respond to final metrics, it incurs significant computational costs. Additionally, this approach
\begin{itemize}
\item\textbf{RL-based Evaluation.}
A straightforward approach to evaluate a reward model is to measure its effectiveness in downstream RL optimization. Specifically, given multiple reward models $\{\mathcal{M}_r^1,\cdots,\mathcal{M}_r^k\}$, we use each reward model to provide reward signals for training a policy model while keeping the training data, RL algorithm, and hyperparameters identical. After optimization, we obtain a set of policies $\{\mathcal{P}_1,\cdots,\mathcal{P}_k\}$, which are evaluated on human preference benchmarks or downstream tasks. The performance of each optimized policy is then used as an indirect measure of the corresponding reward model quality. In this way, we consider that a high-quality reward model should provide more reliable reward signals, enabling the policy to achieve better final performance.
\item\textbf{Pairwise Ranking Evaluation.}
Although RL-based evaluation directly measures whether a reward model can improve downstream policy optimization, it suffers from limitations. First, it introduces significant evaluation costs, as RL training requires substantial computational resources and time, making it difficult to efficiently compare different reward models. Second, the evaluation results can be sensitive to other factors in the RL pipeline, such as the choice of RL algorithms, which may introduce additional variations beyond the quality of the reward model itself. To address these limitations, a more efficient approach is to directly evaluate the preference ranking ability of reward models. Given a prompt $\mathbf{x}$ and two candidate responses $\mathbf{y}^{+}$ and $\mathbf{y}^{-}$, where $\mathbf{y}^{+}$ is preferred over $\mathbf{y}^{-}$ according to human preference annotations, the reward model predicts their relative preference. Specifically, for discriminative reward models, we compare the predicted scores of the two responses. If the reward model assigns a higher score to $\mathbf{y}^{+}$, its prediction is consistent with human preference. For generative reward models, the model directly selects $\mathbf{y}^{+}$ as the preferred response based on its generated preference. Otherwise, the prediction is considered incorrect. By constructing a large number of pairwise ranking samples, we can evaluate the reward model using ranking accuracy:
\begin{equation}
\mathrm{Acc}
=
\frac{1}{N}
\sum_{i=1}^{N}
\mathbb{I}
\left[
R_\theta(\mathbf{x}_i,\mathbf{y}_i^{+})
>
R_\theta(\mathbf{x}_i,\mathbf{y}_i^{-})
\right]
\end{equation}
where $N$ denotes the number of evaluation samples. Examples of this evaluation approach include RewardBench \citep{lambert-etal:Rewardbench,malik-etal:rewardbench}, RM-Bench \citep{liu-etal:rm-bench}, RMB \citep{zhou-etal:rmb}, and JudgeBench \citep{tan-etal:judgebench}.
\item\textbf{Listwise Ranking Evaluation.}
In practical alignment scenarios, the reward model often needs to select the best output from multiple candidates, such as in best-of-$n$ sampling or reranking. Therefore, listwise ranking evaluation measures whether a reward model can produce a consistent ranking over a set of candidate outputs. Specifically, given a prompt $\mathbf{x}$ and a candidate output set:
The predicted ranking is then compared with human preference rankings to evaluate the consistency between the reward model and human judgments. Examples of this evaluation approach include PPE \citep{frick-etal:ppe}.
\end{itemize}
1. Using RLHF to evaluate it. \\
2. Using pair ranking to evaluate it (RM-Bench, Reward-Bench, so on.). \\
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\
