@@ -211,7 +211,12 @@ We can then define a new loss function for training the generative reward model
...
@@ -211,7 +211,12 @@ We can then define a new loss function for training the generative reward model
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
\subsubsection{Reward Model Evaluation}
\subsubsection{Reward Model Evaluation}
How to evaluate a reward model? \\
After training a reward model, \textit{how we to evaluate the reward model?} a common practice for evaluating
the reward is directly assessing the performance of the aligned LLM. While this practice can respond to final metrics, it incurs significant computational costs. Additionally, this approach
1. Using RLHF to evaluate it. \\
1. Using RLHF to evaluate it. \\
2. Using pair ranking to evaluate it (RM-Bench, Reward-Bench, so on.). \\
2. Using pair ranking to evaluate it (RM-Bench, Reward-Bench, so on.). \\
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\
3. When this model is descrminiative model, we can use a probing approach (probing preference representations). \\
@@ -313,7 +313,7 @@ The second approach is \textbf{skill optimization}. One challenge of using memor
...
@@ -313,7 +313,7 @@ The second approach is \textbf{skill optimization}. One challenge of using memor
The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~\ref{eq:tau-define} and~\ref{eq:reward-optimization-for-agent}, where agents improve their behaviors via feedback from experience.
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~(\ref{eq:tau-define}) and~(\ref{eq:reward-optimization-for-agent}), where agents improve their behaviors via feedback from experience.
