title={Gram-r$^2$: Self-training generative foundation reward models for reward reasoning},
title={Gram-r$^2$: Self-training generative foundation reward models for reward reasoning},
author={Wang, Chenglong and Mu, Yongyu and Zhou, Hang and Huo, Yifu and Zhu, Ziming and Zeng, Jiali and Yang, Murun and Li, Bei and Hao, Xiaoyang and Zhang, Chunliang and others},
author={Wang, Chenglong and Mu, Yongyu and Zhou, Hang and Huo, Yifu and Zhu, Ziming and Zeng, Jiali and Yang, Murun and Li, Bei and Hao, Xiaoyang and Zhang, Chunliang and others},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
To further improve the generative reward model, we can label the relevant explanation to enable the generative reward model to produce a CoT rationale \citep{zhang-etal:2024generative,wang-etal:wang2026gram}. In this case, we use a prompt $\mathbf{c}$ with a CoT rationale generation instruction, as shown below.
To further improve the generative reward model, we can label the relevant explanation to enable the generative reward model to produce a CoT rationale \citep{zhang-etal:2024generative,wang-etal:wang2026gramrr}. In this case, we use a prompt $\mathbf{c}$ with a CoT rationale generation instruction, as shown below.
\vspace{0.5em}
\vspace{0.5em}
\begin{tcolorbox}[frame empty]
\begin{tcolorbox}[frame empty]
...
@@ -298,7 +298,7 @@ Moreover, there are other methods like GRPO that design advantage estimation wit
...
@@ -298,7 +298,7 @@ Moreover, there are other methods like GRPO that design advantage estimation wit
Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in the context of LLMs due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.
Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in the context of LLMs due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.
In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.
In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.
\section{Reinforcement Learning for LLM-based Agents}
Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
Using RL to train agents is not a new concept, but its focus and application have evolved significantly over time. In classical RL, an agent is typically a domain-specific decision model trained entirely from scratch. For example, in tasks ranging from playing complex games (like GO) \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} to controlling robotic locomotion \citep{lee-etal:lee2024learning}, the primary objective is to comprehensively explore a highly structured environment and find an optimal policy from a randomly initialized state.
With the rise of LLMs as core components of autonomous systems, RL has been increasingly used to adapt these pretrained models for sequential decision-making in dynamic and open-ended environments. In this setting, agentic RL addresses a key limitation of LLMs: although they encode extensive world knowledge, their outputs are not inherently aligned with long-horizon task requirements. For example, a supervised fine-tuned LLM may correctly generate a script for calling an external API, but it cannot reliably self-correct when the environment returns unexpected errors. This creates new challenges for enabling LLM-based agents to plan autonomously, adapt to environmental feedback, and safely improve their strategies through iterative interaction.
As LLMs become increasingly central to autonomous systems, the application of RL is more broadly used to empower these pre-trained models to make sequential decisions in dynamic, open-ended environments. In this context, the problem that RL addresses is that while LLMs possess vast world knowledge, their outputs may not naturally align with the demands of long-horizon tasks. For example, a supervised fine-tuned LLM might successfully generate a script to call an external API, but it cannot intrinsically self-correct if the environment returns an unpredictable error. This poses new challenges in ensuring that LLM-based agents can autonomously plan, adapt to environmental feedback, and safely refine their strategies through continuous trial and error.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory updates, skill optimization, and trajectory refinement without altering the internal model weights. It is worth noting that while these approaches depart from traditional parametric updates, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL enhances core agentic capabilities, such as planning and tool use. Then we consider the integration of RL within advanced agent systems, such as OpenClaw, to highlight the cutting edge of LLM agent research. It is worth noting that while some of these advanced techniques do not update the model's parameters, i.e., leveraging RL purely for dynamic memory management, which departs from traditional parametric RL, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making.
\subsection{Building Agent Capabilities}
\subsection{An Overview of Agentic Reinforcement Learning}
\subsubsection{Planning}
\subsubsection{Planning}
\subsubsection{Tool Use}
\subsubsection{Tool Use}
\subsubsection{Memory}
\subsection{Training-free Methods}
% 当然还有自我反思和自我反馈，这些之前的章节已经有了
\subsubsection{Memory Update}
\subsubsection{Skill Optimization}
\subsection{Reinforcement Learning for Advanced Agent Systems}
\section{Reinforcement Learning for Multimodal Models}
\section{Multimodal Reinforcement Learning}
Multimodal learning involves models that process and relate information from multiple data modalities (e.g. vision, text, speech), enabling more comprehensive understanding than single-modality systems. By combining modalities, models can make more robust predictions and capture complementary information that one modality alone might miss. Such multimodal approaches have benefits across various applications, such as image caption and image generation. Given these advantages, multimodal learning has become increasingly significant as AI systems aim to perceive and reason more like humans, who naturally integrate sight, sound, and language \citep{baltruvsaitis-etal:2018multimodal}.
Multimodal learning involves models that process and relate information from multiple data modalities (e.g. vision, text, speech), enabling more comprehensive understanding than single-modality systems. By combining modalities, models can make more robust predictions and capture complementary information that one modality alone might miss. Such multimodal approaches have benefits across various applications, such as image caption and image generation. Given these advantages, multimodal learning has become increasingly significant as AI systems aim to perceive and reason more like humans, who naturally integrate sight, sound, and language \citep{baltruvsaitis-etal:2018multimodal}.
In the era of LLM, we can extend their capabilities into the multimodal domain by training them with diverse data types. For example, we can develop a visual language model by training an LLM with image-text pairs using SFT. Here, we return to the issue of aligning models with human preferences. Ideally, once aligned with human preferences through RL, the LLM would also generalize the target modality well by the SFT. However, the reality does not always meet expectations. Although an LLM may align well with human preferences in one aspect, it often struggles to generalize this alignment across a different modality. Thus, to align multimodal models with human preferences, we perform RL to enhance their performance further within the specific modality.
In the era of LLM, we can extend their capabilities into the multimodal domain by training them with diverse data types. For example, we can develop a visual language model by training an LLM with image-text pairs using SFT. Here, we return to the issue of aligning models with human preferences. Ideally, once aligned with human preferences through RL, the LLM would also generalize the target modality well by the SFT. However, the reality does not always meet expectations. Although an LLM may align well with human preferences in one aspect, it often struggles to generalize this alignment across a different modality. Thus, to align multimodal models with human preferences, we perform RL to enhance their performance further within the specific modality.
