Skip to content

R
rl-introduction
Commit 3d2d4d48 by wangchenglong
Options

update.

parent 6d9af4db
正在显示 包含 255 行增加 和 28 行删除
rl-introduction.bbl
This source diff could not be displayed because it is too large. You can view the blob instead.
rl-introduction.sty
......@@ -99,7 +99,7 @@
% Title (includes both anonymized and non-anonymized versions)
\def\@maketitle{\vbox{\hsize\textwidth
%\linewidth\hsize \vskip 0.1in \toptitlebar \centering
{\Large\bf \@title\par\vskip 0.3in}
{\LARGE\bf \@title\par\vskip 0.3in}
%\bottomtitlebar % \vskip 0.1in % minus
\ifcolmfinal
% \lhead{Published as a conference paper at COLM 2025}
......
section2/Figures/llm-and-rl-elements.tex 0 → 100644
% 导言区
% \usepackage{amsmath}
% \usepackage{tikz}
% \usetikzlibrary{arrows.meta,decorations.pathreplacing,calc,fit}
\begin{tikzpicture}[
x=1cm,
y=1cm,
font=\normalsize,
>=Stealth,
line/.style={
draw=black,
line width=0.9pt
},
arrow/.style={
line,
-{Stealth[length=2.5mm,width=1.8mm]}
},
bluearrow/.style={
draw=blue!42,
line width=0.65pt,
-{Stealth[length=2.8mm,width=2mm]}
},
box/.style={
draw=black,
line width=0.65pt,
align=center,
inner sep=7pt,
outer sep=0pt
}
]
% ============================================================
% Colors
% ============================================================
\definecolor{stateorange}{RGB}{252,241,213}
\definecolor{actiongreen}{RGB}{232,243,225}
\definecolor{rewardpink}{RGB}{250,228,228}
\definecolor{valuepurple}{RGB}{240,238,250}
\definecolor{policyblue}{RGB}{232,238,255}
% ============================================================
% Policy LLM
% ============================================================
\node[
box,
fill=white,
minimum width=10.8cm,
minimum height=1.15cm
] (llm) at (0,0)
{\large Agent (an LLM)};
% ============================================================
% Input / generated tokens
% ============================================================
\node (x1) at (-4.55,-1.35) {$x_1$};
\node at (-3.25,-1.35) {$\cdots$};
\node (xm) at (-1.90,-1.35) {$x_m$};
\node (y1) at (-0.25,-1.35) {$y_1$};
\node at ( 1.20,-1.35) {$\cdots$};
\node (ytm) at ( 2.90,-1.35) {$y_{t-1}$};
% Anchor arrow endpoints to the LLM bottom edge
\foreach \n in {x1,xm,y1,ytm}{
\draw[arrow]
(\n.north) -- (\n.north |- llm.south);
}
% ============================================================
% State brace and box
% ============================================================
\draw[
decorate,
decoration={brace,mirror,amplitude=7pt},
line width=0.9pt
]
(-4.90,-1.73) -- (3.45,-1.73);
\node[
box,
fill=stateorange,
minimum width=4.30cm,
minimum height=0.80cm
] (state) at (-0.725,-2.65)
{State $s_t=(\mathbf{x},\mathbf{y}_{<t})$};
% ============================================================
% Token probability distributions
% ============================================================
\node[
box,
fill=white,
minimum width=2.45cm,
minimum height=0.78cm
] (prob1) at (-0.35,1.55)
{$\mathrm{Pr}_{\theta}(\cdot\mid\mathbf{x},y_1)$};
\node (dots) at (1.30,1.55) {$\cdots$};
\node[
box,
fill=white,
minimum width=2.60cm,
minimum height=0.78cm
] (probt) at (3.05,1.55)
{$\mathrm{Pr}_{\theta}(\cdot\mid\mathbf{x},\mathbf{y}_{<t})$};
% LLM -> probability distributions
\draw[line]
(prob1.south |- llm.north) -- (prob1.south);
\draw[line]
(probt.south |- llm.north) -- (probt.south);
% Dashed group sized automatically around its contents
\node[
draw=black,
dashed,
line width=0.6pt,
fit=(prob1)(dots)(probt),
inner xsep=8pt,
inner ysep=7pt
] (probgroup) {};
% ============================================================
% Policy notation: no frame
% ============================================================
\node[
box,
fill=policyblue,
align=center,
minimum width=2.75cm,
minimum height=0.70cm
] (policyPi) at (-4.15,1.55)
{Policy $\pi_{\theta}(\cdot\mid s_t)$};
\draw[arrow]
(probgroup.west) -- (policyPi.east);
% ============================================================
% Action and vertical sampling arrow
% ============================================================
\node[
box,
fill=actiongreen,
minimum width=2.85cm,
minimum height=0.78cm
] (action) at (3.05,3.40)
{Action $a_t=y_t$};
\draw[arrow]
(probt.north)
-- node[right=5pt,font=\normalsize] {Sampling}
(action.south);
% ============================================================
% Reward model
% ============================================================
\node[
box,
fill=rewardpink,
minimum width=3.55cm,
minimum height=1.40cm
] (reward) at (8.60,-0.70)
{
Reward Model\\[4pt]
$R(s_t,a_t)$
};
% Action -> Reward: route outside the LLM
\coordinate (actionTurn) at (5.90,3.40);
\draw[arrow]
(action.east)
-- (actionTurn)
|- ([yshift=6pt]reward.west);
% State/context -> Reward: horizontal arrow
\draw[arrow]
(3.45,-1.30)
-- (reward.west |- {0,-1.30});
% ============================================================
% Value functions
% ============================================================
\node[
box,
fill=valuepurple,
minimum width=3.55cm,
minimum height=1.40cm
] (value) at (8.60,2.55)
{
Value Functions\\[4pt]
$V(s_t),\,Q(s_t,a_t)$
};
\draw[arrow]
(reward.north) -- (value.south);
% ============================================================
% Policy optimization feedback: Value -> Policy
% ============================================================
\coordinate (feedbackTop) at (0,4.35);
\draw[bluearrow]
(value.north)
-- (value.north |- feedbackTop)
-- (policyPi.north |- feedbackTop)
-- ([yshift=5pt]policyPi.north);
\node[
font=\normalsize,
anchor=south
] at (2.225,4.48)
{Policy Optimization};
\end{tikzpicture}
section2/section2.tex
......@@ -10,7 +10,7 @@ Understanding these basic mechanisms is important before introducing RL for LLMs
In this paper, we mainly focus on generative LLMs based on decoder-only Transformers \citep{vaswani-etal:2017attention}. Let $\mathcal{V}$ denote the vocabulary of tokens. A token is the basic unit processed by an LLM. It can be a word, a subword, a punctuation mark, or another text fragment produced by the tokenizer. Before a raw text is input into an LLM, it is first converted into a sequence of tokens from $\mathcal{V}$. Therefore, throughout this paper, all inputs and outputs of an LLM are represented as token sequences. Given a token sequence $\mathbf{z}=z_1...z_N$, a language model parameterized by $\theta$ estimates the probability of the sequence by factorizing it from left to right:
\begin{eqnarray}
\mathrm{Pr}_{\theta}(\mathbf{z}) & = & \prod_{i=1}^{N} \mathrm{Pr}_{\theta}(z_i|\mathbf{z}_{<i}) \nonumber \\
\mathrm{Pr}_{\theta}(\mathbf{z}) & = & \prod_{i=1}^{N} \mathrm{Pr}_{\theta}(z_i|\mathbf{z}_{<i}) \\
\log \mathrm{Pr}_{\theta}(\mathbf{z}) & = & \sum_{i=1}^{N} \log \mathrm{Pr}_{\theta}(z_i|\mathbf{z}_{<i})
\end{eqnarray}
......@@ -111,10 +111,10 @@ Another important concept related to prompting is in-context learning. When prom
Prompting is crucial because it directly influences how the LLM understands the task and generates the output. A well-designed prompt can guide the model to produce more accurate, relevant, and appropriately formatted responses. However, since the model parameters are unchanged, the result can also be sensitive to the wording, structure, and demonstrations provided in the prompt.
Following the notation used above, we denote the entire prompt by $\mathbf{x}=x_1...x_m$. Prompting asks the LLM to generate an output $\mathbf{y}=y_1...y_T$ according to the conditional distribution $\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$. The probability of the output is factorized as
\begin{eqnarray}
\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) & = & \prod_{t=1}^{T} \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t}) \nonumber \\
\log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) & = & \sum_{t=1}^{T} \log \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})
\end{eqnarray}
\begin{align}
\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) & = \prod_{t=1}^{T} \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t}) \\
\log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) & = \sum_{t=1}^{T} \log \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})
\end{align}
where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the tokens that have already been generated. This conditional generation formulation will be repeatedly used in the following sections. From this perspective, a prompt mainly changes the condition under which the same language model distribution is used.
......@@ -213,16 +213,27 @@ Regardless of the decoding strategy, LLM generation can be viewed as a sequence
RL, supported by a well-established theoretical framework, has been widely applied across a broad range of domains. In particular, with the rapid advancement of LLMs, RL has become a standard approach in the post-training stage. Before using LLM training examples to explain RL algorithms, we first provide a brief overview of deep reinforcement learning in this subsection. We then introduce the general formulation of RL, along with key terminologies and notations that are essential for understanding RL.
\subsubsection{Markov Decision Process}
The general framework of RL is illustrated in Figure~\ref{fig:rl_framework}. An RL system primarily consists of two components: an agent and an environment. At each timestep $t$, the agent observes a state $s_t$ from the environment and selects an action $a_t$ according to a policy $\pi$, which is typically parameterized by a neural network. After executing the action, the environment transitions to a new state $s_{t+1}$ and returns a reward $r_t$ corresponding to the taken action.
\begin{figure}[!t]
The RL formulation for LLMs is illustrated in Figure~\ref{fig:llm-rl-policy-optimization}. An RL system primarily consists of two components: an agent and an environment. At each timestep $t$, the agent observes a state $s_t$ from the environment and selects an action $a_t$ according to a policy $\pi$, which is typically parameterized by a neural network. After executing the action, the environment transitions to a new state $s_{t+1}$ and returns a reward $r_t$ corresponding to the taken action.
% \begin{figure}[!t]
% \centering
% \input{section2/Figures/rl_framework}
% \caption{
% A general framework of deep reinforcement learning. The policy is parameterized by neural networks, and the agent learns an optimal policy through interactions with the environment to maximize cumulative rewards.
% }
% \label{fig:rl_framework}
% \end{figure}
\begin{figure*}[!t]
\centering
\input{section2/Figures/rl_framework}
\caption{
A general framework of deep reinforcement learning. The policy is parameterized by neural networks, and the agent learns an optimal policy through interactions with the environment to maximize cumulative rewards.
\resizebox{\linewidth}{!}{
\input{section2/Figures/llm-and-rl-elements.tex}
}
\label{fig:rl_framework}
\end{figure}
\caption{A schematic illustration of RL for LLMs.
% The current context defines the state $s_t=(\mathbf{x},\mathbf{y}_{<t})$, the policy corresponds to the next-token distributions $\Pr_{\theta}(\cdot \mid \mathbf{x},\mathbf{y}_{<t})$, and sampling from the policy yields the action $a_t=y_t$. The reward model provides feedback for the sampled action, while value functions estimate long-term returns for policy optimization.
}
\label{fig:llm-rl-policy-optimization}
\end{figure*}
This interaction process is commonly modeled as a Markov Decision Process (MDP), where the agent interacts with the environment over discrete timesteps \citep{Sutton-and-Barto:2018RL}. A sequence of states and actions forms a trajectory, denoted as $\tau = (s_0, a_0, s_1, a_1, \cdots, s_{H-1}, a_{H-1})$, where $H$ represents the trajectory length. Each trajectory accumulates rewards from the environment.
......
section2/tables/rl-elements-and-llm-counterparts.tex
......@@ -18,13 +18,13 @@
\textbf{Element} & \textbf{Interpretation} \\
\midrule
\textbf{Agent} &
The learner or decision-maker in reinforcement learning. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals. \\
The learner or decision-maker in RL. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals. \\
\midrule
\textbf{Environment} &
Everything external to the agent with which it interacts. Unlike traditional RL settings that involve physical or simulated environments, the environment in LLM-based RL is typically abstract, consisting of the training framework that provides feedback (e.g., reward models, human annotations, or evaluation metrics) for generated outputs. \\
\midrule
\textbf{State ($S$)} &
A state represents the current situation of the environment. For language modeling, the state at timestep $t$ can be defined as the sequence of observed tokens up to that point, i.e., the context used to predict the next token. Formally, the state can be represented as $S=(x,y_{<t})$, where $x$ denotes the input prompt and $y_{<t}$ denotes the previously generated tokens. \\
\textbf{State ($s$)} &
A state represents the current situation of the environment. For language modeling, the state at timestep $t$ can be defined as the sequence of observed tokens up to that point, i.e., the context used to predict the next token. Formally, the state can be represented as $s_t=(\mathbf{x},\mathbf{y}_{<t})$, where $\mathbf{x}$ denotes the input prompt and $\mathbf{y}_{<t}$ denotes the previously generated tokens. \\
\midrule
\textbf{Action ($a$)} &
An action corresponds to a decision made by the agent. In LLMs, actions are naturally defined as selecting the next token from the vocabulary, i.e., $a=y_t$. \\
......@@ -34,16 +34,16 @@ The reward provides feedback from the environment to evaluate the quality of an
\midrule
\textbf{Policy ($\pi$)} &
The policy defines the agent's behavior, i.e., the probability of taking an action given a state. For LLMs, the policy corresponds to the conditional probability distribution over the next token given the context:
\tableeq{\pi(a \mid s)=\Pr(y_t \mid x,y_{<t})}
\tableeq{\pi_{\theta}(a_t \mid s_t)=\mathrm{Pr}_{\theta}(y_t \mid \mathbf{x},\mathbf{y}_{<t})}
\vspace{-1mm}
where $a=y_t$ and $s=(x,y_{<t})$. Under this formulation, an LLM can be naturally interpreted as a parameterized policy. \\
where $a_t=y_t$ and $s_t=(\mathbf{x},\mathbf{y}_{<t})$. Under this formulation, an LLM can be naturally interpreted as a parameterized policy. \\
\midrule
\textbf{\makecell[l]{Value Function\\($V$ and $Q$)}} &
The value function estimates the expected cumulative reward when following a policy. The state-value function $V(s)$ measures the expected discounted return starting from state $s$:
\tableeq{V(s)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t r_t \middle| s_0=s,\pi\right]}
where $\gamma\in[0,1]$ is the discount factor. The action-value function $Q(s,a)$ further conditions on the initial action:
\tableeq{Q(s,a)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t r_t \middle| s_0=s,a_0=a,\pi\right].} \\[0.6em]
\tableeq{Q(s,a)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t r_t \middle| s_0=s,a_0=a,\pi\right]} \\[0.6em]
\bottomrule
\end{tabular}
section2/tables/sft-examples.tex
......@@ -32,7 +32,7 @@ Spam.
\midrule
\instructionbox{\textbf{Machine Translation:} Translate the following sentence from English to Chinese.} \newline
Sentence: Reinforcement learning is widely used to train large language models.
Sentence: RL is widely used to train large language models.
&
强化学习被广泛用于训练大语言模型。
\\
......
section3/algorithm.tex
......@@ -13,7 +13,7 @@
\State \%\%\% \textit{compute policy model loss and value model loss through advantages and returns}
\State initialize the loss values: $\mathrm{loss}_{p} = 0$, $\mathrm{loss}_{v} = 0$
\For{t=1 to T}
\State compute the penalty through Eq. (\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t}) - \log \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State compute the penalty through Eq.~(\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t}) - \log \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State \textbf{if} t==T \textbf{then}
\State \hspace{0.5cm} $r_{t} = R_{\phi}(\mathbf{x}, \mathbf{y}) -\beta \mathrm{Penalty}_{t}$
\State \textbf{else}
......@@ -27,10 +27,10 @@
\State \textbf{end if}
\State \%\%\% \textit{compute policy model loss}
\State compute the advantage $A(\mathbf{x},\mathbf{y}_{<t},y_{t})$: $A(\mathbf{x},\mathbf{y}_{<t},y_{t})=r_{t}+\gamma V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the policy model loss through Eq. (\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p} = \mathrm{loss}_{p} + \mathrm{Clip}\Big( \frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})} \Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State compute the policy model loss through Eq.~(\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p} = \mathrm{loss}_{p} + \mathrm{Clip}\Big( \frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})} \Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State \%\%\% \textit{compute value model loss}
\State compute the return $\mathrm{Return}_{t}$: $\mathrm{Return}_{t}=A(\mathbf{x},\mathbf{y}_{<t},y_t)+V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the value model loss through Eq. (\ref{eq:value-loss}) and add it to $\mathrm{loss}_{v}$: $\mathrm{loss}_v = \mathrm{loss}_v + (\mathrm{Return}_{t}-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t}))^2$
\State compute the value model loss through Eq.~(\ref{eq:value-loss}) and add it to $\mathrm{loss}_{v}$: $\mathrm{loss}_v = \mathrm{loss}_v + (\mathrm{Return}_{t}-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t}))^2$
\EndFor
\State update the parameters of $\pi_{\theta}(\cdot)$ through $\mathrm{loss}_{p}$
\State update the parameters of $V_{\omega}(\cdot)$ through $\mathrm{loss}_{v}$
......
section6/section6.tex
......@@ -60,13 +60,13 @@ Unlike single-turn response generation, agent planning requires a sequence of in
\label{fig:agent-planning-sft}
\end{figure*}
Before applying RL, SFT is commonly adopted to provide a cold start for LLM-based agents by teaching them basic planning and interaction behaviors \citep{chen-etal:fireact,zhang-etal:agentohana,zeng-etal:agenttuning}. As illustrated in Figure \ref{fig:agent-planning-sft}, constructing SFT data for agent planning typically involves three stages: (1) preparing diverse environments and planning tasks; (2) synthesizing expert-level trajectories, consisting of action-observation sequences, through agent-environment interactions; and (3) selecting high-quality trajectories based on predefined evaluation criteria. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as expert agents and retaining reliable demonstrations according to reward signals or task success criteria. Through this process, LLMs can learn planning behaviors from demonstrations and improve their ability to generate structured execution plans.
Before applying RL, SFT is commonly adopted to provide a cold start for LLM-based agents by teaching them basic planning and interaction behaviors \citep{chen-etal:fireact,zhang-etal:agentohana,zeng-etal:agenttuning}. As illustrated in Figure~\ref{fig:agent-planning-sft}, constructing SFT data for agent planning typically involves three stages: (1) preparing diverse environments and planning tasks; (2) synthesizing expert-level trajectories, consisting of action-observation sequences, through agent-environment interactions; and (3) selecting high-quality trajectories based on predefined evaluation criteria. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as expert agents and retaining reliable demonstrations according to reward signals or task success criteria. Through this process, LLMs can learn planning behaviors from demonstrations and improve their ability to generate structured execution plans.
\begin{eqnarray}
\mathbf{x} &=& [e,q] \\
\mathbf{y} &=& [p,u_1,o_1,\cdots,u_T,o_T]
\end{eqnarray}
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section~\ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
Similar to LLM instruction tuning \citep{longpre-etal:flan,zhou-etal:lima}, the scale and quality of trajectory data significantly influence the effectiveness of agent training. High-quality trajectories provide explicit supervision signals for agents to learn how to decompose complex tasks, select appropriate tools, and interact with dynamic environments. Therefore, recent studies have explored constructing trajectory-based instruction tuning datasets to enhance the planning capabilities of LLM-based agents.
......@@ -297,7 +297,7 @@ R(\tau) \rightarrow \{r_p,r_1,\cdots,r_T\}
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
The mechanism of credit assignment is illustrated in Figure~\ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. These step-level rewards are diagnostic signals rather than a conservative numerical decomposition, so they need not sum to the trajectory-level reward. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
\begin{eqnarray}
......
section7/section7.tex
......@@ -23,7 +23,7 @@ A representative example is the multimodal language model\footnote{A multimodal
Although the overall RL formulation can be reused, training VLMs with RL is still challenging in practice. The central difficulty is not the policy optimization algorithm itself, but the scarcity of high-quality multimodal preference data. Compared with textual preference data, visual preference data is more expensive to collect, harder to verify, and often more task-dependent. This makes it difficult to train a reliable visual reward model, which is a key component for applying RLHF-style methods to VLMs.
One straightforward way to alleviate this problem is to generate visual preference data through the automatic preference data generation method described in Section \ref{sec:automatic-preference-data-generation} \citep{yu-etal:2024rlaif}. Another alternative is a multi-stage training approach for the visual reward model, motivated by a simple idea: human preferences are already well captured in text, and these preferences can be transferred across modalities \citep{wang-etal:2024rovrm}. By leveraging textual preference data, this approach reduces the dependence on large-scale visual preference data when training a visual reward model. More specifically, as illustrated in Figure \ref{fig:preference-transfer}, we can train a visual reward model in the following three stages:
One straightforward way to alleviate this problem is to generate visual preference data through the automatic preference data generation method described in Section~\ref{sec:automatic-preference-data-generation} \citep{yu-etal:2024rlaif}. Another alternative is a multi-stage training approach for the visual reward model, motivated by a simple idea: human preferences are already well captured in text, and these preferences can be transferred across modalities \citep{wang-etal:2024rovrm}. By leveraging textual preference data, this approach reduces the dependence on large-scale visual preference data when training a visual reward model. More specifically, as illustrated in Figure~\ref{fig:preference-transfer}, we can train a visual reward model in the following three stages:
\begin{itemize}
\item Stage 1: pre-training with large-scale textual preference data. Given the transferability of human preferences across different modalities, we can begin by using large-scale textual preference data to pre-train the visual reward model. Note that the projector parameters are frozen without images. This stage can be considered as providing a stronger starting point for training the visual reward model, as it enables the model to pre-learn general human preferences.
\item Stage 2: fine-tuning with image caption-based preference data. Pre-learned human preferences cannot be directly applied to vision tasks due to both \textit{task gap} and \textit{modality gap}. To bridge the task gap, we fine-tune the reward model using image caption-based preference data in this stage. The rationale behind this approach is that general textual preference data does not cover vision-specific tasks, such as ``Please describe the content in this image''. We use such data to fine-tune the model, adapting it to vision-specific tasks. The projector parameters are frozen at this stage.
......
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论