@@ -122,7 +122,7 @@ where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the tokens that have already been
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation. One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}. Table~\ref{tab:sft-examples} gives several examples of SFT data for different tasks.
\begin{table}[h]
\begin{table}[!t]
\centering
\small
\input{section2/tables/sft-examples.tex}
...
...
@@ -233,43 +233,16 @@ The objective of RL is to learn an optimal policy $\pi^*$ that maximizes the exp
In practice, the agent improves its policy through repeated interactions with the environment by observing states and receiving feedback in the form of rewards. Each such interaction sequence constitutes a trajectory $\tau$. The process of generating one or more trajectories under a given policy is commonly referred to as \textit{sampling} in the RL literature.
\subsubsection{Key Elements}
To better understand the application of reinforcement learning to large language models (LLMs), we summarize the key elements of the RL framework and reinterpret them in the context of language modeling.
\begin{itemize}
The MDP formulation provides a compact way to describe RL, but its terminology can feel distant from the language modeling setting at first glance. To make the connection more explicit, Table~\ref{tab:rl-elements-and-llm-counterparts} summarizes the core elements of RL and describes how each one can be interpreted in LLM training. This tabular view is intended to serve as a bridge between the standard RL notation introduced above and the token-level generation process discussed earlier.
\item\textbf{Agent.}
The agent is the learner or decision-maker in reinforcement learning. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals.
\item\textbf{Environment.}
The environment comprises everything external to the agent with which it interacts. Unlike traditional RL settings that involve physical or simulated environments, the environment in LLM-based RL is typically abstract, consisting of the training framework that provides feedback (e.g., reward models, human annotations, or evaluation metrics) for generated outputs.
\item\textbf{State ($S$).}
A state represents the current situation of the environment. For language modeling, the state at timestep $t$ can be defined as the sequence of observed tokens up to that point, i.e., the context used to predict the next token. Formally, the state can be represented as $S =(x, y_{<t})$, where $x$ denotes the input prompt and $y_{<t}$ denotes the previously generated tokens.
\item\textbf{Action ($a$).}
An action corresponds to a decision made by the agent. In LLMs, actions are naturally defined as selecting the next token from the vocabulary, i.e., $a = y_t$.
\item\textbf{Reward ($r$).}
The reward provides feedback from the environment to evaluate the quality of an action. In general, the reward function can be defined as $r(s, a, s')$, representing the feedback received when the agent transitions from state $s$ to $s'$ by taking action $a$. At timestep $t$, this can be written as $r_t = r(s_t, a_t, s_{t+1})$. In deterministic settings, where the next state is uniquely determined by $(s_t, a_t)$, the reward can be simplified as $r(s_t, a_t)$.
\item\textbf{Policy ($\pi$).}
The policy defines the agent’s behavior, i.e., the probability of taking an action given a state. For LLMs, the policy corresponds to the conditional probability distribution over the next token given the context:
\begin{equation}
\pi(a \mid s) = \Pr(y_t \mid x, y_{<t})
\end{equation}
where $a = y_t$ and $s =(x, y_{<t})$. Under this formulation, an LLM can be naturally interpreted as a parameterized policy.
\item\textbf{Value Function ($V$ and $Q$).}
The value function estimates the expected cumulative reward when following a policy. The state-value function $V(s)$ measures the expected discounted return starting from state $s$:
\caption{Key elements of RL and their interpretations}
\label{tab:rl-elements-and-llm-counterparts}
\end{table}
By this point, the reader should have developed a foundational understanding of the core concepts and notations in RL. In contrast to conventional RL literature, which typically presents algorithms in the context of classical robotics or control tasks, \textit{we adopt an NLP perspective to introduce RL in the following section}.
With these correspondences in place, we can describe RL algorithms using familiar language-modeling objects such as contexts, tokens, policies, and reward signals. In contrast to conventional RL literature, which often presents algorithms through robotics or control examples, \textit{we adopt an NLP perspective to introduce RL in the following section}.
The learner or decision-maker in reinforcement learning. In the context of LLMs, the agent corresponds to the language model itself, which generates tokens sequentially and updates its behavior based on feedback signals. \\
\midrule
\textbf{Environment}&
Everything external to the agent with which it interacts. Unlike traditional RL settings that involve physical or simulated environments, the environment in LLM-based RL is typically abstract, consisting of the training framework that provides feedback (e.g., reward models, human annotations, or evaluation metrics) for generated outputs. \\
\midrule
\textbf{State ($S$)}&
A state represents the current situation of the environment. For language modeling, the state at timestep $t$ can be defined as the sequence of observed tokens up to that point, i.e., the context used to predict the next token. Formally, the state can be represented as $S=(x,y_{<t})$, where $x$ denotes the input prompt and $y_{<t}$ denotes the previously generated tokens. \\
\midrule
\textbf{Action ($a$)}&
An action corresponds to a decision made by the agent. In LLMs, actions are naturally defined as selecting the next token from the vocabulary, i.e., $a=y_t$. \\
\midrule
\textbf{Reward ($r$)}&
The reward provides feedback from the environment to evaluate the quality of an action. In general, the reward function can be defined as $r(s,a,s')$, representing the feedback received when the agent transitions from state $s$ to $s'$ by taking action $a$. At timestep $t$, this can be written as $r_t=r(s_t,a_t,s_{t+1})$. In deterministic settings, where the next state is uniquely determined by $(s_t,a_t)$, the reward can be simplified as $r(s_t,a_t)$. \\
\midrule
\textbf{Policy ($\pi$)}&
The policy defines the agent's behavior, i.e., the probability of taking an action given a state. For LLMs, the policy corresponds to the conditional probability distribution over the next token given the context:
\tableeq{\pi(a \mid s)=\Pr(y_t \mid x,y_{<t})}
\vspace{-1mm}
where $a=y_t$ and $s=(x,y_{<t})$. Under this formulation, an LLM can be naturally interpreted as a parameterized policy. \\
\midrule
\textbf{\makecell[l]{Value Function\\($V$ and $Q$)}}&
The value function estimates the expected cumulative reward when following a policy. The state-value function $V(s)$ measures the expected discounted return starting from state $s$:
