\bibitem[Fan et~al.(2018)Fan, Lewis, and Dauphin]{fan-etal:hierarchical}
Angela Fan, Mike Lewis, and Yann Dauphin.
\newblock Hierarchical neural story generation.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 889--898, 2018.
\bibitem[Fan et~al.(2023)Fan, Watkins, Du, Liu, Ryu, Boutilier, Abbeel, Ghavamzadeh, Lee, and Lee]{fan-etal:dpok}
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.
\newblock Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.
...
...
@@ -240,6 +245,11 @@ Alex Havrilla, Yuqing Du, Sharath~Chandra Raparthy, Christoforos Nalmpantis, Jan
\newblock Teaching large language models to reason with reinforcement learning, 2024.
\title{Reinforcement Learning without Tears:\\ An Introduction in the Era of Large Language Models}
\title{RL without Tears:\\ An Introduction in the Era of LLMs}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \colmfinalcopy macro remains commented out below.
...
...
@@ -170,6 +170,7 @@
\newcommand{\new}{\marginpar{NEW}}
\begin{document}
\begin{CJK*}{UTF8}{gbsn}
\ifcolmsubmission
\linenumbers
...
...
@@ -179,7 +180,7 @@
\begin{abstract}
Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs). It is widely used to align models with human preferences and improve their reasoning capabilities. More recently, with the rapid rise of LLM-based agents, RL has become increasingly important for enabling models to learn from experience and build more capable and adaptive intelligent systems. Despite its growing importance, RL remains difficult for many LLM researchers to approach. Traditional RL literature often explains key concepts through robotics and control problems, making them less intuitive in the context of LLMs. To bridge this gap, this paper provides a comprehensive introduction to RL in the era of LLMs. We first introduce the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models. We show how RL can enhance reasoning, train agents through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs). It is widely used to align models with human preferences and improve their reasoning capabilities. More recently, with the rapid rise of LLM-based agents, RL has become increasingly important for enabling models to learn from experience and build more capable and adaptive intelligent systems. Despite its growing importance, RL remains difficult for many LLM researchers to approach. Traditional RL literature often explains key concepts through robotics and control problems, making them less intuitive in the context of LLM training. To bridge this gap, this paper provides a comprehensive introduction to RL through LLM training examples. We first introduce the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models. We show how RL can enhance reasoning, train agents through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
\end{abstract}
...
...
@@ -206,11 +207,11 @@ This hypothesis highlights why RL is particularly relevant to the next stage of
Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in the era of LLMs, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLM training seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in LLM training, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
It is natural to learn RL from standard references, which appears straightforward at first. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM research. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through a concrete example of training an LLM. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM training. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through concrete LLM training examples. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
...
...
@@ -259,4 +260,5 @@ We would like to thank those who provided suggestions on improving the early ver
\subsection{Fundamentals of Large Language Models}
Large language models (LLMs) have become the foundation of modern natural language processing. Their success largely follows a simple but powerful paradigm: first learn general language capabilities from large-scale text corpora, and then adapt these capabilities to specific tasks through prompting or supervised fine-tuning. At inference time, an LLM generates an output by predicting tokens sequentially according to the given input and previously generated tokens. This paradigm has substantially changed how NLP systems are developed, replacing many task-specific models with a unified foundation model that can support a wide range of language understanding and generation tasks.
\subsection{Fundamentals of LLMs}
LLMs have become the foundation of modern natural language processing. Their success largely follows a simple but powerful paradigm: first learn general language capabilities from large-scale text corpora, and then adapt these capabilities to specific tasks through prompting or supervised fine-tuning. At inference time, an LLM generates an output by predicting tokens sequentially according to the given input and previously generated tokens. This paradigm has substantially changed how NLP systems are developed, replacing many task-specific models with a unified foundation model that can support a wide range of language understanding and generation tasks.
Understanding these basic mechanisms is important before introducing RL for LLMs. Many RL concepts can be naturally connected to the standard generation process of an LLM. The input and previously generated tokens define the current context, the next-token distribution determines the model's possible actions, and the complete output forms a sequence of decisions that can be evaluated by a reward signal. In this subsection, we briefly review the fundamentals of LLMs and introduce the key concepts and notations used throughout this paper. We first describe pre-training, through which LLMs acquire general language capabilities from large-scale data. We then introduce prompting, which enables a pre-trained model to perform different tasks through natural-language instructions without parameter updates. Next, we discuss supervised fine-tuning, which further adapts the model using labeled input-output pairs. Finally, we describe the inference process and explain how an LLM generates an output token by token. It is worth noting that we focus only on the concepts necessary for understanding the subsequent discussion of RL, rather than providing a comprehensive review of these techniques and their recent developments. Interested readers are referred to \citet{xiao-and-zhu:2025foundations} for a more systematic introduction to LLMs.
...
...
@@ -13,6 +13,7 @@ In this paper, we mainly focus on generative LLMs based on decoder-only Transfor
where $\mathbf{z}_{<i}=z_1...z_{i-1}$ denotes the tokens before $z_i$. A decoder-only Transformer implements this conditional distribution by applying causal self-attention, so that the prediction at each position can only depend on the previous tokens. The model finally produces a probability distribution over $\mathcal{V}$, denoted by $\mathrm{Pr}_{\theta}(\cdot|\mathbf{z}_{<i})$. This next-token prediction view is the central interface through which LLMs are trained, prompted, fine-tuned, and later optimized by RL.
...
...
@@ -24,12 +25,12 @@ Let $\mathcal{S}_{\mathrm{pre}}$ denote the pre-training corpus, where each samp
where $\hat{\theta}$ denotes the optimized pre-trained parameters. Equivalently, this objective can be viewed as minimizing the cross-entropy loss between the observed next token and the model-predicted distribution.
where $\hat{\theta}$ denotes the optimized pre-trained parameters. Equivalently, this objective can be viewed as minimizing the cross-entropy loss between the observed next token and the model-predicted distribution.
After pre-training, the LLM learns to model natural language and acquires rich linguistic and factual knowledge from large-scale text. Given a preceding context, it can predict plausible continuations and generate fluent text. For example, given the prefix ``The capital of France is'', a pre-trained LLM may assign a high probability to the next token ``Paris''. Similarly, given a longer context such as ``To solve this equation, we first move all terms to one side'', the model can continue the text with a linguistically and semantically plausible sequence. These examples reflect the core capability learned during pre-training: predicting likely continuations from the preceding context.
After this stage, the LLM can generate fluent text and encode rich linguistic and factual knowledge. However, the pre-training objective itself does not explicitly teach the model how to follow user instructions, satisfy task-specific formats, or align with human preferences. Therefore, additional adaptation is typically needed before deploying the model for real applications.
However, pre-training primarily teaches the model to perform language modeling rather than to explicitly solve user-specified tasks. In particular, the model is not directly trained to interpret instructions or produce outputs in a desired format. As a result, simply providing a task description to a pre-trained LLM does not necessarily lead to the intended behavior. This limitation motivates us to use additional adaptation approaches to better equip pre-trained LLMs for downstream tasks.
\subsubsection{Prompting}
...
...
@@ -114,11 +115,20 @@ Following the notation used above, we denote the entire prompt by $\mathbf{x}=x_
where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the tokens that have already been generated. This conditional generation formulation will be repeatedly used in the following sections. From this perspective, a prompt mainly changes the condition under which the same language model distribution is used.
\subsubsection{Supervised Fine-Tuning}
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation. One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation. One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}. Table~\ref{tab:sft-examples} gives several examples of SFT data for different tasks.
\begin{table}[h]
\centering
\small
\input{section2/tables/sft-examples.tex}
\caption{Examples of SFT data for different downstream tasks.}
\label{tab:sft-examples}
\end{table}
Specifically, let $\mathbf{x}=x_1...x_m$ be an input and $\mathbf{y}=y_1...y_T$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
\begin{eqnarray}
...
...
@@ -136,19 +146,71 @@ This formulation is equivalent to minimizing the cross-entropy loss over the out
\subsubsection{Inference}
Inference is the process of applying a trained LLM to generate outputs for new inputs. Given an input $\mathbf{x}$, the LLM predicts the next-token distribution $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ and selects a token $y_1$ from this distribution. The selected token is then appended to the context, and the model predicts the next token according to $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x},y_1)$. This procedure is repeated until a stopping criterion is met, such as generating an end-of-sequence token or reaching a maximum length.
Inference is the process of applying a trained LLM to generate outputs for new inputs. Given an input $\mathbf{x}$, the model first predicts a next-token distribution $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ and selects a token $y_1$ according to a decoding strategy. The selected token is then appended to the context, after which the model predicts the next token according to $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x},y_1)$. This process is repeated until a stopping criterion is reached, such as generating an end-of-sequence token or reaching the maximum generation length. We refer to this sequential generation process as \textit{autoregressive generation}.
Different decoding strategies determine how the next token is selected from the model-predicted distribution. Commonly used strategies include:
\begin{itemize}
\item\textbf{Greedy Decoding.}
Greedy decoding is one of the simplest decoding strategies. At each generation step, it selects the token with the highest probability under the model. Consider the probability of the generated sequence up to timestep $t$:
where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the previously generated tokens. Since the first term is fixed once $\mathbf{y}_{<t}$ has been generated, greedy decoding selects
The selected token is appended to the current prefix, and the same procedure is repeated at the next timestep. Greedy decoding is deterministic and computationally efficient, but locally selecting the most probable token does not necessarily produce the most probable complete sequence.
\item\textbf{Beam Search.}
Beam search extends greedy decoding by maintaining multiple candidate sequences during generation. Instead of keeping only the single best prefix at each timestep, it retains the top $B$ candidates, where $B$ is referred to as the \textit{beam size}. Let $\mathcal{B}_{t-1}$ denote the set of candidate prefixes retained at timestep $t-1$. Each prefix $\mathbf{y}_{<t}$ is extended with possible next tokens, and the resulting candidates are ranked according to their accumulated sequence scores:
where $w$ denotes a candidate token in the vocabulary $\mathcal{V}$. Other decoding methods, such as sampling, top-$k$ sampling, top-$p$ sampling, and beam search, use different strategies to balance output quality, diversity, and computational cost. Regardless of the decoding method, the generation process can be viewed as a sequence of token-level decisions.
This view naturally connects LLMs with RL. At timestep $t$, the current context $(\mathbf{x},\mathbf{y}_{<t})$ can be treated as the state, the next token $y_t$ can be treated as the action, and $\mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})$ can be treated as the policy. The whole output $\mathbf{y}$ is then analogous to a trajectory, whose quality can be evaluated by a reward signal. Based on this connection, the following subsection reviews the fundamentals of RL, and Section~\ref{sec:example-using-rl-training-llms} will further show how RL can be used to optimize an SFT LLM.
By exploring multiple prefixes simultaneously, beam search can avoid some locally optimal decisions made by greedy decoding and often produces higher-probability sequences. However, it requires more computation and memory as the beam size increases.
\item\textbf{Sampling.}
Sampling is widely used in modern LLM inference, particularly when diverse outputs are desired. Instead of always selecting the most probable token or maintaining a fixed set of candidate sequences, the next token is sampled from the model-predicted distribution:
\begin{eqnarray}
y_t \sim\mathrm{Pr}_{\theta}\left(
\cdot|\mathbf{x},\mathbf{y}_{<t}
\right)
\end{eqnarray}
This makes generation stochastic, allowing the same input to produce different outputs. In practice, the sampling distribution is often adjusted using a temperature parameter $T$. Given the model logit $l_w$ for token $w$, the temperature-adjusted probability is
A smaller temperature sharpens the distribution and favors high-probability tokens, while a larger temperature produces a flatter distribution and increases diversity. Sampling is also commonly combined with \textit{top-$k$} or \textit{top-$p$} filtering. Top-$k$ sampling restricts the candidate set to the $k$ tokens with the highest probabilities \citep{fan-etal:hierarchical}, whereas top-$p$ sampling retains the smallest set of tokens whose cumulative probability reaches a threshold $p$\citep{holtzman-etal:curious}. The next token is then sampled from the renormalized distribution over the retained candidates. These techniques help reduce the probability of selecting unlikely tokens while preserving diversity.
\end{itemize}
Regardless of the decoding strategy, LLM generation can be viewed as a sequence of token-level decisions. This view naturally connects LLMs with RL. At timestep $t$, the current context $(\mathbf{x},\mathbf{y}_{<t})$ can be treated as the state, the next token $y_t$ as the action, and the next-token distribution $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x},\mathbf{y}_{<t})$ as the policy. The complete output $\mathbf{y}$ then corresponds to a trajectory whose quality can be evaluated by a reward signal. In particular, sampling closely resembles the stochastic policy sampling process used in RL. Based on this connection, the following subsection reviews the fundamentals of RL, and Section~\ref{sec:example-using-rl-training-llms} further shows how RL can be used to optimize an SFT LLM.
\subsection{Fundamentals of Reinforcement Learning}
RL, supported by a well-established theoretical framework, has been widely applied across a broad range of domains. In particular, with the rapid advancement of LLMs, RL has become a standard approach in the post-training stage. Before exploring the interplay between RL and LLMs, we first provide a brief overview of deep reinforcement learning in this subsection. We then introduce the general formulation of RL, along with key terminologies and notations that are essential for understanding RL.
\subsection{Fundamentals of RL}
RL, supported by a well-established theoretical framework, has been widely applied across a broad range of domains. In particular, with the rapid advancement of LLMs, RL has become a standard approach in the post-training stage. Before using LLM training examples to explain RL algorithms, we first provide a brief overview of deep reinforcement learning in this subsection. We then introduce the general formulation of RL, along with key terminologies and notations that are essential for understanding RL.
\subsubsection{Markov Decision Process}
The general framework of RL is illustrated in Figure~\ref{fig:rl_framework}. An RL system primarily consists of two components: an agent and an environment. At each timestep $t$, the agent observes a state $s_t$ from the environment and selects an action $a_t$ according to a policy $\pi$, which is typically parameterized by a neural network. After executing the action, the environment transitions to a new state $s_{t+1}$ and returns a reward $r_t$ corresponding to the taken action.
% An example of REINFORCE and optimization objective
\section{A Simple Example}
\section{Understanding RL in LLM Training}
\label{sec:example-using-rl-training-llms}
We continue with a practical example of using an LLM as a homework assistant. In this context, we aim to improve the capability of an SFT LLM to handle education-related inputs more effectively. Suppose we have a homework assistant powered by an SFT LLM, and a student types the input ``Give me three tips to improve my accuracy in solving math problems.'' A typical SFT LLM might generate a short output, such as
We continue with a practical example of using an LLM as a homework assistant. In this context, we want to improve the capability of an SFT LLM to handle education-related inputs more effectively. Suppose we have a homework assistant powered by an SFT LLM, and a student types the input ``Give me three tips to improve my accuracy in solving math problems.'' A SFT LLM typically generates a short output, such as
\vspace{0.1cm}
\begin{tcolorbox}[frame empty]
...
...
@@ -23,7 +23,7 @@ There are three tips for improving accuracy in solving math problems: \\ [1mm]
Although this output adheres to the given input, it is very short and not sufficiently detailed or comprehensive for this scenario. In practice, the ``short'' feature in the generated output stems from the training approach of the SFT LLM. During SFT, the LLM learns from a large set of labeled samples that typically contain concise and factual answers to common inputs. These samples guide the LLM to prioritize brevity and clarity, so it often generates short outputs, like the one shown above. Of course, we could annotate enough additional data to fine-tune the LLM further and adjust it to generate the desired outputs. However, this approach is limited in its ability to scale. For example, in this scenario, the input involves a variety of potential answers, and the expectations of the student can be pretty diverse. Therefore, describing the ``ideal'' output would require immense annotation effort. Consequently, collecting or annotating fine-tuning data is not as straightforward as it is with SFT, particularly when attempting to cover the breadth of possible student inputs and outputs. Instead, we can use RL to enable the model to discern outputs that better align with human preferences, such as generating more detailed and comprehensive content that not only adheres to the given input but also meets the expectations of the student in this scenario.
In the following sections, we will demonstrate how to use RL to train LLMs through a specific example, enabling the LLM to generate a longer output in the context of a homework assistant. Throughout this example, we introduce RL, including key algorithms and their improvements.
In the following sections, we use a specific LLM training example to explain how RL works, where the LLM is trained to generate a longer output in the context of a homework assistant. Throughout this example, we introduce key RL algorithms and their improvements.
\section{Improved Reinforcement Learning for LLMs}
\section{Improved RL for LLMs}
\vspace{-2mm}
In the previous section, we introduced some improvements to RL, such as importance sampling and reward baseline techniques. However, directly applying them to train LLMs still presents numerous challenges. In this section, we will delve deeper into the improvements for using RL to train LLMs.
...
...
@@ -531,8 +531,8 @@ Moreover, there are other methods like GRPO that design advantage estimation wit
Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in the context of LLMs due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.
\subsection{Efficient RL Methods}
Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in LLM training due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.
In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.
...
...
@@ -542,7 +542,7 @@ As mentioned in Section \ref{sec:policy-gradient}, training LLMs with RL typical
From the perspective of LLM inference, there are many methods to reduce the time overhead of sampling: since the sampling process is implemented by LLM inference, we have reason to believe that any method that accelerates inference can also be applied to reduce the time required for sampling.
Examples include KV-Cache \citep{pope-etal:2023efficiently}, quantization \citep{zhao-etal:2024atom}, and speculative decoding \citep{chen-etal:2023accelerating}. However, in this section, we will not discuss these methods in detail, as there are numerous approaches, each addressing different aspects of LLM inference. We refer the interested readers to these papers for more details. Instead, we will focus on one particular method that aims to reduce sampling overhead from the perspective of optimizing RL for training LLMs.
Before discussing specific methods, let us first review the purpose of sampling. Given a sample $\mathbf{x}$, the goal of sampling is to explore a better output $\mathbf{y}$ from the policy model. This process allows us to evaluate different possible outputs in order to find those that lead to improved outcomes, enabling the model to make more informed decisions and optimize its performance on the state. Unlike typical RL scenarios, where policy models may start with limited information or random initialization, the policy model in the context of LLMs is usually quite advanced. These LLMs often begin as pre-trained models equipped with substantial knowledge acquired through pre-training and SFT. As a result, for certain samples, we can consider that the policy model may already perform well without the need for further exploration and optimization. This can be viewed through the lens of the classic RL dilemma of exploration versus exploitation \citep{Sutton-and-Barto:2018RL}. In such cases, exploration may be less necessary, and exploiting the knowledge already embedded in the pre-trained model could be more effective. Thus, a balance must be struck between exploring new possibilities and leveraging the pre-existing knowledge of the policy model to maximize efficiency.
Before discussing specific methods, let us first review the purpose of sampling. Given a sample $\mathbf{x}$, the goal of sampling is to explore a better output $\mathbf{y}$ from the policy model. This process allows us to evaluate different possible outputs in order to find those that lead to improved outcomes, enabling the model to make more informed decisions and optimize its performance on the state. Unlike typical RL scenarios, where policy models may start with limited information or random initialization, the policy model in LLM training is usually quite advanced. These LLMs often begin as pre-trained models equipped with substantial knowledge acquired through pre-training and SFT. As a result, for certain samples, we can consider that the policy model may already perform well without the need for further exploration and optimization. This can be viewed through the lens of the classic RL dilemma of exploration versus exploitation \citep{Sutton-and-Barto:2018RL}. In such cases, exploration may be less necessary, and exploiting the knowledge already embedded in the pre-trained model could be more effective. Thus, a balance must be struck between exploring new possibilities and leveraging the pre-existing knowledge of the policy model to maximize efficiency.
To achieve this balance between exploration and exploitation, one simple and straightforward approach to improving the efficiency of sampling is to identify the samples that are required to explore and perform sampling only for those samples. Given an input-only dataset $\mathcal{S}_x =\{\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N\}$, where $N$ denotes the size of the dataset, we can use a reward model to evaluate the need for exploration by the policy model. First, we generate $\hat{\mathbf{y}}$ for each $\mathbf{x}$ using greedy search. Then, by taking $\mathbf{x}$ and $\hat{\mathbf{y}}$ as input to the reward model, the reward for the $k$-th sample $\mathbf{x}_k$ is computed as $R_{\phi}(\mathbf{x}_{k}, \hat{\mathbf{y}}_{k})$.
\section{Reinforcement Learning for LLM Reasoning}
\section{RL for LLM Reasoning}
So far, our discussion has mainly focused on various aspects of using and improving RL for training LLMs. The methods mentioned can be easily adapted to a wide range of scenarios where the correctness of an output can be examined by checking whether the desired result is included. For example, in the task of calculating a mathematical expression, a reward model can provide positive feedback if the answer is correct and negative feedback if the answer is wrong. However, in many problems that require complex reasoning, simply examining the correctness of the final answer is insufficient for learning. Imagine a student who is only given the final answer to a challenging math problem. Knowing whether the final answer is right or wrong does not help the student figure out where they went wrong and how to calculate the correct answer. A better approach would be to guide the student with a step-by-step breakdown of the problem-solving process and encourage understanding of the underlying concepts and logic behind these steps. To address this, researchers also explore the potential of RL to teach LLMs not just to generate correct answers but to develop and present coherent, step-by-step reasoning that aligns with human cognitive processes. In this regard, RL has previously demonstrated its effectiveness in training neural networks for complex planning and reasoning within game environments, as evidenced by notable successes such as AlphaGo \citep{silver-etal:2016mastering} and AlphaStar \citep{vinyals-rtal:2019grandmaster}. Given these advancements and the inherently interactive nature of problem-solving, it is also natural to consider the application of RL to LLM reasoning.
In this section, we delve deeper into the application of RL to enhance the reasoning capabilities of the LLM, a topic that has recently garnered significant attention. We begin by giving a general introduction to test-time scaling that fully unleashes the reasoning potential of LLMs, including best-of-$N$ sampling, step-by-step verification, and Monte Carlo Tree Search. We then discuss two critical issues: how to scale RL effectively, and how to iterate the RL process to enhance the reasoning capabilities of LLMs. Note that the following methods are primarily illustrated through mathematical reasoning problems, but they are applicable to a broad range of decision-making problems.
...
...
@@ -92,7 +95,7 @@ In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popu
\item\textbf{Backpropagation}. The process updates the UCT values of prior nodes using the results of these simulations. Key updates include the visitation counts, $N_p$ and $N_{\bar{\mathbf{y}}_{k}}$ in Eq. (\ref{eq:uct}), reflecting how frequently each node has been explored. Additionally, this stage allows for the incorporation of delayed rewards to adjust the $\bar{R}(\bar{\mathbf{y}}_k)$ values.
\end{itemize}
\subsection{Iterative Reinforcement Learning}
\subsection{Iterative RL}
Training LLMs with RL usually follows a two-phase approach: training a pre-trained LLM with SFT and further training with an RL algorithm applied to the SFT LLM. However, using large-scale RL in such a two-phase approach to train LLMs in reasoning may lead to significant knowledge forgetting as the model continuously adjusts to fit the reward model. For example, as mentioned in DeepSeek-R1 \citep{guo:2025deepseek}, directly applying large-scale RL can achieve the desired reasoning outcomes, but often at the cost of reduced readability in the reasoning process. To address these challenges, we can utilize an iterative RL approach to continuously enhance the various capabilities of the LLM. Here we consider DeepSeek-R1 as an example to illustrate how to perform an iterative RL. The idea is to split the RL process into multiple phases, each designed to enhance different capabilities using varied rewards, to develop a robust reasoning model that is able to generate clearer and more comprehensible reasoning paths. Figure \ref{fig:iterative-rl} provides a schematic illustration of the iterative RL process. Here we give a brief outline of each phase involved.
...
...
@@ -117,7 +120,7 @@ An interesting issue arises with this design of iterative RL: why is RL aimed at
Since the optimization objective of each phase is different, the design of iterative RL can also be analyzed and understood from the perspective of multi-objective optimization \citep{wang-etal:2024hybrid}. In the context of multi-objective optimization, iterative RL for LLMs can be likened to interactive methods where the solution process is iterative, and preferences are actively defined and refined by the decision-maker during the search for the most preferred solutions \citep{miettinen-etal:2008introduction,deb-etal:2016multi}. More specifically, in this scenario, RL can be seen as a decision-maker, iteratively refining and enhancing the different capabilities of the LLM across different phases.
\subsection{Large-scale Reinforcement Learning}
\subsection{Large-scale RL}
\label{sec:large-scale-rl}
While test-time scaling is instrumental in exploring more effective reasoning paths, its potential is restricted if the model has not learned to generate high-quality reasoning paths. In other words, in practice, test-time scaling struggles to achieve desired outcomes without foundational reasoning ability, no matter how extensive it is \citep{yuan-etal:2023scaling,zeng-etal:2025revisiting}. Consequently, there has been a growing research interest in training LLMs specifically to develop reasoning abilities that mimic human-like processes. A straightforward approach is to use annotated reasoning data to train the LLM through SFT. However, a significant challenge in this approach is the high cost of annotations, especially since annotating detailed step-by-step reasoning paths is significantly more cost-intensive than annotating SFT data in other tasks like machine translation and summarization.
Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Atari and Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
...
...
@@ -78,7 +78,7 @@ Beyond selecting high-quality trajectories, another challenge lies in effectivel
Furthermore, even with effective trajectory selection and representation, scaling the generation of diverse and high-quality trajectory data remains challenging. Existing approaches often rely on manually designed environments and planning tasks, which limits the diversity and coverage of collected trajectories. Designing new environments requires substantial human expertise, while constructing tasks with appropriate difficulty levels remains difficult. To address this limitation, AGENTGEN explores automatically generating diverse environments and planning tasks with LLMs, enabling large-scale synthesis of trajectory data with varying task complexity for agent training \citep{hu-etal:agentgen}.
\subsubsubsection{Learning to Plan with Reinforcement Learning}
\subsubsubsection{Learning to Plan with RL}
Although SFT can provide a useful cold start for agent planning, it mainly teaches the model to imitate offline demonstrations. This limits its ability to improve beyond the quality and coverage of the collected trajectories. In contrast, RL further optimizes planning through direct interaction with the environment, where the agent receives feedback based on the outcome of its generated plans and executed behaviors.
% This makes RL particularly suitable for improving long-horizon planning, because the quality of a plan is often determined by the success of the entire interaction trajectory rather than by the correctness of any single intermediate step.
...
...
@@ -170,7 +170,7 @@ A representative approach to addressing this data construction problem is Toolfo
Beyond learning to use a small set of tools, later studies further scale tool-use training to larger and more realistic API spaces. API-Bank builds a benchmark and training set for tool-augmented LLMs, where models need to plan, retrieve, and call APIs in executable environments \citep{li-etal:apibank}. ToolLLM constructs ToolBench with more than 16,000 real-world APIs and uses automatically generated instruction-solution trajectories to train ToolLLaMA \citep{qin-etal:toolllm}. Gorilla focuses on API calling at scale and fine-tunes LLaMA-based models to generate accurate API calls. It also uses a document retriever to reduce API hallucination and adapt to changing API documents \citep{patil-etal:gorilla}.
% 4. RL for Tool Use
\subsubsubsection{Learning to Use Tools with Reinforcement Learning}
\subsubsubsection{Learning to Use Tools with RL}
Although SFT can teach LLMs to imitate tool-use demonstrations, it is still limited by the coverage and quality of offline trajectories. The model mainly learns the tool-use patterns that appear in the supervised data. As a result, it may fail to generalize to unfamiliar tools, complex tool combinations, or new interaction patterns. More importantly, SFT does not directly optimize whether a tool call is necessary, whether the selected tool is appropriate, or whether the returned observation improves the final answer. This limitation becomes more serious in multi-step tool-use scenarios, \textit{where the model needs to decide when to call a tool, how to formulate the tool input, how to interpret the tool output, and when to stop using tools}.
Multimodal learning aims to build models that can process and integrate information from multiple modalities, such as vision and text. By combining complementary signals from different modalities, multimodal models can achieve more comprehensive perception and reasoning abilities than single-modality systems. These capabilities have enabled a wide range of applications, including image captioning, visual reasoning, speech understanding, and image generation \citep{baltruvsaitis-etal:2018multimodal}.
In this paper, we have introduced the fundamental concepts of RL from the perspective of LLM research. We began with basic RL formulations and algorithms, including policy gradients, advantage estimation, importance sampling, and reward modeling, and then discussed recent advances in RL. We further extended the discussion to reasoning models, agentic systems, and multimodal models, showing how RL has evolved from a general policy optimization framework into a key paradigm for enhancing LLM reasoning, interactive decision-making, and multimodal generation capabilities.
In this paper, we have introduced the fundamental concepts of RL from the perspective of LLM training. We began with basic RL formulations and algorithms, including policy gradients, advantage estimation, importance sampling, and reward modeling, and explained them through LLM training examples. We then discussed recent advances in RL. We further extended the discussion to reasoning models, agentic systems, and multimodal models, showing how RL has evolved from a general policy optimization framework into a key paradigm for enhancing LLM reasoning, interactive decision-making, and multimodal generation capabilities.
While RL has become well established in LLM training and has achieved remarkable results, several promising directions remain for future exploration:
...
...
@@ -12,4 +12,4 @@ While RL has become well established in LLM training and has achieved remarkable
\item\textbf{Predicting the scaling limits of RL.} The final gains from RL depend on many factors, including model capability, training data, reward quality, and optimization settings. In practice, we often need to complete expensive RL runs before knowing the final performance. Developing methods to predict the potential gains and performance limits of RL before full-scale training could significantly improve the efficiency of model development.
\item\textbf{Continual and self-evolving RL.} Most current RL pipelines optimize a model on a fixed training distribution and stop once training is completed. One more ambitious direction is to enable models to continuously learn from their own interactions, feedback, and accumulated experiences after deployment. This requires new mechanisms for experience selection, memory and skill evolution, and stable policy updating without catastrophic forgetting.
