update.

appendix/appendix.tex

@@ -9,7 +9,7 @@
     \centering
     \centering
     \resizebox{0.98\linewidth}{!}{
     \resizebox{0.98\linewidth}{!}{
     \input{appendix/tables/dataset}}
     \input{appendix/tables/dataset}}
-    \caption{Preference, reasoning, and trainable agentic RL datasets and environments for LLM. Here, RM denotes reward modeling, Rsn. denotes reasoning, Env. denotes environment-based feedback, Env. gen. denotes procedurally generated environments, and TIR denotes tool-integrated reasoning.}
+    \caption{Preference, reasoning, and trainable agentic RL datasets and environments for LLMs. Here, RM denotes reward modeling, Rsn. denotes reasoning, Env. denotes environment-based feedback, Env. gen. denotes procedurally generated environments, and TIR denotes tool-integrated reasoning.}
     \label{tab:dataset}
     \label{tab:dataset}
 \end{table}
 \end{table}
 
 
@@ -22,6 +22,6 @@
     \centering
     \centering
     \resizebox{0.98\linewidth}{!}{
     \resizebox{0.98\linewidth}{!}{
     \input{appendix/tables/systems}}
     \input{appendix/tables/systems}}
-    \caption{RL systems for LLM and multimodal post-training, including their supported modalities and corresponding training approaches.}
+    \caption{RL systems for LLM-based and multimodal post-training, including their supported modalities and corresponding training approaches.}
     \label{tab:system}
     \label{tab:system}
 \end{table}
 \end{table}

rl-introduction.tex

@@ -198,7 +198,7 @@ License: CC-BY 4.0, see \url{https://creativecommons.org/licenses/by/4.0/}.
 
 
 RL is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles. 
 RL is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles. 
 
 
-A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of LLMs. For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1,kimi-team:kimi}. As a result, interest in NLP has exploded. There have been many, many conference papers discussing RL in LLMs, and even more in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI. This trend is further strengthened by the rise of LLM-based agents, where models must learn from interaction and experience to make effective decisions in dynamic environments. From this perspective, RL is becoming a key technique for building more capable and adaptive intelligent systems. This view closely echoes the ``Reward Is Enough'' hypothesis proposed by \citet{silver-etal:reward}:
+A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of LLMs. For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1,kimi-team:kimi}. As a result, interest in RL for LLMs has exploded. Numerous conference papers now discuss RL in LLMs, with even more appearing in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI. This trend is further strengthened by the rise of LLM-based agents, where models must learn from interaction and experience to make effective decisions in dynamic environments. From this perspective, RL is becoming a key technique for building more capable and adaptive intelligent systems. This view closely echoes the ``Reward Is Enough'' hypothesis proposed by \citet{silver-etal:reward}:
 \begin{quote}
 \begin{quote}
 ``Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.''
 ``Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.''
 \end{quote}
 \end{quote}
@@ -208,7 +208,7 @@ This hypothesis highlights why RL is particularly relevant to the next stage of
 
 
 Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in the era of LLMs, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
 Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in the era of LLMs, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
 
 
-Of course, we'd like to learn about RL, which appears straightforward. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
+It is natural to learn RL from standard references, which appears straightforward at first. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
 
 
 The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM research. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through a concrete example of training an LLM. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
 The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM research. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through a concrete example of training an LLM. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
 
 
@@ -245,7 +245,7 @@ The aim of this paper is to provide a comprehensive introduction to RL from the
 
 
 
 
 \section*{Acknowledgements}
 \section*{Acknowledgements}
-We would like to thank  for their suggestions on improving the early version of this work.
+We would like to thank those who provided suggestions on improving the early version of this work.
 
 
 % \clearpage
 % \clearpage
 \input{appendix/appendix}
 \input{appendix/appendix}

section2/section2.tex

@@ -4,10 +4,10 @@
 \label{sec:preliminary}
 \label{sec:preliminary}
 
 
 \subsection{Fundamentals of Large Language Models}
 \subsection{Fundamentals of Large Language Models}
-In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in future sections.
+In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in the following sections.
 
 
 Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
 Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
-One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
+One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
 
 
 Specifically, let $\mathbf{x}=x_1...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
 Specifically, let $\mathbf{x}=x_1...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
 \begin{eqnarray}
 \begin{eqnarray}

section7/section7.tex

@@ -76,9 +76,9 @@ To address this challenge, researchers have explored RL as an effective approach
 \label{eq:optimization_objective}
 \label{eq:optimization_objective}
 \end{eqnarray}
 \end{eqnarray}
 
 
-Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. For example, given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color:
+Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. Given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color.
 
 
-However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because that diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as a MDP. Specifically, we can consider the denoising procedure as a multi-step MDP and applies a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
+However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as an MDP. Specifically, we can consider the denoising procedure as a multi-step MDP and apply a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
 \begin{eqnarray}
 \begin{eqnarray}
 s_t=(\mathbf{z},\mathbf{x}_{T-t}), \quad a_t=\mathbf{x}_{T-t-1}
 s_t=(\mathbf{z},\mathbf{x}_{T-t}), \quad a_t=\mathbf{x}_{T-t-1}
 \end{eqnarray}
 \end{eqnarray}
@@ -102,7 +102,7 @@ where $S_{z}$ denotes the training set, $\tau_\mathrm{dm}$ denotes the denoising
 
 
 \subsubsection{Flow Matching Models}
 \subsubsection{Flow Matching Models}
 
 
-Different from diffusion models, flow matching models achieve the multimodal generation process based on ordinary differential equations (ODEs), where data transformation is modeled as a continuous-time flow. Instead of gradually adding and removing noise through a stochastic diffusion process, flow matching learns a continuous vector field that transports samples from a simple prior distribution to the target data distribution. In this subsection, we briefly introduce the training and generation processes of flow matching models, and then discuss how RL can be applied to optimize them. Note that we do not provide a detailed introduction to the underlying principles of flow matching models in this subsection. Interested readers can refer to existing tutorials for further details \citep{xiao-etal:ordinary}.
+Different from diffusion models, flow matching models perform multimodal generation based on ordinary differential equations (ODEs), where data transformation is modeled as a continuous-time flow. Instead of gradually adding and removing noise through a stochastic diffusion process, flow matching learns a continuous vector field that transports samples from a simple prior distribution to the target data distribution. In this subsection, we briefly introduce the training and generation processes of flow matching models, and then discuss how RL can be applied to optimize them. Note that we do not provide a detailed introduction to the underlying principles of flow matching models in this subsection. Interested readers can refer to existing tutorials for further details \citep{xiao-etal:ordinary}.
 
 
 Let $\mathbf{x}_0 \sim X_0$ denote a data sample from the target distribution and $\mathbf{x}_1 \sim X_1$ denote a noise sample from the prior distribution. Flow matching constructs an intermediate state by interpolating between the data and noise distributions:
 Let $\mathbf{x}_0 \sim X_0$ denote a data sample from the target distribution and $\mathbf{x}_1 \sim X_1$ denote a noise sample from the prior distribution. Flow matching constructs an intermediate state by interpolating between the data and noise distributions:
 \begin{eqnarray}
 \begin{eqnarray}
@@ -137,7 +137,7 @@ a_t=\mathbf{x}_{t-\Delta t}
 
 
 Since the flow model deterministically predicts the velocity field, the policy can be represented as:
 Since the flow model deterministically predicts the velocity field, the policy can be represented as:
 \begin{eqnarray}
 \begin{eqnarray}
-\mathrm{Pr}_{\theta}(a_t|s_t) = \delta(a_t-\mathbf{v}_{\theta}(\mathbf{x}_t,t,\mathbf{z}))
+\mathrm{Pr}_{\theta}(a_t|s_t) = \delta(a_t-(\mathbf{x}_t-\Delta t\,\mathbf{v}_{\theta}(\mathbf{x}_t,t,\mathbf{z})))
 \end{eqnarray}
 \end{eqnarray}
 
 
 where $\delta(\cdot)$ denotes the Dirac delta distribution, indicating that the next state is deterministically determined by the current state and the learned velocity field.
 where $\delta(\cdot)$ denotes the Dirac delta distribution, indicating that the next state is deterministically determined by the current state and the learned velocity field.