update.

a4caf07d · wangchenglong · 6f2e0a1f · a4caf07d · a4caf07d · a4caf07d
Commit a4caf07d authored Sep 06, 2026 by wangchenglong
--- a/appendix/appendix.tex
+++ b/appendix/appendix.tex
 \clearpage
-\section*{Appendix: Useful Systems and Datasets}
+\section*{Appendix: Datasets and Systems}
 \phantomsection
-\addcontentsline{toc}{section}{Appendix: Useful Systems and Datasets}
+\addcontentsline{toc}{section}{Appendix: Datasets and Systems}
 \begin{table}[h]

--- a/rl-introduction.bbl
+++ b/rl-introduction.bbl
-\begin{thebibliography}{186}
+\begin{thebibliography}{188}
 \providecommand{\natexlab}[1]{#1}
 \providecommand{\url}[1]{\texttt{#1}}
 \expandafter\ifx\csname urlstyle\endcsname\relax
@@ -174,6 +174,11 @@ Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.
 \newblock Kto: Model alignment as prospect theoretic optimization, 2024.
 \newblock URL \url{https://arxiv.org/abs/2402.01306}.
+\bibitem[Fan et~al.(2018)Fan, Lewis, and Dauphin]{fan-etal:hierarchical}
+Angela Fan, Mike Lewis, and Yann Dauphin.
+\newblock Hierarchical neural story generation.
+\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\  889--898, 2018.
 \bibitem[Fan et~al.(2023)Fan, Watkins, Du, Liu, Ryu, Boutilier, Abbeel, Ghavamzadeh, Lee, and Lee]{fan-etal:dpok}
 Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.
 \newblock Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.
@@ -240,6 +245,11 @@ Alex Havrilla, Yuqing Du, Sharath~Chandra Raparthy, Christoforos Nalmpantis, Jan
 \newblock Teaching large language models to reason with reinforcement learning, 2024.
 \newblock URL \url{https://arxiv.org/abs/2403.04642}.
+\bibitem[Holtzman et~al.(2019)Holtzman, Buys, Du, Forbes, and Choi]{holtzman-etal:curious}
+Ari Holtzman, Jan Buys, Li~Du, Maxwell Forbes, and Yejin Choi.
+\newblock The curious case of neural text degeneration.
+\newblock \emph{arXiv preprint arXiv:1904.09751}, 2019.
 \bibitem[Hong et~al.(2024)Hong, Lee, and Thorne]{hong:2024orpo}
 Jiwoo Hong, Noah Lee, and James Thorne.
 \newblock {ORPO}: Monolithic preference optimization without reference model.

--- a/rl-introduction.bib
+++ b/rl-introduction.bib
+@article{holtzman-etal:curious,
+  title={The curious case of neural text degeneration},
+  author={Holtzman, Ari and Buys, Jan and Du, Li and Forbes, Maxwell and Choi, Yejin},
+  journal={arXiv preprint arXiv:1904.09751},
+  year={2019}
+}
+@inproceedings{fan-etal:hierarchical,
+  title={Hierarchical neural story generation},
+  author={Fan, Angela and Lewis, Mike and Dauphin, Yann},
+  booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
+  pages={889--898},
+  year={2018}
+}
 @inproceedings{hashemi-etal:llm-rubric,
    address = {Bangkok, Thailand},
    author = {Hashemi, Helia  and

--- a/rl-introduction.tex
+++ b/rl-introduction.tex
@@ -112,7 +112,7 @@
 \hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}
-\title{Reinforcement Learning without Tears:\\ An Introduction in the Era of Large Language Models}
+\title{RL without Tears:\\ An Introduction in the Era of LLMs}
 % Authors must not appear in the submitted version. They should be hidden
 % as long as the \colmfinalcopy macro remains commented out below.
@@ -170,6 +170,7 @@
 \newcommand{\new}{\marginpar{NEW}}
 \begin{document}
+\begin{CJK*}{UTF8}{gbsn}
 \ifcolmsubmission
 \linenumbers
@@ -179,7 +180,7 @@
 \begin{abstract}
-Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs). It is widely used to align models with human preferences and improve their reasoning capabilities. More recently, with the rapid rise of LLM-based agents, RL has become increasingly important for enabling models to learn from experience and build more capable and adaptive intelligent systems. Despite its growing importance, RL remains difficult for many LLM researchers to approach. Traditional RL literature often explains key concepts through robotics and control problems, making them less intuitive in the context of LLMs. To bridge this gap, this paper provides a comprehensive introduction to RL in the era of LLMs. We first introduce the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models. We show how RL can enhance reasoning, train agents through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
+Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs). It is widely used to align models with human preferences and improve their reasoning capabilities. More recently, with the rapid rise of LLM-based agents, RL has become increasingly important for enabling models to learn from experience and build more capable and adaptive intelligent systems. Despite its growing importance, RL remains difficult for many LLM researchers to approach. Traditional RL literature often explains key concepts through robotics and control problems, making them less intuitive in the context of LLM training. To bridge this gap, this paper provides a comprehensive introduction to RL through LLM training examples. We first introduce the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models. We show how RL can enhance reasoning, train agents through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
 \end{abstract}
@@ -206,11 +207,11 @@ This hypothesis highlights why RL is particularly relevant to the next stage of 
-Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in the era of LLMs, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
+Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLM training seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in LLM training, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
 It is natural to learn RL from standard references, which appears straightforward at first. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
-The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM research. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through a concrete example of training an LLM. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
+The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLM training. We begin by introducing the basic concepts and algorithms of RL using terminology familiar to LLM researchers, and illustrate them through concrete LLM training examples. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, training efficiency, and policy optimization. Beyond standard LLM alignment, we further introduce how RL is applied to enhance reasoning capabilities, support long-horizon decision-making in LLM-based agents, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and provide an overview of commonly used systems and datasets.
@@ -259,4 +260,5 @@ We would like to thank those who provided suggestions on improving the early ver
 % \appendix
+\end{CJK*}
 \end{document}
--- a/section2/section2.tex
+++ b/section2/section2.tex
--- a/section2/tables/rl-elements-and-llm-counterparts.tex
+++ b/section2/tables/rl-elements-and-llm-counterparts.tex
 \begin{tabular}{ll}
 \toprule[1.1pt]
-\multicolumn{1}{c}{\textbf{Element in Reinforcement Learning}} &  \multicolumn{1}{c}{\textbf{Counterpart in LLMs}} \\  \midrule
+\multicolumn{1}{c}{\textbf{Element in RL}} &  \multicolumn{1}{c}{\textbf{Counterpart in LLMs}} \\  \midrule
 \parbox{4cm}{
 Action ($a$)
 } & \parbox{7.2cm}{

--- a/section3/section3.tex
+++ b/section3/section3.tex
 % An example of REINFORCE and optimization objective
-\section{A Simple Example}
+\section{Understanding RL in LLM Training}
 \label{sec:example-using-rl-training-llms}
-We continue with a practical example of using an LLM as a homework assistant. In this context, we aim to improve the capability of an SFT LLM to handle education-related inputs more effectively. Suppose we have a homework assistant powered by an SFT LLM, and a student types the input ``Give me three tips to improve my accuracy in solving math problems.'' A typical SFT LLM might generate a short output, such as
+We continue with a practical example of using an LLM as a homework assistant. In this context, we want to improve the capability of an SFT LLM to handle education-related inputs more effectively. Suppose we have a homework assistant powered by an SFT LLM, and a student types the input ``Give me three tips to improve my accuracy in solving math problems.'' A  SFT LLM typically generates a short output, such as
 \vspace{0.1cm}
 \begin{tcolorbox}[frame empty]
@@ -23,7 +23,7 @@ There are three tips for improving accuracy in solving math problems: \\ [1mm]
 Although this output adheres to the given input, it is very short and not sufficiently detailed or comprehensive for this scenario. In practice, the ``short'' feature in the generated output stems from the training approach of the SFT LLM. During SFT, the LLM learns from a large set of labeled samples that typically contain concise and factual answers to common inputs. These samples guide the LLM to prioritize brevity and clarity, so it often generates short outputs, like the one shown above. Of course, we could annotate enough additional data to fine-tune the LLM further and adjust it to generate the desired outputs. However, this approach is limited in its ability to scale. For example, in this scenario, the input involves a variety of potential answers, and the expectations of the student can be pretty diverse. Therefore, describing the ``ideal'' output would require immense annotation effort. Consequently, collecting or annotating fine-tuning data is not as straightforward as it is with SFT, particularly when attempting to cover the breadth of possible student inputs and outputs. Instead, we can use RL to enable the model to discern outputs that better align with human preferences, such as generating more detailed and comprehensive content that not only adheres to the given input but also meets the expectations of the student in this scenario. 
-In the following sections, we will demonstrate how to use RL to train LLMs through a specific example, enabling the LLM to generate a longer output in the context of a homework assistant. Throughout this example, we introduce RL, including key algorithms and their improvements.
+In the following sections, we use a specific LLM training example to explain how RL works, where the LLM is trained to generate a longer output in the context of a homework assistant. Throughout this example, we introduce key RL algorithms and their improvements.
 % policy gradient
 \subsection{Policy Gradient}

--- a/section4/section4.tex
+++ b/section4/section4.tex
-\section{Improved Reinforcement Learning for LLMs}
+\section{Improved RL for LLMs}
 \vspace{-2mm}
 In the previous section, we introduced some improvements to RL, such as importance sampling and reward baseline techniques. However, directly applying them to train LLMs still presents numerous challenges. In this section, we will delve deeper into the improvements for using RL to train LLMs.
@@ -531,8 +531,8 @@ Moreover, there are other methods like GRPO that design advantage estimation wit
-\subsection{Efficient Reinforcement Learning Methods}
+\subsection{Efficient RL Methods}
-Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in the context of LLMs due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.
+Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in LLM training due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.
 In this section, we will not discuss all the issues related to the efficiency of RL, which is an extensive topic. Instead, we will focus on the commonly used efficient methods for training LLMs with RL. Some of these methods refine the sampling process, while others aim to eliminate certain components, such as the reward model, and utilize alternative lightweight methods in their place. Nonetheless, although these efficient methods are suggested for training LLMs, they are rather general and can be utilized in other RL scenarios.
@@ -542,7 +542,7 @@ As mentioned in Section \ref{sec:policy-gradient}, training LLMs with RL typical
 From the perspective of LLM inference, there are many methods to reduce the time overhead of sampling: since the sampling process is implemented by LLM inference, we have reason to believe that any method that accelerates inference can also be applied to reduce the time required for sampling. 
 Examples include KV-Cache \citep{pope-etal:2023efficiently}, quantization \citep{zhao-etal:2024atom}, and speculative decoding \citep{chen-etal:2023accelerating}. However, in this section, we will not discuss these methods in detail, as there are numerous approaches, each addressing different aspects of LLM inference. We refer the interested readers to these papers for more details. Instead, we will focus on one particular method that aims to reduce sampling overhead from the perspective of optimizing RL for training LLMs.
-Before discussing specific methods, let us first review the purpose of sampling. Given a sample $\mathbf{x}$, the goal of sampling is to explore a better output $\mathbf{y}$ from the policy model. This process allows us to evaluate different possible outputs in order to find those that lead to improved outcomes, enabling the model to make more informed decisions and optimize its performance on the state. Unlike typical RL scenarios, where policy models may start with limited information or random initialization, the policy model in the context of LLMs is usually quite advanced. These LLMs often begin as pre-trained models equipped with substantial knowledge acquired through pre-training and SFT. As a result, for certain samples, we can consider that the policy model may already perform well without the need for further exploration and optimization. This can be viewed through the lens of the classic RL dilemma of exploration versus exploitation \citep{Sutton-and-Barto:2018RL}. In such cases, exploration may be less necessary, and exploiting the knowledge already embedded in the pre-trained model could be more effective. Thus, a balance must be struck between exploring new possibilities and leveraging the pre-existing knowledge of the policy model to maximize efficiency. 
+Before discussing specific methods, let us first review the purpose of sampling. Given a sample $\mathbf{x}$, the goal of sampling is to explore a better output $\mathbf{y}$ from the policy model. This process allows us to evaluate different possible outputs in order to find those that lead to improved outcomes, enabling the model to make more informed decisions and optimize its performance on the state. Unlike typical RL scenarios, where policy models may start with limited information or random initialization, the policy model in LLM training is usually quite advanced. These LLMs often begin as pre-trained models equipped with substantial knowledge acquired through pre-training and SFT. As a result, for certain samples, we can consider that the policy model may already perform well without the need for further exploration and optimization. This can be viewed through the lens of the classic RL dilemma of exploration versus exploitation \citep{Sutton-and-Barto:2018RL}. In such cases, exploration may be less necessary, and exploiting the knowledge already embedded in the pre-trained model could be more effective. Thus, a balance must be struck between exploring new possibilities and leveraging the pre-existing knowledge of the policy model to maximize efficiency. 
 To achieve this balance between exploration and exploitation, one simple and straightforward approach to improving the efficiency of sampling is to identify the samples that are required to explore and perform sampling only for those samples. Given an input-only dataset $\mathcal{S}_x = \{\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N\}$, where $N$ denotes the size of the dataset, we can use a reward model to evaluate the need for exploration by the policy model. First, we generate $\hat{\mathbf{y}}$ for each $\mathbf{x}$ using greedy search. Then, by taking $\mathbf{x}$ and $\hat{\mathbf{y}}$ as input to the reward model, the reward for the $k$-th sample $\mathbf{x}_k$ is computed as $R_{\phi}(\mathbf{x}_{k}, \hat{\mathbf{y}}_{k})$.

--- a/section5/Figures/figure-iterative-rl.tex
+++ b/section5/Figures/figure-iterative-rl.tex
@@ -14,9 +14,9 @@
    \begin{scope}
      \node [anchor=center, block] (b1) at (0,0) {Pre-training an LLM};
      \node [anchor=south, block] (b2) at ([yshift=\sep]b1.north) {\footnotesize{Cold Start} \\ \footnotesize{for Reasoning}};
-      \node [anchor=south, block] (b3) at ([yshift=\sep]b2.north) {\footnotesize{Large-scale Reinforcement}\\ \footnotesize{Learning for Reasoning}};
+      \node [anchor=south, block] (b3) at ([yshift=\sep]b2.north) {\footnotesize{Large-scale RL}\\ \footnotesize{for Reasoning}};
      \node [anchor=south, block] (b4) at ([yshift=\sep]b3.north) {\footnotesize{Rejection Sampling} \\ \footnotesize{for Multi-task}};
-      \node [anchor=south, block] (b5) at ([yshift=\sep]b4.north) {\footnotesize{Reinforcement Learning} \\ \footnotesize{for Multi-task}};
+      \node [anchor=south, block] (b5) at ([yshift=\sep]b4.north) {\footnotesize{RL} \\ \footnotesize{for Multi-task}};
      % \node [anchor=south] (b6) at ([yshift=\sep]b5.north) {\LARGE{$\cdots$}};
      \draw [->] (b1.north) -- (b2.south);
      \draw [->] (b2.north) -- (b3.south);

--- a/section5/section5.tex
+++ b/section5/section5.tex
-\section{Reinforcement Learning for LLM Reasoning}
+\section{RL for LLM Reasoning}
 So far, our discussion has mainly focused on various aspects of using and improving RL for training LLMs. The methods mentioned can be easily adapted to a wide range of scenarios where the correctness of an output can be examined by checking whether the desired result is included. For example, in the task of calculating a mathematical expression, a reward model can provide positive feedback if the answer is correct and negative feedback if the answer is wrong. However, in many problems that require complex reasoning, simply examining the correctness of the final answer is insufficient for learning. Imagine a student who is only given the final answer to a challenging math problem. Knowing whether the final answer is right or wrong does not help the student figure out where they went wrong and how to calculate the correct answer. A better approach would be to guide the student with a step-by-step breakdown of the problem-solving process and encourage understanding of the underlying concepts and logic behind these steps. To address this, researchers also explore the potential of RL to teach LLMs not just to generate correct answers but to develop and present coherent, step-by-step reasoning that aligns with human cognitive processes. In this regard, RL has previously demonstrated its effectiveness in training neural networks for complex planning and reasoning within game environments, as evidenced by notable successes such as AlphaGo \citep{silver-etal:2016mastering} and AlphaStar \citep{vinyals-rtal:2019grandmaster}. Given these advancements and the inherently interactive nature of problem-solving, it is also natural to consider the application of RL to LLM reasoning. 
 In this section, we delve deeper into the application of RL to enhance the reasoning capabilities of the LLM, a topic that has recently garnered significant attention. We begin by giving a general introduction to test-time scaling that fully unleashes the reasoning potential of LLMs, including best-of-$N$ sampling, step-by-step verification, and Monte Carlo Tree Search. We then discuss two critical issues: how to scale RL effectively, and how to iterate the RL process to enhance the reasoning capabilities of LLMs. Note that the following methods are primarily illustrated through mathematical reasoning problems, but they are applicable to a broad range of decision-making problems.
@@ -92,7 +95,7 @@ In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popu
    \item \textbf{Backpropagation}. The process updates the UCT values of prior nodes using the results of these simulations. Key updates include the visitation counts, $N_p$ and $N_{\bar{\mathbf{y}}_{k}}$ in Eq. (\ref{eq:uct}), reflecting how frequently each node has been explored. Additionally, this stage allows for the incorporation of delayed rewards to adjust the $\bar{R}(\bar{\mathbf{y}}_k)$ values.
 \end{itemize}
-\subsection{Iterative Reinforcement Learning}
+\subsection{Iterative RL}
 Training LLMs with RL usually follows a two-phase approach: training a pre-trained LLM with SFT and further training with an RL algorithm applied to the SFT LLM. However, using large-scale RL in such a two-phase approach to train LLMs in reasoning may lead to significant knowledge forgetting as the model continuously adjusts to fit the reward model. For example, as mentioned in DeepSeek-R1 \citep{guo:2025deepseek}, directly applying large-scale RL can achieve the desired reasoning outcomes, but often at the cost of reduced readability in the reasoning process. To address these challenges, we can utilize an iterative RL approach to continuously enhance the various capabilities of the LLM. Here we consider DeepSeek-R1 as an example to illustrate how to perform an iterative RL. The idea is to split the RL process into multiple phases, each designed to enhance different capabilities using varied rewards, to develop a robust reasoning model that is able to generate clearer and more comprehensible reasoning paths. Figure \ref{fig:iterative-rl} provides a schematic illustration of the iterative RL process. Here we give a brief outline of each phase involved.
@@ -117,7 +120,7 @@ An interesting issue arises with this design of iterative RL: why is RL aimed at
 Since the optimization objective of each phase is different, the design of iterative RL can also be analyzed and understood from the perspective of multi-objective optimization \citep{wang-etal:2024hybrid}. In the context of multi-objective optimization, iterative RL for LLMs can be likened to interactive methods where the solution process is iterative, and preferences are actively defined and refined by the decision-maker during the search for the most preferred solutions \citep{miettinen-etal:2008introduction,deb-etal:2016multi}. More specifically, in this scenario, RL can be seen as a decision-maker, iteratively refining and enhancing the different capabilities of the LLM across different phases.
-\subsection{Large-scale Reinforcement Learning}
+\subsection{Large-scale RL}
 \label{sec:large-scale-rl}
 While test-time scaling is instrumental in exploring more effective reasoning paths, its potential is restricted if the model has not learned to generate high-quality reasoning paths. In other words, in practice, test-time scaling struggles to achieve desired outcomes without foundational reasoning ability, no matter how extensive it is \citep{yuan-etal:2023scaling,zeng-etal:2025revisiting}. Consequently, there has been a growing research interest in training LLMs specifically to develop reasoning abilities that mimic human-like processes. A straightforward approach is to use annotated reasoning data to train the LLM through SFT. However, a significant challenge in this approach is the high cost of annotations, especially since annotating detailed step-by-step reasoning paths is significantly more cost-intensive than annotating SFT data in other tasks like machine translation and summarization.

--- a/section6/section6.tex
+++ b/section6/section6.tex
-\section{Agentic Reinforcement Learning}
+\section{Agentic RL}
 Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Atari and Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
@@ -78,7 +78,7 @@ Beyond selecting high-quality trajectories, another challenge lies in effectivel
 Furthermore, even with effective trajectory selection and representation, scaling the generation of diverse and high-quality trajectory data remains challenging. Existing approaches often rely on manually designed environments and planning tasks, which limits the diversity and coverage of collected trajectories. Designing new environments requires substantial human expertise, while constructing tasks with appropriate difficulty levels remains difficult. To address this limitation, AGENTGEN explores automatically generating diverse environments and planning tasks with LLMs, enabling large-scale synthesis of trajectory data with varying task complexity for agent training \citep{hu-etal:agentgen}.
-\subsubsubsection{Learning to Plan with Reinforcement Learning}
+\subsubsubsection{Learning to Plan with RL}
 Although SFT can provide a useful cold start for agent planning, it mainly teaches the model to imitate offline demonstrations. This limits its ability to improve beyond the quality and coverage of the collected trajectories. In contrast, RL further optimizes planning through direct interaction with the environment, where the agent receives feedback based on the outcome of its generated plans and executed behaviors. 
 % This makes RL particularly suitable for improving long-horizon planning, because the quality of a plan is often determined by the success of the entire interaction trajectory rather than by the correctness of any single intermediate step.
@@ -170,7 +170,7 @@ A representative approach to addressing this data construction problem is Toolfo
 Beyond learning to use a small set of tools, later studies further scale tool-use training to larger and more realistic API spaces. API-Bank builds a benchmark and training set for tool-augmented LLMs, where models need to plan, retrieve, and call APIs in executable environments \citep{li-etal:apibank}. ToolLLM constructs ToolBench with more than 16,000 real-world APIs and uses automatically generated instruction-solution trajectories to train ToolLLaMA \citep{qin-etal:toolllm}. Gorilla focuses on API calling at scale and fine-tunes LLaMA-based models to generate accurate API calls. It also uses a document retriever to reduce API hallucination and adapt to changing API documents \citep{patil-etal:gorilla}.
 % 4. RL for Tool Use
-\subsubsubsection{Learning to Use Tools with Reinforcement Learning}
+\subsubsubsection{Learning to Use Tools with RL}
 Although SFT can teach LLMs to imitate tool-use demonstrations, it is still limited by the coverage and quality of offline trajectories. The model mainly learns the tool-use patterns that appear in the supervised data. As a result, it may fail to generalize to unfamiliar tools, complex tool combinations, or new interaction patterns. More importantly, SFT does not directly optimize whether a tool call is necessary, whether the selected tool is appropriate, or whether the returned observation improves the final answer. This limitation becomes more serious in multi-step tool-use scenarios, \textit{where the model needs to decide when to call a tool, how to formulate the tool input, how to interpret the tool output, and when to stop using tools}.

--- a/section7/section7.tex
+++ b/section7/section7.tex
-\section{Multimodal Reinforcement Learning}
+\section{Multimodal RL}
 Multimodal learning aims to build models that can process and integrate information from multiple modalities, such as vision and text. By combining complementary signals from different modalities, multimodal models can achieve more comprehensive perception and reasoning abilities than single-modality systems. These capabilities have enabled a wide range of applications, including image captioning, visual reasoning, speech understanding, and image generation \citep{baltruvsaitis-etal:2018multimodal}.

--- a/section8/section8.tex
+++ b/section8/section8.tex
 \section{Conclusions and Future Directions}
-In this paper, we have introduced the fundamental concepts of RL from the perspective of LLM research. We began with basic RL formulations and algorithms, including policy gradients, advantage estimation, importance sampling, and reward modeling, and then discussed recent advances in RL. We further extended the discussion to reasoning models, agentic systems, and multimodal models, showing how RL has evolved from a general policy optimization framework into a key paradigm for enhancing LLM reasoning, interactive decision-making, and multimodal generation capabilities.
+In this paper, we have introduced the fundamental concepts of RL from the perspective of LLM training. We began with basic RL formulations and algorithms, including policy gradients, advantage estimation, importance sampling, and reward modeling, and explained them through LLM training examples. We then discussed recent advances in RL. We further extended the discussion to reasoning models, agentic systems, and multimodal models, showing how RL has evolved from a general policy optimization framework into a key paradigm for enhancing LLM reasoning, interactive decision-making, and multimodal generation capabilities.
 While RL has become well established in LLM training and has achieved remarkable results, several promising directions remain for future exploration: