\title{Reinforcement Learning without Tears:\\ An Introduction for Large Language Model Researchers}
\title{Reinforcement Learning without Tears:\\ An Introduction in the Era of Large Language Models}
% Authors must not appear in the submitted version. They should be hidden
% Authors must not appear in the submitted version. They should be hidden
% as long as the \colmfinalcopy macro remains commented out below.
% as long as the \colmfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
% Non-anonymous submissions will be rejected without review.
\author{Chenglong Wang, Hang Zhou, Tong Xiao \& Jingbo Zhu \\
% \thanks{ Use footnote for providing further information about author (webpage, alternative address)---\emph{not} for acknowledging funding agencies. Funding acknowledgements go at the end of the paper.} \\
\emph{Institute of Psychology, Chinese Academy of Sciences, China}
\end{tabular}
&
{\small\textsc{LIUTR@PSYCH.AC.CN}}
\\[2em]
\begin{tabular}[t]{@{}l@{}}
\textbf{Jingbo Zhu}\\
\emph{Northeastern University, China}
\end{tabular}
&
{\small\textsc{ZHUJINGBO@MAIL.NEU.EDU.CN}}
\\[2em]
\begin{tabular}[t]{@{}l@{}}
\textbf{Tong Xiao}\\
\emph{Northeastern University, China}
\end{tabular}
&
{\small\textsc{XIAOTONG@MAIL.NEU.EDU.CN}}
\end{tabular*}
}
}
% The \author macro works with any number of authors. There are two commands
% The \author macro works with any number of authors. There are two commands
...
@@ -157,13 +179,15 @@ Beijing, China \\
...
@@ -157,13 +179,15 @@ Beijing, China \\
\begin{abstract}
\begin{abstract}
Reinforcement learning (RL) has become a powerful and versatile paradigm for developing large language models (LLMs), such as aligning models with human preferences and improving reasoning capabilities. More recently, as modern AI increasingly shifts toward learning from experience, RL has attracted renewed attention as a promising path toward more capable and adaptive intelligent systems, enabling LLM-based agents to continuously improve their decision-making through environment interaction rather than relying solely on static supervision. However, RL introduces many concepts and algorithms that may be unfamiliar to LLM researchers, while traditional RL literature often explains them in the context of robotics and control. This paper provides a comprehensive introduction to RL from the perspective of LLM research. We begin with the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. Beyond standard LLM training, we further introduce how RL is used to enhance reasoning, train LLM-based agents through environment interaction, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs), with applications ranging from aligning models with human preferences to improving their reasoning capabilities. More recently, as modern AI increasingly shifts toward learning from experience, RL has attracted renewed attention as a path toward more capable and adaptive intelligent systems, enabling LLM-based agents to continuously improve their decision-making through environment interaction rather than relying solely on static supervision. However, RL introduces many concepts and algorithms that can be difficult to understand in the context of LLMs, while traditional RL literature often introduces them through robotics or control problems. To this end, this paper provides a comprehensive introduction to RL in the era of LLMs. We begin with the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models, showing how RL can enhance reasoning, support learning through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper
provides an accessible introduction to RL and helps LLM researchers better understand
and apply RL techniques in the era of LLMs.
\end{abstract}
\end{abstract}
\vfill
\vfill
\noindent{\footnotesize
\noindent{\footnotesize
\textcopyright 2026 Wang, Zhou, Xiao, Zhu, and Liu.\\
\textcopyright 2026 Wang, Zhou, Liu, Zhu, and Xiao.\\
License: CC-BY 4.0, see \url{https://creativecommons.org/licenses/by/4.0/}.
License: CC-BY 4.0, see \url{https://creativecommons.org/licenses/by/4.0/}.
}
}
...
@@ -184,7 +208,7 @@ This hypothesis highlights why RL is particularly relevant to the next stage of
...
@@ -184,7 +208,7 @@ This hypothesis highlights why RL is particularly relevant to the next stage of
However, RL was not so popular in the long history of NLP, and the field has just begun to explore its potential. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard other researchers discuss advanced RL techniques, such as ``use a value function to estimate the expected cumulative reward'', ``learn a policy via PPO'', or ``we can use GRPO to solve this problem''. It seems that these techniques are so simple that presenters do not even need to give any explanation. But we were lost when seeing the math of RL, such as all those expectation signs, as well as the names of various algorithms that we are unfamiliar with.
Historically, RL has played a relatively limited role in mainstream NLP, where supervised learning has long been the dominant paradigm. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard statements such as ``use a value function to estimate the expected cumulative reward'', ``learn the policy with PPO'', or simply ``we need to train a reward model''. These techniques were often presented as if they required little explanation. Yet, for researchers trained primarily in supervised learning, the mathematical formulations of RL, with their many expectations, value functions, and advantages, can be difficult to follow. This gap becomes increasingly important in the era of LLMs, where understanding RL is no longer a specialized topic, but is becoming essential for following and developing modern AI systems.
Of course, we'd like to learn about RL, which appears straightforward. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
Of course, we'd like to learn about RL, which appears straightforward. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
@@ -477,6 +477,7 @@ By repeating the above process, the agent can improve its skill library through
...
@@ -477,6 +477,7 @@ By repeating the above process, the agent can improve its skill library through
\subsubsection{Trajectory Refinement}
\subsubsection{Trajectory Refinement}
\vspace{-4mm}
Different from memory management and skill optimization, which focus on storing and abstracting experiences, trajectory refinement directly improves agent behaviors by revising generated trajectories. This idea is motivated by the self-refinement capability of LLMs. As discussed in Section~\ref{sec:tts}, reasoning-enhanced RL training enables LLMs to evaluate their intermediate reasoning processes and revise incorrect solutions. For example, when solving mathematical problems, an LLM can verify intermediate results after each reasoning step and adjust its solution trajectory based on the verification outcomes.
Different from memory management and skill optimization, which focus on storing and abstracting experiences, trajectory refinement directly improves agent behaviors by revising generated trajectories. This idea is motivated by the self-refinement capability of LLMs. As discussed in Section~\ref{sec:tts}, reasoning-enhanced RL training enables LLMs to evaluate their intermediate reasoning processes and revise incorrect solutions. For example, when solving mathematical problems, an LLM can verify intermediate results after each reasoning step and adjust its solution trajectory based on the verification outcomes.
