Reinforcement learning (RL) has become a powerful and versatile paradigm for developing large language models (LLMs), such as aligning models with human preferences and improving reasoning capabilities. More recently, as modern AI increasingly shifts toward learning from experience, RL has attracted renewed attention as a promising path toward more capable and adaptive intelligent systems, enabling LLM-based agents to continuously improve their decision-making through environment interaction rather than relying solely on static supervision. However, RL introduces many concepts and algorithms that may be unfamiliar to LLM researchers, while traditional RL literature often explains them in the context of robotics and control. This paper provides a comprehensive introduction to RL from the perspective of LLM research. We begin with the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. Beyond standard LLM training, we further introduce how RL is used to enhance reasoning, train LLM-based agents through environment interaction, and optimize multimodal understanding and generation models. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
\end{abstract}
\end{abstract}
\vfill
\noindent{\footnotesize
\textcopyright 2026 Wang, Zhou, Xiao, Zhu, and Liu.\\
License: CC-BY 4.0, see \url{https://creativecommons.org/licenses/by/4.0/}.
}
\clearpage
\clearpage
\tableofcontents
\tableofcontents
...
@@ -166,9 +174,9 @@ Beijing, China \\
...
@@ -166,9 +174,9 @@ Beijing, China \\
\clearpage
\clearpage
\section{Introduction}
\section{Introduction}
Reinforcement learning (RL) is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles.
RL is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles.
A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of large language models (LLMs). For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1,kimi-team:kimi}. As a result, interest in NLP has exploded. There have been many, many conference papers discussing RL in LLMs, and even more in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI. This trend is further strengthened by the rise of LLM-based agents, where models must learn from interaction and experience to make effective decisions in dynamic environments. From this perspective, RL is becoming a key technique for building more capable and adaptive intelligent systems. This view closely echoes the ``Reward Is Enough'' hypothesis proposed by \citet{silver-etal:reward}:
A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of LLMs. For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1,kimi-team:kimi}. As a result, interest in NLP has exploded. There have been many, many conference papers discussing RL in LLMs, and even more in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI. This trend is further strengthened by the rise of LLM-based agents, where models must learn from interaction and experience to make effective decisions in dynamic environments. From this perspective, RL is becoming a key technique for building more capable and adaptive intelligent systems. This view closely echoes the ``Reward Is Enough'' hypothesis proposed by \citet{silver-etal:reward}:
\begin{quote}
\begin{quote}
``Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.''
``Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.''
