Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs), with applications ranging from aligning models with human preferences to improving their reasoning capabilities. More recently, as modern AI increasingly shifts toward learning from experience, RL has attracted renewed attention as a path toward more capable and adaptive intelligent systems, enabling LLM-based agents to continuously improve their decision-making through environment interaction rather than relying solely on static supervision. However, RL introduces many concepts and algorithms that can be difficult to understand in the context of LLMs, while traditional RL literature often introduces them through robotics or control problems. To this end, this paper provides a comprehensive introduction to RL in the era of LLMs. We begin with the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models, showing how RL can enhance reasoning, support learning through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper
Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs), with applications ranging from aligning models with human preferences to improving their reasoning capabilities. More recently, as modern AI increasingly shifts toward learning from experience, RL has attracted renewed attention as a path toward more capable and adaptive intelligent systems, enabling LLM-based agents to continuously improve their decision-making through environment interaction. Despite its growing importance in LLM development, RL remains difficult for many LLM researchers to approach. Traditional RL literature often explains its concepts through robotics and control problems, making them less intuitive in the context of LLMs. To this end, this paper provides a comprehensive introduction to RL in the era of LLMs. We begin with the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models, showing how RL can enhance reasoning, support learning through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in the era of LLMs.
provides an accessible introduction to RL and helps LLM researchers better understand
@@ -28,7 +28,7 @@ In the following sections, we will demonstrate how to use RL to train LLMs throu
...
@@ -28,7 +28,7 @@ In the following sections, we will demonstrate how to use RL to train LLMs throu
% policy gradient
% policy gradient
\subsection{Policy Gradient}
\subsection{Policy Gradient}
\label{sec:policy-gradient}
\label{sec:policy-gradient}
In this section, we aim to lengthen the LLM output for the ``give me three tips'' input using a classical RL algorithm called policy gradient.
In this section, we aim to lengthen the LLM output for the ``give me three tips'' input using a classical RL algorithm called \textbf{policy gradient}.
To define our optimization objective, let $R(\cdot)$ be the reward function that describes the goal the algorithm aims to optimize. In this scenario, the reward is designed to align the output of the LLM with a specific behaviour, that is, generating longer outputs. Therefore, we define the reward function based on a length-related metric. More specifically, the reward can be proportional to the length of the output, that is, longer outputs can receive higher rewards:
To define our optimization objective, let $R(\cdot)$ be the reward function that describes the goal the algorithm aims to optimize. In this scenario, the reward is designed to align the output of the LLM with a specific behaviour, that is, generating longer outputs. Therefore, we define the reward function based on a length-related metric. More specifically, the reward can be proportional to the length of the output, that is, longer outputs can receive higher rewards:
