# RL without Tears: An Introduction in the Era of LLMs
Reinforcement learning (RL) has become an important training paradigm for large language models (LLMs). It is widely used to align models with human preferences, improve reasoning capabilities, and enable agentic systems to learn from interaction and feedback. At its core, RL studies how an agent improves its behavior by interacting with an environment and receiving reward signals. A standard objective can be written as:
Reinforcement learning (RL) has become an important training paradigm for large language models (LLMs). It is widely used for preference alignment, reasoning, and agentic learning from interaction and feedback. At its core, RL studies how an agent improves its behavior by interacting with an environment and receiving rewards. A standard objective is
```math
\max_{\pi} \; \mathbb{E}_{\tau \sim \pi}
\left[
\sum_{t=0}^{T} r_t
\right]
````
```
where $\pi$ denotes the policy, $\tau$ denotes a trajectory, and $r_t$ is the reward received at step $t$.
For LLMs, this formulation has a natural interpretation. The LLM acts as the agent. Its next-token distribution defines the policy. Generated tokens correspond to actions, and the complete response forms a trajectory. Reward signals are then used to evaluate the quality of the generated output.
where $\pi$ denotes the policy, $\tau$ denotes a trajectory of interactions, and $r_t$ is the reward received at step $t$.
Although RL is becoming increasingly important for LLMs, many of its concepts remain unfamiliar to researchers with a background in NLP and supervised learning. Traditional RL literature often explains these concepts through robotics, control, or game-playing tasks. This makes it difficult to directly connect RL formulations with modern LLM training. This gap motivates **RL without Tears: An Introduction in the Era of LLMs**.
In the era of LLMs, this formulation naturally connects to language generation. An LLM can be viewed as a policy, generated tokens or responses can be treated as actions and trajectories, and reward signals can be used to evaluate whether an output is helpful, correct, safe, or aligned with human intent.
In this paper, we introduce RL from the perspective of LLM research. We explain key concepts and algorithms through LLM-oriented examples, including policy gradients, advantage estimation, importance sampling, PPO, and reward modeling. We then discuss recent advances in RL for LLMs. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models.
In **RL without Tears: An Introduction in the Era of LLMs**, we introduce RL from the perspective of LLM research and training. We begin with the fundamental concepts of RL and explain key algorithms through concrete LLM-oriented examples, including policy gradients, advantage estimation, importance sampling, proximal policy optimization, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward modeling, advantage estimation, efficient training, and policy optimization. Beyond standard LLM training, we further extend the discussion to reasoning models, LLM-based agents, and multimodal models, showing how RL can enhance reasoning, support learning through environment interaction, and optimize multimodal understanding and generation.
We aim to make RL easier to understand for the LLM community. Rather than relying on traditional control examples, we use terminology and examples that are familiar to LLM researchers. We hope this paper provides an accessible introduction to RL and helps readers better understand and apply RL techniques in modern foundation models.
This paper aims to balance technical depth with accessibility. It is intended for readers who are familiar with LLMs but may find it difficult to connect traditional RL literature with modern language model training. Rather than introducing RL primarily through robotics or control problems, we explain its key concepts and algorithms using terminology and examples that are familiar to the LLM community. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models.
