LLMs have become the foundation of modern natural language processing. Their success largely follows a simple but powerful paradigm: first learn general language capabilities from large-scale text corpora, and then adapt these capabilities to specific tasks through prompting or supervised fine-tuning. At inference time, an LLM generates an output by predicting tokens sequentially according to the given input and previously generated tokens. This paradigm has substantially changed how NLP systems are developed, replacing many task-specific models with a unified foundation model that can support a wide range of language understanding and generation tasks.
LLMs have become the foundation of modern natural language processing. Their success largely follows a simple but powerful paradigm: first learn general language capabilities from large-scale text corpora, and then adapt these capabilities to specific tasks through prompting or supervised fine-tuning. At inference time, an LLM generates an output by predicting tokens sequentially according to the given input and previously generated tokens. This paradigm has substantially changed how NLP systems are developed, replacing many task-specific models with a unified foundation model that can support a wide range of language understanding and generation tasks.
Understanding these basic mechanisms is important before introducing RL for LLMs. Many RL concepts can be naturally connected to the standard generation process of an LLM. The input and previously generated tokens define the current context, the next-token distribution determines the model's possible actions, and the complete output forms a sequence of decisions that can be evaluated by a reward signal. In this subsection, we briefly review the fundamentals of LLMs and introduce the key concepts and notations used throughout this paper. We first describe pre-training, through which LLMs acquire general language capabilities from large-scale data. We then introduce prompting, which enables a pre-trained model to perform different tasks through natural-language instructions without parameter updates. Next, we discuss supervised fine-tuning, which further adapts the model using labeled input-output pairs. Finally, we describe the inference process and explain how an LLM generates an output token by token. It is worth noting that we focus only on the concepts necessary for understanding the subsequent discussion of RL, rather than providing a comprehensive review of these techniques and their recent developments. Interested readers are referred to \citet{xiao-and-zhu:2025foundations} for a more systematic introduction to LLMs.
Understanding these basic mechanisms is important before introducing RL for LLMs. Many RL concepts can be naturally connected to the standard generation process of an LLM. The input and previously generated tokens define the current context, the next-token distribution determines the model's possible actions, and the complete output forms a sequence of decisions that can be evaluated by a reward signal. In this subsection, we briefly review the fundamentals of LLMs and introduce the key concepts and notations used throughout this paper. We focus only on the concepts necessary for understanding the subsequent discussion of RL, rather than providing a comprehensive review of LLM techniques and their recent developments. Interested readers are referred to \citet{xiao-and-zhu:2025foundations} for a more systematic introduction to LLMs.
In this paper, we mainly focus on generative LLMs based on decoder-only Transformers \citep{vaswani-etal:2017attention}. Let $\mathcal{V}$ denote the vocabulary of tokens. A token is the basic unit processed by an LLM. It can be a word, a subword, a punctuation mark, or another text fragment produced by the tokenizer. Before a raw text is input into an LLM, it is first converted into a sequence of tokens from $\mathcal{V}$. Therefore, throughout this paper, all inputs and outputs of an LLM are represented as token sequences. Given a token sequence $\mathbf{z}=z_1...z_N$, a language model parameterized by $\theta$ estimates the probability of the sequence by factorizing it from left to right:
In this paper, we mainly focus on generative LLMs based on decoder-only Transformers \citep{vaswani-etal:2017attention}. Let $\mathcal{V}$ denote the vocabulary of tokens. A token is the basic unit processed by an LLM. It can be a word, a subword, a punctuation mark, or another text fragment produced by the tokenizer. Before a raw text is input into an LLM, it is first converted into a sequence of tokens from $\mathcal{V}$. Therefore, throughout this paper, all inputs and outputs of an LLM are represented as token sequences. Given a token sequence $\mathbf{z}=z_1...z_N$, a language model parameterized by $\theta$ estimates the probability of the sequence by factorizing it from left to right:
\begin{eqnarray}
\begin{eqnarray}
...
@@ -14,8 +14,24 @@ In this paper, we mainly focus on generative LLMs based on decoder-only Transfor
...
@@ -14,8 +14,24 @@ In this paper, we mainly focus on generative LLMs based on decoder-only Transfor
where $\mathbf{z}_{<i}=z_1...z_{i-1}$ denotes the tokens before $z_i$. A decoder-only Transformer implements this conditional distribution by applying causal self-attention, so that the prediction at each position can only depend on the previous tokens. The model finally produces a probability distribution over $\mathcal{V}$, denoted by $\mathrm{Pr}_{\theta}(\cdot|\mathbf{z}_{<i})$. This next-token prediction view is the central interface through which LLMs are trained, prompted, fine-tuned, and later optimized by RL.
where $\mathbf{z}_{<i}=z_1...z_{i-1}$ denotes the tokens before $z_i$. A decoder-only Transformer implements this conditional distribution with causal self-attention, so that each position can only depend on previous tokens. The model then produces a probability distribution over $\mathcal{V}$, denoted by $\mathrm{Pr}_{\theta}(\cdot|\mathbf{z}_{<i})$. This next-token prediction view is the central interface through which LLMs are trained, prompted, fine-tuned, and later optimized by RL. Figure~\ref{fig:language-modeling-and-generation} illustrates this process together with prompt-based autoregressive generation. During training, the predicted distributions are matched to the observed target tokens. During inference, each selected token is appended to the context and used to condition the next prediction.
