\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{vaswani-etal:2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan~N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna~M. Wallach, Rob Fergus, S.~V.~N. Vishwanathan, and Roman Garnett (eds.), \emph{Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, {USA}}, pp.\ 5998--6008, 2017.
\subsection{Fundamentals of Large Language Models}
\subsection{Fundamentals of Large Language Models}
In this subsection, we introduce the fundamentals of LLMs, focusing on the key concepts and notations needed for the following discussions.
Large language models (LLMs) have become the foundation of modern natural language processing. Their success largely follows a simple but powerful paradigm: first learn general language capabilities from large-scale text corpora, and then adapt these capabilities to specific tasks through prompting or supervised fine-tuning. At inference time, an LLM generates an output by predicting tokens sequentially according to the given input and previously generated tokens. This paradigm has substantially changed how NLP systems are developed, replacing many task-specific models with a unified foundation model that can support a wide range of language understanding and generation tasks.
Understanding these basic mechanisms is important before introducing RL for LLMs. Many RL concepts can be naturally connected to the standard generation process of an LLM. The input and previously generated tokens define the current context, the next-token distribution determines the model's possible actions, and the complete output forms a sequence of decisions that can be evaluated by a reward signal. In this subsection, we briefly review the fundamentals of LLMs and introduce the key concepts and notations used throughout this paper. We first describe pre-training, through which LLMs acquire general language capabilities from large-scale data. We then introduce prompting, which enables a pre-trained model to perform different tasks through natural-language instructions without parameter updates. Next, we discuss supervised fine-tuning, which further adapts the model using labeled input-output pairs. Finally, we describe the inference process and explain how an LLM generates an output token by token. It is worth noting that we focus only on the concepts necessary for understanding the subsequent discussion of RL, rather than providing a comprehensive review of these techniques and their recent developments. Interested readers are referred to \citet{xiao-and-zhu:2025foundations} for a more systematic introduction to LLMs.
In this paper, we mainly focus on generative LLMs based on decoder-only Transformers \citep{vaswani-etal:2017attention}. Let $\mathcal{V}$ denote the vocabulary of tokens. A token is the basic unit processed by an LLM. It can be a word, a subword, a punctuation mark, or another text fragment produced by the tokenizer. Before a raw text is input into an LLM, it is first converted into a sequence of tokens from $\mathcal{V}$. Therefore, throughout this paper, all inputs and outputs of an LLM are represented as token sequences. Given a token sequence $\mathbf{z}=z_1...z_N$, a language model parameterized by $\theta$ estimates the probability of the sequence by factorizing it from left to right:
where $\mathbf{z}_{<i}=z_1...z_{i-1}$ denotes the tokens before $z_i$. A decoder-only Transformer implements this conditional distribution by applying causal self-attention, so that the prediction at each position can only depend on the previous tokens. The model finally produces a probability distribution over $\mathcal{V}$, denoted by $\mathrm{Pr}_{\theta}(\cdot|\mathbf{z}_{<i})$. This next-token prediction view is the central interface through which LLMs are trained, prompted, fine-tuned, and later optimized by RL.
\subsubsection{Pre-training}
Pre-training is the first stage of building most LLMs. The goal is to train the model on large-scale unlabeled text so that it can acquire general knowledge about languages and the world. In generative LLMs, this is usually achieved through self-supervised next-token prediction: the model observes the preceding tokens and learns to predict the next token. Although no human-annotated labels are required, each token in the text can naturally serve as a supervision signal for predicting itself from its left context.
Let $\mathcal{S}_{\mathrm{pre}}$ denote the pre-training corpus, where each sample is a token sequence $\mathbf{z}=z_1...z_N$. The pre-training objective is to maximize the log-likelihood of all tokens in the corpus:
where $\hat{\theta}$ denotes the optimized pre-trained parameters. Equivalently, this objective can be viewed as minimizing the cross-entropy loss between the observed next token and the model-predicted distribution. After this stage, the LLM can generate fluent text and encode rich linguistic and factual knowledge. However, the pre-training objective itself does not explicitly teach the model how to follow user instructions, satisfy task-specific formats, or align with human preferences. Therefore, additional adaptation is typically needed before deploying the model for real applications.
\subsubsection{Prompting}
Prompting is a simple and lightweight way to adapt an LLM to different tasks without updating its parameters. In the context of LLMs, a \textit{prompt} refers to the input text provided to guide the model toward a desired task or output. It may contain an instruction, user-provided content, output requirements, or demonstrations. For example, if we want an LLM to act as a homework assistant and answer a student's question, we can provide the following prompt:
\vspace{0.1cm}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{2em}
\setlength{\rightskip}{2em}
You are a homework assistant helping students improve their problem-solving skills.
Give me three tips to improve my accuracy in solving math problems.
Please make the response clear, practical, and easy to follow. \\[1mm]
\underline{\hspace{4cm}}
\endgroup
\end{tcolorbox}
\vspace{0.5em}
In this example, the prompt itself specifies the task, and the LLM generates the answer by continuing the sequence. A prompt can also be obtained from a prompt template. A prompt template is a piece of text containing placeholders, where each placeholder is filled with concrete information before being sent to the LLM. For example, we can use the following template to construct prompts for a homework assistant:
\vspace{0.1cm}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{2em}
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation.
\setlength{\rightskip}{2em}
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_1...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
You are a homework assistant helping students improve their problem-solving skills.
Give me three tips to improve my accuracy in solving \{*subject*\} problems.
Please make the response clear, practical, and easy to follow. \\[1mm]
\underline{\hspace{4cm}}
\endgroup
\end{tcolorbox}
\vspace{0.5em}
If we set \{*subject*\} to ``math'', the template becomes the prompt used above. In this way, users can easily construct different prompts by changing the placeholder while keeping the main task description unchanged.
Another important concept related to prompting is in-context learning. When prompting an LLM, we can add demonstrations to the context and let the model infer the desired input-output pattern from these examples. For instance, we can show the model how to answer similar student questions before asking it to respond to a new one:
\vspace{0.1cm}
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\leftskip}{2em}
\setlength{\rightskip}{2em}
\textit{Input:} Give me two tips to improve my English writing. \\ [1mm]
2. Revise your sentences to make them clear. \\ [2mm]
\textit{Input:} Give me three tips to improve my accuracy in solving math problems. \\ [1mm]
\textit{Output:}\underline{\hspace{4cm}}
\endgroup
\end{tcolorbox}
\vspace{0.5em}
Prompting is crucial because it directly influences how the LLM understands the task and generates the output. A well-designed prompt can guide the model to produce more accurate, relevant, and appropriately formatted responses. However, since the model parameters are unchanged, the result can also be sensitive to the wording, structure, and demonstrations provided in the prompt.
Following the notation used above, we denote the entire prompt by $\mathbf{x}=x_1...x_m$. Prompting asks the LLM to generate an output $\mathbf{y}=y_1...y_T$ according to the conditional distribution $\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$. The probability of the output is factorized as
where $\mathbf{y}_{<t}=y_1...y_{t-1}$ denotes the tokens that have already been generated. This conditional generation formulation will be repeatedly used in the following sections. From this perspective, a prompt mainly changes the condition under which the same language model distribution is used.
\subsubsection{Supervised Fine-Tuning}
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question answering, and machine translation. One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific inputs (i.e., instruction + user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content added to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. Solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_1...x_m$ be an input and $\mathbf{y}=y_1...y_T$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
where $\Pr(\cdot)$ denotes the probability distribution, $\mathcal{S}$ denotes the labeled data, $\tilde{\theta}$ denotes the parameters optimized via SFT, and $\hat{\theta}^+$ represents an adjustment to $\hat{\theta}$. Here, we will omit the superscript $+$ and use $\theta$ to represent $\hat{\theta}^+$ to keep the notation uncluttered. However, the reader should remember that the fine-tuning starts from the pre-trained parameters rather than randomly initialized ones.
where $\mathrm{Pr}(\cdot)$ denotes the probability distribution, $\mathcal{S}$ denotes the labeled data, $\tilde{\theta}$ denotes the parameters optimized via SFT, and $\hat{\theta}^+$ represents an adjustment to $\hat{\theta}$. Here, we will omit the superscript $+$ and use $\theta$ to represent $\hat{\theta}^+$ to keep the notation uncluttered. However, the reader should remember that the fine-tuning starts from the pre-trained parameters rather than randomly initialized ones.
The objective function $\log\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$ is computed by summing the log-probabilities of the tokens in $\mathbf{y}$, conditional on the input $\mathbf{x}$ and all the previous output tokens $\mathbf{y}_{<t}$:
This formulation is equivalent to minimizing the cross-entropy loss over the output segment. In practice, the input tokens in $\mathbf{x}$ are used as the context, while the loss is usually computed only on the output tokens in $\mathbf{y}$. In this way, SFT teaches the model not only what knowledge to express, but also how to respond in a desired instruction-following format. In the following section, we refer to the LLM after SFT as the ``SFT LLM'' for short.
\subsubsection{Inference}
Inference is the process of applying a trained LLM to generate outputs for new inputs. Given an input $\mathbf{x}$, the LLM predicts the next-token distribution $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x})$ and selects a token $y_1$ from this distribution. The selected token is then appended to the context, and the model predicts the next token according to $\mathrm{Pr}_{\theta}(\cdot|\mathbf{x},y_1)$. This procedure is repeated until a stopping criterion is met, such as generating an end-of-sequence token or reaching a maximum length.
The objective function $\log\mathrm{Pr}_{\theta}(y_i|\mathbf{x},\mathbf{y}_{<i})$ is computed by summing the log-probabilities of the tokens in $\mathbf{y}$, conditional on the input $\mathbf{x}$ and all the previous tokens $\mathbf{y}_{<i}$:
The simplest decoding rule is greedy decoding, which selects the token with the highest probability at each timestep:
where $w$ denotes a candidate token in the vocabulary $\mathcal{V}$. Other decoding methods, such as sampling, top-$k$ sampling, top-$p$ sampling, and beam search, use different strategies to balance output quality, diversity, and computational cost. Regardless of the decoding method, the generation process can be viewed as a sequence of token-level decisions.
This formulation is equivalent to minimizing the cross-entropy loss. In the following section, we refer to the LLM after SFT as the ``SFT LLM'' for short. Beyond SFT, the model architecture design and the pre-training process are also fundamentals of building an LLM. However, we will not discuss these topics. Interested readers can find further details in \citet{xiao-and-zhu:2025foundations}'s book.
This view naturally connects LLMs with RL. At timestep $t$, the current context $(\mathbf{x},\mathbf{y}_{<t})$ can be treated as the state, the next token $y_t$ can be treated as the action, and $\mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})$ can be treated as the policy. The whole output $\mathbf{y}$ is then analogous to a trajectory, whose quality can be evaluated by a reward signal. Based on this connection, the following subsection reviews the fundamentals of RL, and Section~\ref{sec:example-using-rl-training-llms} will further show how RL can be used to optimize an SFT LLM.
\subsection{Fundamentals of Reinforcement Learning}
\subsection{Fundamentals of Reinforcement Learning}
We begin by considering a practical scenario: using an LLM as a homework assistant. In this context, we aim to improve the capability of an SFT LLM to handle education-related inputs more effectively. Suppose we have a homework assistant powered by an SFT LLM, and a student types the input ``Give me three tips to improve my accuracy in solving math problems.'' A typical SFT LLM might generate a short output, such as
We continue with a practical example of using an LLM as a homework assistant. In this context, we aim to improve the capability of an SFT LLM to handle education-related inputs more effectively. Suppose we have a homework assistant powered by an SFT LLM, and a student types the input ``Give me three tips to improve my accuracy in solving math problems.'' A typical SFT LLM might generate a short output, such as
