\documentclass{article} % For LaTeX2e
\usepackage[final]{rl-introduction}

\usepackage{microtype}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{fontawesome5}
\usepackage{lineno}
\usepackage{bbding}
% Write outlines on Chinese
\usepackage{CJK}

%%%%%%%%%%%%%%%%%%%%%% algorithm %%%%%%%%%%%%%%%%%%%%%%%
\usepackage{algorithm} % http://ctan.org/pkg/algorithms
\usepackage{algpseudocode} 
% \makeatletter
% \renewcommand{\ALG@beginalgorithmic}{\normal}
% \makeatother
\algnewcommand\algorithmicinput{\textbf{Input:}}
\algnewcommand\algorithmicoutput{\textbf{Output:}}
\algnewcommand\INPUT{\item[\algorithmicinput]}
\algnewcommand\OUTPUT{\item[\algorithmicoutput]}


%%%%%%%%%%%%%%%%%%%%%% figures %%%%%%%%%%%%%%%%%%%%%%%
\usepackage{tikz-qtree}
\usepackage{tikz}
\usepackage{tcolorbox}
\usetikzlibrary {arrows.meta,bending,positioning}
\usetikzlibrary{angles,quotes}
\usetikzlibrary{fit}
\usetikzlibrary{backgrounds}
\usetikzlibrary{shadows}
\usetikzlibrary{calc}
\usetikzlibrary{matrix}
\usepackage{pgfplots}
\usepackage{soul}
\usepackage{xcolor}
\usepackage{makecell}
\usepackage[normalem]{ulem}
\usepackage{ulem}
\usepackage{multirow}
\usepackage{setspace}
\usepackage{fontawesome5}
\usepackage{varwidth}
\usetikzlibrary{arrows.meta,positioning,calc,fit,backgrounds}
\definecolor{ocrebase}{RGB}{243,102,25}
\definecolor{ocre}{RGB}{0,0,0}
\definecolor{amber}{rgb}{1.0, 0.75, 0.0}
\definecolor{ublue}{rgb}{0.152,0.250,0.545}
\definecolor{ugreen}{rgb}{0,0.5,0}
\definecolor{lgreen}{rgb}{0.9,1,0.8}
\definecolor{lightgreen}{rgb}{0.56, 0.93, 0.56}
\definecolor{kellygreen}{rgb}{0.3, 0.73, 0.09}
\definecolor{xtgreen}{rgb}{0.914,0.945,0.902}
\definecolor{lightgray}{gray}{0.85}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\definecolor{shadecolor}{rgb}{0.96,0.96,0.93}

\definecolor{lightcyan}{RGB}{194,232,247}
\definecolor{lightorange}{RGB}{255,226,187}
\definecolor{lightpink}{RGB}{252,224,225}
\definecolor{lightgreen}{RGB}{204,231,207}

% some colors
% from https://www.webdesignrankings.com/resources/lolcolors/
\definecolor{lolgreen}{RGB}{103,213,181}
\definecolor{lolred}{RGB}{238,119,133}
\definecolor{lolpurple}{RGB}{200,158,196}
\definecolor{lolblue}{RGB}{132,177,237}
\definecolor{lolorange}{RGB}{246,179,82}
\definecolor{lightsalmon}{RGB}{255,160,122}
\definecolor{lightskyblue}{RGB}{135,206,250}


%%%%%%%%%%%%%%%%%%%%%% new commands %%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\ctext}[3][RGB]{%
  \begingroup
  \definecolor{hlcolor}{#1}{#2}\sethlcolor{hlcolor}%
  \hl{#3}%
  \endgroup
}
\newcommand{\mindex}[1]{\textbf{#1}\index{#1}}

\newcommand*\circled[1]{\tikz[baseline=(char.base)]{
            \node[shape=circle,draw,inner sep=1pt] (char) {#1};}}

%%%%%%%%%%%%%%%%%%%%%% math %%%%%%%%%%%%%%%%%%%%%%%
\usepackage{amsmath}
\usepackage{amssymb}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\DeclareSymbolFont{EulerExtension}{U}{euex}{m}{n}%将积分号修改为正体
\DeclareMathSymbol{\euintop}{\mathop} {EulerExtension}{"52}
%\DeclareMathSymbol{\euointop}{\mathop} {EulerExtension}{"48}
\let\intop\euintop
%\let\ointop\euointop

\newcommand{\intoo}[2]{\mathopen{]}#1\,;#2\mathclose{[}}
\newcommand{\ud}{\mathop{\mathrm{{}d}}\mathopen{}}
\newcommand{\intff}[2]{\mathopen{[}#1\,;#2\mathclose{]}}



\definecolor{darkblue}{rgb}{0, 0, 0.5}
\hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}


\title{Reinforcement Learning without Tears:\\ An Introduction for Large Language Model Researchers}

% Authors must not appear in the submitted version. They should be hidden
% as long as the \colmfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.

\author{Chenglong Wang, Hang Zhou, Tong Xiao \& Jingbo Zhu \\
% \thanks{ Use footnote for providing further information about author (webpage, alternative address)---\emph{not} for acknowledging funding agencies.  Funding acknowledgements go at the end of the paper.} \\
School of Computer Science and Engineering\\
Northeastern University\\
Shenyang, China \\
\texttt{\{clwang1119,stceum\}@gmail.com}\\
\texttt{\{xiaotong,zhujingbo\}@mail.neu.edu.cn} \\
\AND
Tongran Liu \\
CAS Key Laboratory of Behavioral Science \\
Institute of Psychology \\
Chinese Academy of Sciences\\
Beijing, China \\
\texttt{liutr@psych.ac.cn} \\
}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\begin{document}

\ifcolmsubmission
\linenumbers
\fi

\maketitle

\begin{abstract}
abstract.    
\end{abstract}

\tableofcontents

\clearpage
\section{Introduction}

Reinforcement learning (RL) is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles.

A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of large language models (LLMs). For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1}. As a result, interest in NLP has exploded. There have been many, many conference papers discussing RL in LLMs, and even more in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI.

However, RL was not so popular in the long history of NLP, and the field has just begun to explore its potential. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard other researchers discuss advanced RL techniques, such as “use a value function to estimate the expected cumulative reward” or “learn a policy via PPO”. It seems that these techniques are so simple that presenters do not even need to give any explanation. But we were lost when seeing the math of RL, such as all those expectation signs, as well as the names of various algorithms that we are unfamiliar with.

Of course, we'd like to learn about RL, which appears straightforward. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.

The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLMs. We begin by introducing basic concepts and algorithms of RL using the LLM language. In particular, we illustrate their application through an example of training LLMs, making the concepts more accessible. We then discuss a series of refinements to the basic RL framework for LLM alignment, including advanced reward modeling, improved advantage estimation, efficient sampling, and direct LLM optimization without RL. Furthermore, we discuss how to apply RL techniques to LLM reasoning. These methods are closely related to recent LLMs, such as OpenAI o1/o3 and DeepSeek R1, which adopt large-scale RL and test-time scaling to significantly advance their reasoning abilities. In addition, we discuss applications of RL to multimodal LLMs to demonstrate how RL can be adapted to various problems. Finally, we conclude by outlining promising future research directions and discussing relevant systems and datasets.


% \clearpage
% New section: training LLMs with reinforcement learning
\input{section2/section2}

% \clearpage
% New section: training LLMs with reinforcement learning
\input{section3/section3}

% \clearpage
% New section: improved reinforcement learning for LLMs
\input{section4/section4}


% \clearpage
% New section: reinforcement learning for LLM inference
\input{section5/section5}

% \clearpage
% New section: reinforcement learning for MLLMs
\input{section6/section6}

% \clearpage
% New section: reinforcement learning for MLLMs
\input{section7/section7}

% \clearpage
% conclusion
\input{section8/section8}

% \clearpage
% systems & datasets  todo: ganyang
\input{section9/section9}


\clearpage


\bibliography{rl-introduction}
\bibliographystyle{rl-introduction}

% \appendix

\end{document}
