Skip to content

R
rl-introduction
Commit 7e3220a0 by wangchenglong
Options

Initial commit

parents
正在显示 包含 2390 行增加 和 0 行删除
README.md 0 → 100644
# Template
Template and style files for CoLM 2025
outline.tex 0 → 100644
\documentclass{article} % For LaTeX2e
\usepackage[final]{rlhf-introduction}
\usepackage{microtype}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{lineno}
\usepackage{CJK}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}
\title{Reinforcement Learning without Tears: An Introduction for Large Language Model Researchers}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \colmfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Antiquus S.~Hippocampus, Natalia Cerebro \& Amelie P. Amygdale \thanks{ Use footnote for providing further information
about author (webpage, alternative address)---\emph{not} for acknowledging
funding agencies. Funding acknowledgements go at the end of the paper.} \\
Department of Computer Science\\
Cranberry-Lemon University\\
Pittsburgh, PA 15213, USA \\
\texttt{\{hippo,brain,jen\}@cs.cranberry-lemon.edu} \\
\And
Ji Q. Ren \& Yevgeny LeNet \\
Department of Computational Neuroscience \\
University of the Witwatersrand \\
Joburg, South Africa \\
\texttt{\{robot,net\}@wits.ac.za} \\
\AND
Coauthor \\
Affiliation \\
Address \\
\texttt{email}
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\begin{document}
\ifcolmsubmission
\linenumbers
\fi
\maketitle
\begin{abstract}
abstract.
\end{abstract}
\section{Introduction}
Introduction.
\section{Basics of Reinforcement Learning}
\begin{CJK}{UTF8}{gbsn}
这部分主要讲述强化学习技术和一些符号的定义。\\
（1）介绍RL总体的架构和一些重要的元素（奖励，状态，环境） \\
（2）算法包括REINFORCE,A2C.另外提一下它们的应用场景和优缺点，这样可以为后面“训练LLM”部分做更好的衔接。
\end{CJK}
\section{Training LLMs with Reinforcement Learning}
\begin{CJK}{UTF8}{gbsn}
这部分主要讲述强化学习在大语言模型训练过程中是如何进行建模的。\\
（1）这里首先需要简单说一下大语言模型部分，pre-training，instruction tuning. \\
（2）如何进行优化LLM? 对应到大模型的时候，策略是什么？状态对应的什么？如何进行采样？如何进行参数更新的？ \\
（3）奖励如何设计的？（训练奖励模型 or 一些奖励规则--训练math推理的时候）\\
（4）另外结束的时候需要加个notation的表格。讲述各种符号分别对应的关系（在强化学习-》在大模型训练的过程）。
\end{CJK}
\subsection{Challenges}
\begin{CJK}{UTF8}{gbsn}
这里主要进行对训练LLM的强化学习训练的关键组件进行拆解，进行介绍当下的研究点：数据，效率，稳定性，以及可解释性（这部分考虑融合到上面一个章节，作为一个子章节）。
\end{CJK}
\section{Improved Reinforcement Learning for LLMs}
\begin{CJK}{UTF8}{gbsn}
对应上面的挑战，来去说现在的一些主流的方法提升强化学习在LLM中的工作。\\
（1）偏好数据、偏好泛化 \\
（2）更好的奖励建模 \\
（3）简化优化框架（dynamic RL，GRPO，remax等工作）\\
（3）直接偏好优化
% \include{section4-DPO}
\end{CJK}
\section{Reinforcement Learning for LLM Inference}
\begin{CJK}{UTF8}{gbsn}
除了直接训练LLMs之后，强化学习还从什么方面来去优化了LLMs? \\
（1）inference-time alignment \\
（2）test-time ranking, o1, R1
\end{CJK}
\section{Reinforcement Learning for Multi-Modal Language Models}
\begin{CJK}{UTF8}{gbsn}
单独的使用一个章节来去说多模态中的RL。不过这个章节不用像之前文本那样描述的比较细致，更多是一些扩展就好。\\
（1）视觉（图生文）\\
（2）使用RL训练diffusion model \\
（3）音频和视频。 \\
\end{CJK}
\section{Conclusion}
\begin{CJK}{UTF8}{gbsn}
总结和一些方向，，是否还要放一些分析（加一些实验）？
\end{CJK}
\section{Some Tools, Datasets, or Systems}
\begin{CJK}{UTF8}{gbsn}
（1）一些RL训练系统 \\
（2）一些教程 \\
（3）可视化分析工具（我记得有一个） \\
\end{CJK}
\bibliography{rlhf-introduction}
\bibliographystyle{rlhf-introduction}
\appendix
\section{Appendix}
This is an appendix.
\end{document}
rl-introduction.sty 0 → 100644
%%%% COLM Macros (LaTex)
%%%% Adapted by Yoav Artzi and Sasha Rush from Hugo Larochelle's adaptation for ICLR, which has been adaptated from the NIPS stylefile Macros
%%%% Style File
%%%% Dec 12, 1990 Rev Aug 14, 1991; Sept, 1995; April, 1997; April, 1999; October 2014
% This file can be used with Latex2e whether running in main mode, or
% 2.09 compatibility mode.
%
% If using main mode, you need to include the commands
% \documentclass{article}
% \usepackage{colm14submit_e}
%
% Define options
\newif\ifcolmsubmission
\newif\ifcolmpreprint
\newif\ifcolmfinal
% Set submission as default
\colmsubmissiontrue
\colmpreprintfalse
\colmfinalfalse
% Define option handling
\DeclareOption{submission}{\colmsubmissiontrue\colmpreprintfalse\colmfinalfalse}
\DeclareOption{preprint}{\colmsubmissionfalse\colmpreprinttrue\colmfinalfalse}
\DeclareOption{final}{\colmsubmissionfalse\colmpreprintfalse\colmfinaltrue}
\ProcessOptions\relax
% Palatino font
\RequirePackage{tgpagella} % text only
\RequirePackage{mathpazo} % math & text
\RequirePackage{inconsolata} % for tt font
% Change the overall width of the page. If these parameters are
% changed, they will require corresponding changes in the
% maketitle section.
%
\usepackage{eso-pic} % used by \AddToShipoutPicture
\RequirePackage{fancyhdr}
\RequirePackage{natbib}
% modification to natbib citations
\setcitestyle{authoryear,round,citesep={;},aysep={,},yysep={;}}
\renewcommand{\topfraction}{0.95} % let figure take up nearly whole page
\renewcommand{\textfraction}{0.05} % let figure take up nearly whole page
% Specify the dimensions of each page
\setlength{\paperheight}{11in}
\setlength{\paperwidth}{8.5in}
\oddsidemargin .09in % Note \oddsidemargin = \evensidemargin
\evensidemargin .09in
\marginparwidth 0.07 true in
%\marginparwidth 0.75 true in
%\topmargin 0 true pt % Nominal distance from top of page to top of
%\topmargin 0.125in
\topmargin -0.625in
\addtolength{\headsep}{0.25in}
\textheight 9.0 true in % Height of text (including footnotes & figures)
\textwidth 6.4 true in % Width of text line.
\widowpenalty=10000
\clubpenalty=10000
% \thispagestyle{empty} \pagestyle{empty}
\flushbottom \sloppy
% We're never going to need a table of contents, so just flush it to
% save space --- suggested by drstrip@sandia-2
\def\addcontentsline#1#2#3{}
% Title stuff, taken from deproc.
\def\maketitle{\par
\begingroup
\def\thefootnote{\fnsymbol{footnote}}
\def\@makefnmark{\hbox to 0pt{$^{\@thefnmark}$\hss}} % for perfect author
% name centering
% The footnote-mark was overlapping the footnote-text,
% added the following to fix this problem (MK)
\long\def\@makefntext##1{\parindent 1em\noindent
\hbox to1.8em{\hss $\m@th ^{\@thefnmark}$}##1}
\@maketitle \@thanks
\endgroup
\setcounter{footnote}{0}
\let\maketitle\relax \let\@maketitle\relax
\gdef\@thanks{}\gdef\@author{}\gdef\@title{}\let\thanks\relax}
% The toptitlebar has been raised to top-justify the first page
\usepackage{fancyhdr}
\pagestyle{fancy}
% \renewcommand{\headrulewidth}{1.5pt}
\renewcommand{\headrulewidth}{0pt}
\fancyhead{}
% Title (includes both anonymized and non-anonymized versions)
\def\@maketitle{\vbox{\hsize\textwidth
%\linewidth\hsize \vskip 0.1in \toptitlebar \centering
{\Large\bf \@title\par}
%\bottomtitlebar % \vskip 0.1in % minus
\ifcolmfinal
% \lhead{Published as a conference paper at COLM 2025}
\def\And{\end{tabular}\hfil\linebreak[0]\hfil
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
\def\AND{\end{tabular}\hfil\linebreak[4]\hfil
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\@author\end{tabular}%
\else\ifcolmpreprint
\lhead{Preprint. Under review.}
\def\And{\end{tabular}\hfil\linebreak[0]\hfil
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
\def\AND{\end{tabular}\hfil\linebreak[4]\hfil
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\@author\end{tabular}%
\else
\lhead{Under review as a conference paper at COLM 2025}
\def\And{\end{tabular}\hfil\linebreak[0]\hfil
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
\def\AND{\end{tabular}\hfil\linebreak[4]\hfil
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
\begin{tabular}[t]{l}\bf\rule{\z@}{24pt}Anonymous authors\\Paper under double-blind review\end{tabular}%
\fi\fi
\vskip 0.3in minus 0.1in}}
\renewenvironment{abstract}{\vskip.075in\centerline{\large\bf
Abstract}\vspace{0.5ex}\begin{quote}}{\par\end{quote}\vskip 1ex}
% Less leading in most fonts (due to the narrow columns)
% The choices were between 1-pt and 1.5-pt leading
%\def\@normalsize{\@setsize\normalsize{11pt}\xpt\@xpt} % got rid of @ (MK)
\def\normalsize{\@setsize\normalsize{12.5pt}\xpt\@xpt}
\def\small{\@setsize\small{10pt}\ixpt\@ixpt}
\def\footnotesize{\@setsize\footnotesize{10pt}\ixpt\@ixpt}
\def\scriptsize{\@setsize\scriptsize{8pt}\viipt\@viipt}
\def\tiny{\@setsize\tiny{7pt}\vipt\@vipt}
\def\large{\@setsize\large{14pt}\xiipt\@xiipt}
\def\Large{\@setsize\Large{16pt}\xivpt\@xivpt}
\def\LARGE{\@setsize\LARGE{20pt}\xviipt\@xviipt}
\def\huge{\@setsize\huge{23pt}\xxpt\@xxpt}
\def\Huge{\@setsize\Huge{28pt}\xxvpt\@xxvpt}
% sections with less space
\def\section{\@startsection {section}{1}{\z@}{-2.0ex plus
-0.5ex minus -.2ex}{1.5ex plus 0.3ex
minus0.2ex}{\large\bf\raggedright}}
\def\subsection{\@startsection{subsection}{2}{\z@}{-1.8ex plus
-0.5ex minus -.2ex}{0.8ex plus .2ex}{\normalsize\bf\raggedright}}
\def\subsubsection{\@startsection{subsubsection}{3}{\z@}{-1.5ex
plus -0.5ex minus -.2ex}{0.5ex plus
.2ex}{\normalsize\bf\itshape\raggedright}}
\def\paragraph{\@startsection{paragraph}{4}{\z@}{1.5ex plus
0.5ex minus .2ex}{-1em}{\normalsize\bf}}
\def\subparagraph{\@startsection{subparagraph}{5}{\z@}{1.5ex plus
0.5ex minus .2ex}{-1em}{\normalsize\it}}
\def\subsubsubsection{\vskip
5pt{\noindent\normalsize\raggedright}}
% Footnotes
\footnotesep 6.65pt %
\skip\footins 9pt plus 4pt minus 2pt
\def\footnoterule{\kern-3pt \hrule width 12pc \kern 2.6pt }
\setcounter{footnote}{0}
% Lists and paragraphs
\parindent 0pt
\topsep 4pt plus 1pt minus 2pt
\partopsep 1pt plus 0.5pt minus 0.5pt
\itemsep 2pt plus 1pt minus 0.5pt
\parsep 2pt plus 1pt minus 0.5pt
\parskip .6pc
%\leftmargin2em
\leftmargin3pc
\leftmargini\leftmargin \leftmarginii 2em
\leftmarginiii 1.5em \leftmarginiv 1.0em \leftmarginv .5em
%\labelsep \labelsep 5pt
\def\@listi{\leftmargin\leftmargini}
\def\@listii{\leftmargin\leftmarginii
\labelwidth\leftmarginii\advance\labelwidth-\labelsep
\topsep 2pt plus 1pt minus 0.5pt
\parsep 1pt plus 0.5pt minus 0.5pt
\itemsep \parsep}
\def\@listiii{\leftmargin\leftmarginiii
\labelwidth\leftmarginiii\advance\labelwidth-\labelsep
\topsep 1pt plus 0.5pt minus 0.5pt
\parsep \z@ \partopsep 0.5pt plus 0pt minus 0.5pt
\itemsep \topsep}
\def\@listiv{\leftmargin\leftmarginiv
\labelwidth\leftmarginiv\advance\labelwidth-\labelsep}
\def\@listv{\leftmargin\leftmarginv
\labelwidth\leftmarginv\advance\labelwidth-\labelsep}
\def\@listvi{\leftmargin\leftmarginvi
\labelwidth\leftmarginvi\advance\labelwidth-\labelsep}
\abovedisplayskip 7pt plus2pt minus5pt%
\belowdisplayskip \abovedisplayskip
\abovedisplayshortskip 0pt plus3pt%
\belowdisplayshortskip 4pt plus3pt minus3pt%
\def\toptitlebar{\hrule height4pt\vskip .25in\vskip-\parskip}
\def\bottomtitlebar{\vskip .29in\vskip-\parskip\hrule height1pt\vskip
.09in} %
%Reduced second vskip to compensate for adding the strut in \@author
rl-introduction.tex 0 → 100644
\documentclass{article} % For LaTeX2e
\usepackage[final]{rl-introduction}
\usepackage{microtype}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{fontawesome5}
\usepackage{lineno}
\usepackage{bbding}
% Write outlines on Chinese
\usepackage{CJK}
%%%%%%%%%%%%%%%%%%%%%% algorithm %%%%%%%%%%%%%%%%%%%%%%%
\usepackage{algorithm} % http://ctan.org/pkg/algorithms
\usepackage{algpseudocode}
% \makeatletter
% \renewcommand{\ALG@beginalgorithmic}{\normal}
% \makeatother
\algnewcommand\algorithmicinput{\textbf{Input:}}
\algnewcommand\algorithmicoutput{\textbf{Output:}}
\algnewcommand\INPUT{\item[\algorithmicinput]}
\algnewcommand\OUTPUT{\item[\algorithmicoutput]}
%%%%%%%%%%%%%%%%%%%%%% figures %%%%%%%%%%%%%%%%%%%%%%%
\usepackage{tikz-qtree}
\usepackage{tikz}
\usepackage{tcolorbox}
\usetikzlibrary {arrows.meta,bending,positioning}
\usetikzlibrary{angles,quotes}
\usetikzlibrary{fit}
\usetikzlibrary{backgrounds}
\usetikzlibrary{shadows}
\usetikzlibrary{calc}
\usetikzlibrary{matrix}
\usepackage{pgfplots}
\usepackage{soul}
\usepackage{xcolor}
\usepackage{makecell}
\usepackage[normalem]{ulem}
\usepackage{ulem}
\usepackage{multirow}
\usepackage{setspace}
\usepackage{fontawesome5}
\usepackage{varwidth}
\usetikzlibrary{shadows}
\definecolor{ocrebase}{RGB}{243,102,25}
\definecolor{ocre}{RGB}{0,0,0}
\definecolor{amber}{rgb}{1.0, 0.75, 0.0}
\definecolor{ublue}{rgb}{0.152,0.250,0.545}
\definecolor{ugreen}{rgb}{0,0.5,0}
\definecolor{lgreen}{rgb}{0.9,1,0.8}
\definecolor{lightgreen}{rgb}{0.56, 0.93, 0.56}
\definecolor{kellygreen}{rgb}{0.3, 0.73, 0.09}
\definecolor{xtgreen}{rgb}{0.914,0.945,0.902}
\definecolor{lightgray}{gray}{0.85}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\definecolor{shadecolor}{rgb}{0.96,0.96,0.93}
\definecolor{lightcyan}{RGB}{194,232,247}
\definecolor{lightorange}{RGB}{255,226,187}
\definecolor{lightpink}{RGB}{252,224,225}
\definecolor{lightgreen}{RGB}{204,231,207}
% some colors
% from https://www.webdesignrankings.com/resources/lolcolors/
\definecolor{lolgreen}{RGB}{103,213,181}
\definecolor{lolred}{RGB}{238,119,133}
\definecolor{lolpurple}{RGB}{200,158,196}
\definecolor{lolblue}{RGB}{132,177,237}
\definecolor{lolorange}{RGB}{246,179,82}
\definecolor{lightsalmon}{RGB}{255,160,122}
\definecolor{lightskyblue}{RGB}{135,206,250}
%%%%%%%%%%%%%%%%%%%%%% new commands %%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\ctext}[3][RGB]{%
\begingroup
\definecolor{hlcolor}{#1}{#2}\sethlcolor{hlcolor}%
\hl{#3}%
\endgroup
}
\newcommand{\mindex}[1]{\textbf{#1}\index{#1}}
\newcommand*\circled[1]{\tikz[baseline=(char.base)]{
\node[shape=circle,draw,inner sep=1pt] (char) {#1};}}
%%%%%%%%%%%%%%%%%%%%%% math %%%%%%%%%%%%%%%%%%%%%%%
\usepackage{amsmath}
\usepackage{amssymb}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareSymbolFont{EulerExtension}{U}{euex}{m}{n}%将积分号修改为正体
\DeclareMathSymbol{\euintop}{\mathop} {EulerExtension}{"52}
%\DeclareMathSymbol{\euointop}{\mathop} {EulerExtension}{"48}
\let\intop\euintop
%\let\ointop\euointop
\newcommand{\intoo}[2]{\mathopen{]}#1\,;#2\mathclose{[}}
\newcommand{\ud}{\mathop{\mathrm{{}d}}\mathopen{}}
\newcommand{\intff}[2]{\mathopen{[}#1\,;#2\mathclose{]}}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}
\title{Reinforcement Learning without Tears:\\ An Introduction for Large Language Model Researchers}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \colmfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Chenglong Wang, Hang Zhou, Tong Xiao \& Jingbo Zhu \\
% \thanks{ Use footnote for providing further information about author (webpage, alternative address)---\emph{not} for acknowledging funding agencies. Funding acknowledgements go at the end of the paper.} \\
School of Computer Science and Engineering\\
Northeastern University\\
Shenyang, China \\
\texttt{\{clwang1119,stceum\}@gmail.com}\\
\texttt{\{xiaotong,zhujingbo\}@mail.neu.edu.cn} \\
\AND
Tongran Liu \\
CAS Key Laboratory of Behavioral Science \\
Institute of Psychology \\
Chinese Academy of Sciences\\
Beijing, China \\
\texttt{liutr@psych.ac.cn} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\begin{document}
\ifcolmsubmission
\linenumbers
\fi
\maketitle
\begin{abstract}
abstract.
\end{abstract}
\tableofcontents
\clearpage
\section{Introduction}
Reinforcement learning (RL) is an interdisciplinary field, and has its origins in both psychology and optimal control. Over the years, the concept has been further developed and formalized within the realms of artificial intelligence and machine learning. The basic idea is that an agent learns to make decisions by receiving feedback on its actions from the environment. This approach is very general and applies to a wide range of problems, from playing complex games like chess and Go to controlling autonomous vehicles.
A recent breakthrough is the large-scale application of RL in the field of natural language processing (NLP), in particular in the development of large language models (LLMs). For example, RL has been widely considered an effective way of aligning pre-trained LLMs with human preferences and values \citep{christiano-etal:2017deep}. One such method, called reinforcement learning from human feedback (RLHF), forms the basis for the development of advanced LLMs like OpenAI's GPT-4. More recently, RL has shown great promise in enabling LLMs to perform complex reasoning \citep{openai:2024learning,deepseek:2025r1}. As a result, interest in NLP has exploded. There have been many, many conference papers discussing RL in LLMs, and even more in the past few months. The research community is highly enthusiastic, and many in the field are anticipating a new turning point where large-scale RL algorithms could further advance AI.
However, RL was not so popular in the long history of NLP, and the field has just begun to explore its potential. Although applying RL to LLMs seems a natural choice for machine learning researchers, it introduces many new concepts to the NLP community, which has traditionally been dominated by supervised learning methodologies. As NLP researchers, when we attended presentations on LLMs, we often heard other researchers discuss advanced RL techniques, such as “use a value function to estimate the expected cumulative reward” or “learn a policy via PPO”. It seems that these techniques are so simple that presenters do not even need to give any explanation. But we were lost when seeing the math of RL, such as all those expectation signs, as well as the names of various algorithms that we are unfamiliar with.
Of course, we'd like to learn about RL, which appears straightforward. However, common RL textbooks, such as \citet{Sutton-and-Barto:2018RL}'s book, are mostly based on classic robotics or control problems. This makes it difficult to align the concepts of RL with those of LLMs. On the other hand, most LLM literature lacks an in-depth discussion of RL techniques, often omitting related details. As a result, we find ourselves in an awkward middle ground between RL and LLMs, struggling to bridge the two fields.
The aim of this paper is to provide a comprehensive introduction to RL from the perspective of LLMs. We begin by introducing basic concepts and algorithms of RL using the LLM language. In particular, we illustrate their application through an example of training LLMs, making the concepts more accessible. We then discuss a series of refinements to the basic RL framework for LLM alignment, including advanced reward modeling, improved advantage estimation, efficient sampling, and direct LLM optimization without RL. Furthermore, we discuss how to apply RL techniques to LLM reasoning. These methods are closely related to recent LLMs, such as OpenAI o1/o3 and Deepseek R1, which adopt large-scale RL and test-time scaling to significantly advance their reasoning abilities. In addition, we discuss applications of RL to multimodal LLMs to demonstrate how RL can be adapted to various problems. Finally, we conclude by outlining promising future research directions and discussing relevant systems and datasets.
% \clearpage
% New section: training LLMs with reinforcement learning
% \input{section2/section2}
% \clearpage
% New section: training LLMs with reinforcement learning
% \input{section3/section3}
% \clearpage
% New section: improved reinforcement learning for LLMs
\input{section4/section4}
% \clearpage
% New section: reinforcement learning for LLM inference
% \input{section5/section5}
% \clearpage
% New section: reinforcement learning for MLLMs
% \input{section6/section6}
% \clearpage
% conclusion
% \input{section7/section7}
% \clearpage
% systems & datasets todo: ganyang
% \input{section8/section8}
\clearpage
\bibliography{rl-introduction}
\bibliographystyle{rl-introduction}
% \appendix
\end{document}
section2/Figures/figure-llm-as-policy.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{0.5cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
%%% step t
\begin{scope}
\node [anchor=west] (x0) at (0,0) {\scriptsize{$x_0$}};
\node [anchor=west] (x1) at ([xshift=\ssep]x0) {\scriptsize{$x_1$}};
\node [anchor=west] (x2) at ([xshift=\ssep]x1) {\scriptsize{$...$}};
\node [anchor=west] (x3) at ([xshift=\ssep]x2) {\scriptsize{$x_m$}};
\node [anchor=west] (y1) at ([xshift=\ssep]x3) {\scriptsize{$y_1$}};
\node [anchor=west] (y2) at ([xshift=\ssep]y1) {\scriptsize{$...$}};
\node [anchor=west] (y3) at ([xshift=\ssep]y2) {\scriptsize{$y_{t-1}$}};
%\node [anchor=west] (y4) at ([xshift=\ssep]y3) {\scriptsize{$y_{t}$}};
%\node [anchor=west] (y5) at ([xshift=\ssep]y4) {\scriptsize{$...$}};
\node [lnode,anchor=south] (llm) at ([xshift=0,yshift=0.3cm]y1.north) {\large{Policy (LLM)}};
\draw [->] ([yshift=0.18cm]x0.center) -- ([yshift=0.47cm]x0.center);
\draw [->] ([yshift=0.18cm]x1.center) -- ([yshift=0.47cm]x1.center);
\draw [->] ([yshift=0.18cm]x3.center) -- ([yshift=0.47cm]x3.center);
\draw [->] ([yshift=0.18cm]y1.center) -- ([yshift=0.47cm]y1.center);
\draw [->] ([yshift=0.18cm]y3.center) -- ([yshift=0.47cm]y3.center);
%\draw [->] ([yshift=0.18cm]y4.center) -- ([yshift=0.47cm]y4.center);
%\node [anchor=center] (ox0) at ([yshift=2.0cm]x0.north) {\scriptsize{\color{gray!50} $x_1$}};
%\node [anchor=center] (ox1) at ([yshift=2.0cm]x1.north) {\scriptsize{\color{gray!50} $x_2$}};
%\node [anchor=center] (ox2) at ([yshift=2.0cm]x2.north) {\scriptsize{\color{gray!50} $...$}};
\node [anchor=center] (ox3) at ([yshift=2.0cm]x3.north) {\scriptsize{\color{gray!50} $y_1$}};
\node [anchor=center] (oy1) at ([yshift=2.0cm]y1.north) {\scriptsize{\color{gray!50} $y_2$}};
\node [anchor=center] (oy2) at ([yshift=2.0cm]y2.north) {\scriptsize{\color{gray!50} $...$}};
\node [anchor=center] (oy3) at ([yshift=2.0cm]y3.north) {\scriptsize{$y_t$}};
%\node [anchor=center] (oy4) at ([yshift=2.0cm]y4.north) {\scriptsize{$y_{t+1}$}};
%\node [anchor=center] (oy5) at ([yshift=2.0cm]y5.north) {\scriptsize{$...$}};
%\draw [<-,gray!50] ([yshift=-0.18cm]ox0.center) -- ([yshift=-0.47cm]ox0.center);
%\draw [<-,gray!50] ([yshift=-0.18cm]ox1.center) -- ([yshift=-0.47cm]ox1.center);
\draw [<-,gray!50] ([yshift=-0.18cm]ox3.center) -- ([yshift=-0.47cm]ox3.center);
\draw [<-,gray!50] ([yshift=-0.18cm]oy1.center) -- ([yshift=-0.47cm]oy1.center);
\draw [<-] ([yshift=-0.18cm]oy3.center) -- ([yshift=-0.47cm]oy3.center);
%\draw [<-] ([yshift=-0.18cm]oy4.center) -- ([yshift=-0.47cm]oy4.center);
\draw [decoration={brace,amplitude=4pt},decorate] ([xshift=0,yshift=-0.2cm]y3.east) -- ([xshift=0,yshift=-0.2cm]x0.west) node [pos=0.5,below,yshift=-0.2cm,fill=lightsalmon!80] (statebox) {\scriptsize{State $s_t$ ($\mathbf{x}$ and $\mathbf{y}_{<t}$)}};
\node [anchor=south,fill=lightskyblue!80] (actionbox) at ([yshift=-0.0cm]oy3.north) {\scriptsize{Action $a_t$}};
\node [anchor=south west,minimum width=3.5cm,minimum height=1.3cm,draw,thick,align=center] (rewardmodel) at ([yshift=-1.4cm,xshift=2cm]llm.south east) {\footnotesize{Reward Model}\\[-0.0cm] \small{$R(\ $\ctext[RGB]{255,160,122}{$s_t$},\ \ctext[RGB]{135,206,250}{$a_t$}\ $)$}};
\node [anchor=south west,minimum width=3.5cm,minimum height=1.3cm,draw,thick,align=center] (valuefunction) at ([yshift=1.4cm]rewardmodel.north west) {\footnotesize{Value Functions}\\[-0.0cm] \small{$V($\ \ctext[RGB]{255,160,122}{$s_t$}\ $)$} \small{and} \small{$Q($\ \ctext[RGB]{255,160,122}{$s_t$},\ \ctext[RGB]{135,206,250}{$a_t$}\ $)$}};
\draw [->,thin] ([xshift=2pt]actionbox.east) -- ([xshift=2cm]actionbox.east) -- ([xshift=2cm,yshift=-3cm]actionbox.east) -- ([xshift=3.05cm,yshift=-3cm]actionbox.east);
\draw [->,thin] ([xshift=2pt]statebox.east) -- ([xshift=4.74cm]statebox.east);
\draw [->,thick] ([yshift=2pt]rewardmodel.north) -- ([yshift=-2pt]valuefunction.south);
\draw [->,thick,dotted] ([yshift=2pt]valuefunction.north) -- ([yshift=1.2cm]valuefunction.north) -- ([yshift=1.2cm,xshift=-5.6cm]valuefunction.north) node [pos=0.5,above] {\small{Feedback}} -- ([yshift=-0.17cm,xshift=-5.6cm]valuefunction.north);
\end{scope}
\end{tikzpicture}
\end{center}
section2/section2.tex 0 → 100644
\section{Preliminary}
In this section, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and nations necessary for the discussions in future sections.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question-answer, and machine translation.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Specifically, let $\mathbf{x}=x_0...x_m$ be an input (e.g., instruction + user input) and $\mathbf{y}=y_1...y_n$ be the corresponding output. In SFT, we aim to maximize the probability of the output $\mathbf{y}$ given the input $\mathbf{x}$. Consider an LLM with pre-trained parameters $\hat{\theta}$. The fine-tuning objective can then be formulated as:
\begin{eqnarray}
\tilde{\theta} & = & \argmax_{\hat{\theta}^+} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{S}} \log \mathrm{Pr}_{\hat{\theta}^+}(\mathbf{y}|\mathbf{x}) \label{eq:instruction-fine-tuning}
\end{eqnarray}
where $\Pr(\cdot)$ denotes the probability distribution, $\mathcal{S}$ denotes the labeled data, $\tilde{\theta}$ denotes the parameters optimized via SFT, and $\hat{\theta}^+$ represents an adjustment to $\hat{\theta}$. Here, we will omit the superscript $+$ and use $\theta$ to represent $\hat{\theta}^+$ to keep the notation uncluttered. However, the reader should remember that the fine-tuning starts from the pre-trained parameters rather than randomly initialized ones.
The objective function $\log \mathrm{Pr}_{\theta}(y_i|\mathbf{x},\mathbf{y}_{<i})$ is computed by summing the log-probabilities of the tokens in $\mathbf{y}$, conditional on the input $\mathbf{x}$ and all the previous tokens $\mathbf{y}_{<i}$:
\begin{eqnarray}
\log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) & = & \sum_{i=1}^{n} \log \mathrm{Pr}_{\theta}(y_i|\mathbf{x},\mathbf{y}_{<i})
\end{eqnarray}
This formulation is equivalent to minimizing the cross-entropy loss. In the following section, we refer to the LLM after SFT as the ``SFT LLM'' for short. Beyond SFT, the model architecture design and the pre-training process are also fundamentals of building an LLM. However, we will not discuss these topics. Interested readers can find further details in \citet{xiao-and-zhu:2025foundations}'s book.
%%%%%%%%%% Preliminary修改建议 %%%%%%%%%%
% 1. 再加一些强化学习的基础知识
% 包含对环境、动作、policy这些术语的定义，方面后面对应起来的好定义。
% 如果修改这一章节 那么3.1这部分要对应起来，有些需要删除掉。
section2/tables/rl-elements-and-llm-counterparts.tex 0 → 100644
\begin{tabular}{ll}
\toprule[1.1pt]
\multicolumn{1}{c}{\textbf{Element in Reinforcement Learning}} & \multicolumn{1}{c}{\textbf{Counterpart in LLMs}} \\ \midrule
\parbox{4cm}{
Action ($a$)
} & \parbox{7.2cm}{
The possible predicted tokens in the vocabulary of the LLM.
} \\ \midrule
Reward ($R$) & \\
& \\
& \\
& \\
\bottomrule[1.1pt]
\end{tabular}
\ No newline at end of file
section3/Figures/figure-an-overlap-example.tex 0 → 100644
\begin{center}
\def\tgtwidth{0.8\textwidth}
\begin{tikzpicture}
\tikzstyle{ynode} = [draw,rounded corners=4pt, minimum width=\tgtwidth,inner sep=4pt,align=left,text width=\tgtwidth]
\begin{scope}
\node (input) at (0,0) {$\mathbf{x}$};
\node [ynode, anchor=west] (note) at ([xshift=.2cm]input.east) {
\begin{varwidth}{\textwidth}
\scriptsize{Give me three tips to improve my accuracy in solving math problems.}
\end{varwidth}
};
\node [ynode,anchor=north west] (output1) at ([yshift=-.2cm]note.south west) {
\scriptsize{Improving accuracy in solving math problems is crucial for success in mathematics. Here are three tips that can help: \\
1.Practice regularly... \\
2.Understand the concepts... \\
\ctext[RGB]{255,204,204}{3.Double-check your work...any-one can significantly enhance their math problem-solving abilities.} \\
\textbf{\{Total Tokens: 160, Reward: 16\}}}
};
\node [anchor=east](y1) at ([xshift=-.2cm]output1.west) {$\mathbf{y}_1$};
\node [ynode,anchor=north] (output2) at ([yshift=-.2cm]output1.south) {
\scriptsize{Improving math accuracy requires careful attention to detail, strategic problem-solving techniques, and consistent practice. Here are three tips: \\
1.Avoid rushing and read carefully... \\
2.Use estimation to validate answers... \\
\ctext[RGB]{255,204,204}{3.Double-check your work...any-one can significantly enhance their math problem-solving abilities.} \\
\textbf{\{Total Tokens: 750, Reward: 75\}}}
};
\node [anchor=east](y2) at ([xshift=-.2cm]output2.west) {$\mathbf{y}_2$};
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-architecture-reward-model.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{1cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
\begin{scope}
\node [anchor=west] (x0) at (0,0) {\footnotesize{$x_0$}};
\node [anchor=center] (x1) at ([xshift=\ssep]x0.center) {\footnotesize{$x_1$}};
\node [anchor=center] (x2) at ([xshift=\ssep]x1.center) {\footnotesize{$x_2$}};
\node [anchor=center] (x3) at ([xshift=\ssep]x2.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (x4) at ([xshift=\ssep]x3.center) {\footnotesize{$x_m$}};
\node [anchor=center] (y1) at ([xshift=\ssep]x4.center) {\footnotesize{$y_1$}};
\node [anchor=center] (y2) at ([xshift=\ssep]y1.center) {\footnotesize{$y_2$}};
\node [anchor=center] (y3) at ([xshift=\ssep]y2.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (y4) at ([xshift=\ssep]y3.center) {\footnotesize{$y_n$}};
\node [anchor=north] (y4label) at ([yshift=0.1cm]y4.south) {\scriptsize{(Last Token $\langle \mathrm{EOS} \rangle$)}};
\draw [->] ([yshift=0.2cm]x0.center) -- ([yshift=0.55cm]x0.center);
\draw [->] ([yshift=0.2cm]x1.center) -- ([yshift=0.55cm]x1.center);
\draw [->] ([yshift=0.2cm]x2.center) -- ([yshift=0.55cm]x2.center);
\draw [->] ([yshift=0.2cm]x4.center) -- ([yshift=0.55cm]x4.center);
\draw [->] ([yshift=0.2cm]y1.center) -- ([yshift=0.55cm]y1.center);
\draw [->] ([yshift=0.2cm]y2.center) -- ([yshift=0.55cm]y2.center);
\draw [->] ([yshift=0.2cm]y4.center) -- ([yshift=0.55cm]y4.center);
\node [anchor=center] (ox0) at ([yshift=2.6cm]x0.center) {\footnotesize{$\mathbf{h}_{x_0}$}};
\node [anchor=center] (ox1) at ([yshift=2.6cm]x1.center) {\footnotesize{$\mathbf{h}_{x_1}$}};
\node [anchor=center] (ox2) at ([yshift=2.6cm]x2.center) {\footnotesize{$\mathbf{h}_{x_2}$}};
\node [anchor=center] (ox3) at ([yshift=2.6cm]x3.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (ox4) at ([yshift=2.6cm]x4.center) {\footnotesize{$\mathbf{h}_{x_m}$}};
\node [anchor=center] (oy1) at ([yshift=2.6cm]y1.center) {\footnotesize{$\mathbf{h}_{y_1}$}};
\node [anchor=center] (oy2) at ([yshift=2.6cm]y2.center) {\footnotesize{$\mathbf{h}_{y_2}$}};
\node [anchor=center] (oy3) at ([yshift=2.6cm]y3.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (oy4) at ([yshift=2.6cm]y4.center) {\footnotesize{$\mathbf{h}_{\mathrm{last}}$}};
\node [anchor=south,draw,thick,minimum width=9cm,minimum height=1.3cm] (llm) at ([yshift=0.4cm]x4.north) {\large{Transformer Decoder (LLM)}};
\node [anchor=east] (representation) at ([xshift=-0.3cm]ox0.west) {\scriptsize{Representation}};
\node [anchor=north west] (representation2) at ([yshift=0.1cm]representation.south west) {\scriptsize{at Each Position}};
\draw [<-] ([yshift=-0.25cm]ox0.center) -- ([yshift=-0.6cm]ox0.center);
\draw [<-] ([yshift=-0.25cm]ox1.center) -- ([yshift=-0.6cm]ox1.center);
\draw [<-] ([yshift=-0.25cm]ox2.center) -- ([yshift=-0.6cm]ox2.center);
\draw [<-] ([yshift=-0.25cm]ox4.center) -- ([yshift=-0.6cm]ox4.center);
\draw [<-] ([yshift=-0.25cm]oy1.center) -- ([yshift=-0.6cm]oy1.center);
\draw [<-] ([yshift=-0.25cm]oy2.center) -- ([yshift=-0.6cm]oy2.center);
\draw [<-] ([yshift=-0.25cm]oy4.center) -- ([yshift=-0.6cm]oy4.center);
\filldraw [fill=red!20,draw=white] ([xshift=-0.5cm,yshift=0.2cm]oy4.north west) -- ([xshift=0.5cm,yshift=0.2cm]oy4.north east) -- ([yshift=1cm,xshift=0]oy4.north east) -- ([yshift=1cm,xshift=0]oy4.north west) -- ([xshift=-0.5cm,yshift=0.2cm]oy4.north west);
\node [anchor=south] (reward) at ([yshift=1.2cm]oy4.north) {\footnotesize{Reward (Scalar)}};
\node [anchor=south] (Wr) at ([yshift=0.3cm]oy4.north) {\footnotesize{$\mathbf{W}_r$}};
\node [anchor=west] (linear) at ([xshift=0.5cm]Wr.east) {\scriptsize{Linear Map}};
\draw [->] ([yshift=-0.1cm]oy4.north) -- ([yshift=0.18cm]oy4.north);
\draw [<-] ([yshift=0.1cm]reward.south) -- ([yshift=-0.18cm]reward.south);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-architecture-value-function.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{1.5cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
\begin{scope}
\node [anchor=center] (x1) at (0,0) {\footnotesize{$x_1$}};
\node [anchor=center] (xcdots) at ([xshift=\ssep]x1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (xm) at ([xshift=\ssep]xcdots.center) {\footnotesize{$x_m$}};
\node [anchor=center] (y1) at ([xshift=\ssep]xm.center) {\footnotesize{$y_1$}};
\node [anchor=center] (ycdots) at ([xshift=\ssep]y1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (yt) at ([xshift=\ssep]ycdots.center) {\footnotesize{$y_{T}$}};
\foreach \x in {x1, xm, y1, yt}
\draw [->] ([yshift=0.2cm]\x.center) -- ([yshift=0.55cm]\x.center);
\node [anchor=center] (oy1) at ([yshift=2.6cm]y1.center) {\footnotesize{$\mathbf{h}_{y_1}$}};
\node [anchor=center] (oycdots) at ([yshift=2.6cm]ycdots.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (oyt) at ([yshift=2.6cm]yt.center) {\footnotesize{$\mathbf{h}_{y_T}$}};
\node [anchor=south,draw,thick,minimum width=9cm,minimum height=1.3cm] (llm) at ([yshift=0.4cm]$(xm.north)!.5!(y1.north)$) {\large{Value Model (LLM)}};
\foreach \x in {oy1, oyt}
\draw [<-] ([yshift=-0.25cm]\x.center) -- ([yshift=-0.6cm]\x.center);
\filldraw [fill=red!20,draw=white] ([xshift=-0.5cm,yshift=0.2cm]oyt.north west) -- ([xshift=0.5cm,yshift=0.2cm]oyt.north east) -- ([yshift=1cm,xshift=0]oyt.north east) -- ([yshift=1cm,xshift=0]oyt.north west) -- ([xshift=-0.5cm,yshift=0.2cm]oyt.north west);
\filldraw [fill=red!20,draw=white] ([xshift=-0.5cm,yshift=0.2cm]oy1.north west) -- ([xshift=0.5cm,yshift=0.2cm]oy1.north east) -- ([yshift=1cm,xshift=0]oy1.north east) -- ([yshift=1cm,xshift=0]oy1.north west) -- ([xshift=-0.5cm,yshift=0.2cm]oy1.north west);
\node at ([yshift=0.6cm]$(oy1.north)!.5!(oyt.north)$) {\footnotesize{$\cdots$}};
\node [anchor=south] (Wr) at ([yshift=0.3cm]oyt.north) {\footnotesize{$\mathbf{W}_r$}};
\node [anchor=south] (output) at ([yshift=0.5cm]Wr.north) {\footnotesize{Value (Scalar)}};
\draw [->] ([yshift=0.2cm]Wr.north) -- (output.south);
\node [anchor=west] (linear) at ([xshift=0.5cm]Wr.east) {\scriptsize{Linear Map}};
\node [anchor=south] (Wr) at ([yshift=0.3cm]oy1.north) {\footnotesize{$\mathbf{W}_r$}};
\node [anchor=south] (output) at ([yshift=0.5cm]Wr.north) {\footnotesize{Value (Scalar)}};
\draw [->] ([yshift=0.2cm]Wr.north) -- (output.south);
\node [anchor=east,align=center] at ([xshift=-0.0cm]oy1.west) {\scriptsize{Representation}};
\draw [->] ([yshift=-0.1cm]oyt.north) -- ([yshift=0.18cm]oyt.north);
\draw [->] ([yshift=-0.1cm]oy1.north) -- ([yshift=0.18cm]oy1.north);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-importance-sampling-bad-sampling.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\tikzset{
block/.style = {
minimum width=3.5cm,minimum height=1.5cm,
draw=#1,fill=white,line width=2pt,
drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex}
},
block/.default=black
}
\def\sep{1.5cm}
\def\textwid{7.0cm}
\begin{scope}
\node [anchor=east] at (0,0) {\large{$\cdots$}};
\node [anchor=west, block=lolred] (llm) at (\sep/2,0) {Reference Policy (LLM)};
\draw [->] (0,0) -- (\sep/2,0);
\node [anchor=north] (x) at ([yshift=-.5cm]llm.south) {\large{$\mathbf{x}$}};
\node [draw, dashed, rounded corners=2pt,anchor=south, align=left, text width=\textwid, minimum height=2.6cm] (output) at ([yshift=.5cm]llm.north) {\footnotesize{There are three tips }\footnotesize{\sethlcolor{red!20}\hl{for improving for improving for improving} \footnotesize{accuracy in solving math problems:}} \\
\footnotesize{1.Practice regularly. \\
2.Understand the concepts. \\
3.Double-check your work.}};
\node [draw, dashed, rounded corners=2pt,anchor=north, align=left, text width=\textwid] (input) at ([yshift=0cm]x.south) {\footnotesize{Give me three tips to improve my accuracy in solving math problems.}};
\node [anchor=south] (y) at ([yshift=0cm]output.north) {\large{$\mathbf{y}$}};
\node [align=left] (step) at ([yshift=-.5cm]input.south) {\footnotesize{sampling in a terrible step}};
\draw [->] (x.north) -- ([yshift=-.1cm]llm.south);
\draw [<-] ([yshift=-.1cm]output.south) -- ([yshift=.1cm]llm.north);
%%
\node [block=lolred,anchor=west] (llm1) at ([xshift=\sep*3]llm.east) {Reference Policy (LLM)};
\node [anchor=north] (x) at ([yshift=-.5cm]llm1.south) {\large{$\mathbf{x}$}};
\node [draw, dashed, rounded corners=2pt,anchor=south, align=left, text width=\textwid, minimum height=2.6cm] (output) at ([yshift=.5cm]llm1.north) {\footnotesize{There are three tips }\footnotesize{\sethlcolor{red!20}\hl{for improving improving improving improving improving improving improving improving } \footnotesize{accuracy in solving math problems:}} \\
\footnotesize{1.Practice }\sethlcolor{red!20}\hl{regularly regularly regularly regularly regularly regularly regularly }. \footnotesize{...}\\};
\node [draw, dashed, rounded corners=2pt,anchor=north, align=left, text width=\textwid] (input) at ([yshift=0cm]x.south) {\footnotesize{Give me three tips to improve my accuracy in solving math problems.}};
\node [anchor=south] (y) at ([yshift=0cm]output.north) {\large{$\mathbf{y}$}};
\node [align=left] (step) at ([yshift=-.5cm]input.south) {\footnotesize{sampling after a terrible step}};
\draw [->] (x.north) -- ([yshift=-.1cm]llm1.south);
\draw [<-] ([yshift=-.1cm]output.south) -- ([yshift=.1cm]llm1.north);
% \node [anchor=north, align=left] (step) at (llm1.south |- step.north) {\footnotesize{xxxxxx}};
\node [anchor=west] (llm2) at ([xshift=\sep/2]llm1.east) {\large{$\cdots$}};
\draw [->] (llm.east) -- node [midway, align=center] {\footnotesize{resync the reference policy} \\\footnotesize{after a terrible step}} (llm1.west);
\draw [->] (llm1.east) -- (llm2.west);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-importance-sampling.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\tikzset{
block/.style = {
minimum width=3.5cm,minimum height=1.5cm,
draw=#1,fill=white,line width=2pt,
drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex}
},
block/.default=black
}
\def\sep{1.3cm}
\begin{scope}
\node [block=lolblue] (llm) at (0,0) {Policy (LLM)};
\node [anchor=south, block=lolred] (old llm) at ([yshift=1cm]llm.north) {Reference Policy (LLM)};
\draw [->] ([yshift=.1cm]llm.north) -- node [midway,right] {copy} ([yshift=-.1cm]old llm.south);
\node [align=left] (step) at ([yshift=-.6cm]llm.south) {\scriptsize{Step (1): resync the reference policy}};
%%
\node [block=lolred,anchor=west] (old llm1) at ([xshift=\sep]$(old llm.east)!.5!(llm.east)$) {Reference Policy (LLM)};
\node [anchor=north] (x) at ([yshift=-.5cm]old llm1.south) {\large{$\mathbf{x}$}};
\node [anchor=south] (y) at ([yshift=.5cm]old llm1.north) {\large{$\mathbf{y}$}};
\draw [->] (x.north) -- ([yshift=-.1cm]old llm1.south);
\draw [<-] (y.south) -- ([yshift=.1cm]old llm1.north);
\node [anchor=north, align=left] (step) at (old llm1.south |- step.north) {\scriptsize{Step (2): sample an output from}\\ \scriptsize{the reference policy}};
%%
\node [block=lolblue,anchor=west] (llm2) at ([xshift=\sep]old llm1.east) {Policy (LLM)};
\node [anchor=north] (x) at ([xshift=-1.5cm, yshift=-.5cm]llm2.south) {\large{$\mathbf{x}$}};
\node [anchor=north] (y) at ([xshift=-1.1cm, yshift=-.5cm]llm2.south) {\large{$\mathbf{y}$}};
\node [align=center] at ([yshift=-.3cm]$(x.south)!.5!(y.south)$) {\scriptsize{1st time}\\[-2pt]\scriptsize{update}};
\draw [->] (x.north) -- ([yshift=-.1cm]x.north|-llm2.south);
\draw [->] (y.north) -- ([yshift=-.1cm]y.north|-llm2.south);
\node [anchor=north] (x) at ([xshift=-.2cm, yshift=-.5cm]llm2.south) {\large\textcolor{black!40!white}{$\mathbf{x}$}};
\node [anchor=north] (y) at ([xshift=.2cm, yshift=-.5cm]llm2.south) {\large\textcolor{black!40!white}{$\mathbf{y}$}};
\node [align=center] at ([yshift=-.3cm]$(x.south)!.5!(y.south)$) {\scriptsize{2nd time}\\[-2pt]\scriptsize{update}};
\draw [->,black!40!white] (x.north) -- ([yshift=-.1cm]x.north|-llm2.south);
\draw [->,black!40!white] (y.north) -- ([yshift=-.1cm]y.north|-llm2.south);
\node [anchor=north] (cdots) at ([xshift=1.3cm, yshift=-.5cm]llm2.south) {\large$\cdots$};
\node [align=center] at ([yshift=-.3cm]cdots.south) {\large$\cdots$};
\node [anchor=north,align=left] (step) at (llm2.south |- step.north) {\scriptsize{Step (3): update the policy multiple}\\ \scriptsize{times and go to Step (1)}};
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-one-example.tex 0 → 100644
\begin{center}
\def\tgtwidth{0.5\textwidth}
\begin{tikzpicture}
\tikzstyle{ynode} = [draw,rounded corners=4pt, minimum width=\tgtwidth,inner sep=4pt,align=left,text width=\tgtwidth]
\begin{scope}
\node (input) at (0,0) {$\mathbf{x}$};
\node [ynode, anchor=west] (note) at ([xshift=.2cm]input.east) {
\begin{varwidth}{\textwidth}
\scriptsize{Give me three tips to improve my accuracy in solving math problems.}
\end{varwidth}
};
\node [ynode,anchor=north west] (output1) at ([yshift=-.2cm]note.south west) {
\scriptsize{There are three tips for improving accuracy in solving math problems: \\
1.Practice regularly. \\
2.Understand the concepts. \\
3.Double-check your work. \\
\textbf{\{Total Tokens: 20\}}}
};
\node [anchor=east](y1) at ([xshift=-.2cm]output1.west) {$\mathbf{y}_1$};
\node [ynode,anchor=north] (output2) at ([yshift=-.2cm]output1.south) {
\scriptsize{Improving accuracy in solving math problems is crucial for success in mathematics. Here are three tips that can help: \\
1.Practice regularly: \\
~~~- Consistent practice builds fluency and familiarity with different problem types. \\
... \\
By combining your advice with these additional tips, anyone can significantly enhance their math problem-solving abilities. \\
\textbf{\{Total Tokens: 320\}}}
};
\node [anchor=east](y2) at ([xshift=-.2cm]output2.west) {$\mathbf{y}_2$};
\node [ynode,anchor=north] (output3) at ([yshift=-.2cm]output2.south) {
\scriptsize{Improving math accuracy requires careful attention to detail, strategic problem-solving techniques, and consistent practice. Here are three tips: \\
1.Practice regularly: \\
~~~- Mathematics is a skill, and like any skill, it improves with consistent practice.... \\
... \\
This is a great approach for anyone looking to improve their math skills, from students to professionals. \\
\textbf{\{Total Tokens: 800\}}}
};
\node [anchor=east](y3) at ([xshift=-.2cm]output3.west) {$\mathbf{y}_3$};
\begin{axis}[
compat=1.3,
anchor=north west,
at={(output1.north east)},
axis x line=bottom,axis y line=left,
xshift=1.4cm,
yshift=-1cm,
xtick={0,1,2,3},
xticklabels={$\mathbf{y}_1$,$\mathbf{y}_2$,$\mathbf{y}_3$,$\cdots$},
every tick label/.append style={font=\footnotesize},
every axis label/.append style={font=\footnotesize},
xtick distance=.1cm,
ymin=0,
ymax=.6,
ylabel=$\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$,
ylabel shift=-.1cm,
enlarge x limits=0.3,
width=5cm,
height=4cm,
ybar]
\addplot[fill=lolblue,draw=none] coordinates {
(0, .5)
(1, .3)
(2, .1)
(3, .1)
};
\end{axis}
\begin{axis}[
compat=1.3,
anchor=south west,
at={(output3.south east)},
axis x line=bottom,axis y line=left,
xshift=1.4cm,
yshift=.5cm,
xtick={0,1,2,3},
xticklabels={$\mathbf{y}_1$,$\mathbf{y}_2$,$\mathbf{y}_3$,$\cdots$},
every tick label/.append style={font=\footnotesize},
every axis label/.append style={font=\footnotesize},
xtick distance=.1cm,
ymin=0,
ymax=.6,
ylabel=$\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$,
ylabel shift=-.1cm,
enlarge x limits=0.3,
width=5cm,
height=4cm,
ybar]
\addplot [fill=lolblue,draw=none] coordinates {
(0, .1)
(1, .3)
(2, .5)
(3, .1)
};
\end{axis}
\matrix [
matrix anchor=north,
matrix of nodes,
nodes={inner sep=2pt},
column sep=.1cm,
inner sep=0,
] at ([xshift=3cm,yshift=0cm]note.north east) {
\hspace{-0.25cm} \footnotesize{$R(\mathbf{y}_1)=2$} \\
\footnotesize{$R(\mathbf{y}_2)=32$} \\
\footnotesize{$R(\mathbf{y}_3)=80$} \\
};
\node at ([xshift=2.03cm,yshift=-1.2cm]output1.north east) {\color{lolred}{\faTimes}};
\node at ([xshift=3.5cm,yshift=-2.85cm]output1.north east) {\color{lolgreen}{\faCheck}};
\node at ([xshift=2.03cm,yshift=1.1cm]output3.south east) {\color{lolred}{\faTimes}};
\node at ([xshift=3.5cm,yshift=+2.70cm]output3.south east) {\color{lolgreen}{\faCheck}};
\draw [->,thick] ([xshift=3cm,yshift=-4cm]output1.north east) -- node [right,align=left] {
\scriptsize{Increase the probability} \\[-1mm]
\scriptsize{of longer outputs;} \\ [-1mm]
\scriptsize{Decrease the probability} \\[-1mm]
\scriptsize{of shorter outputs}
}([xshift=3cm,yshift=3.2cm]output3.south east);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-pairwise-reward-loss-expectation.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{.7cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
\begin{scope}
\node [draw,thick,minimum width=5.2cm,minimum height=1.3cm,fill=white] (llm) at (0,0) {Reward Model (LLM)};
\node [anchor=south east] (oy last) at ([yshift=.5cm,xshift=0cm]llm.north east) {\ctext[RGB]{153,204,255}{$R_{\phi}(\mathbf{x},\mathbf{y}_a)$}};
\draw [->] ([yshift=.1cm]llm.north-|oy last.south) -- (oy last.south);
\node [draw,rounded corners=2pt,inner sep=2pt,text width=5.5em, anchor=north west,font=\linespread{0.8}\selectfont, minimum height=9ex] (x) at ([yshift=-1cm,xshift=-1cm]llm.south west) {\scriptsize{Give me three tips to improve my accuracy in solving math problems.\\}};
\node [draw,rounded corners=2pt,inner sep=2pt,text width=13.5em, anchor=north east,font=\linespread{0.8}\selectfont, minimum height=9ex] (y) at ([yshift=-1cm,xshift=1cm]llm.south east) {\scriptsize{Improving math accuracy requires careful attention to detail, ... \\
1.Practice regularly: ... \\
... , anyone can significantly enhance their math problem-solving abilities.\\}};
\node [anchor=north] at ([yshift=-.2cm]x.south) {$\mathbf{x}$};
\node [anchor=north] at ([yshift=-.2cm]y.south) {$\mathbf{y}_a$};
\draw [<-] ([yshift=0cm,xshift=-.5cm]llm.south) .. controls +(0, -.95cm) and +(0,+.95cm) .. +(-2cm, -.95cm);
\draw [<-] ([yshift=0cm,xshift=.5cm]llm.south) .. controls +(0, -.95cm) and +(0,+.95cm) .. +(+2cm, -.95cm);
%%
\node [draw,thick,minimum width=5.2cm,minimum height=1.3cm,fill=white] (llm) at (+8cm,0) {Reward Model (LLM)};
\node [anchor=south east] (oy last1) at ([yshift=.5cm,xshift=0cm]llm.north east) {\ctext[RGB]{255,204,204}{$R_{\phi}(\mathbf{x},\mathbf{y}_b)$}};
\draw [->] ([yshift=.1cm]llm.north-|oy last1.south) -- (oy last1.south);
\node [draw,rounded corners=2pt,inner sep=2pt,text width=5.5em, anchor=north west,font=\linespread{0.8}\selectfont, minimum height=9ex] (x) at ([yshift=-1cm,xshift=-1cm]llm.south west) {\scriptsize{Give me three tips to improve my accuracy in solving math problems.\\}};
\node [draw,rounded corners=2pt,inner sep=2pt,text width=13.5em, anchor=north east,font=\linespread{0.8}\selectfont, minimum height=9ex] (y) at ([yshift=-1cm,xshift=1cm]llm.south east) {\scriptsize{There are three tips for improving accuracy in solving math problems: \\
1.Practice regularly. \\
2.Understand the concepts.\\
3.Double-check your work.\\}};
\node [anchor=north] at ([yshift=-.2cm]x.south) {$\mathbf{x}$};
\node [anchor=north] at ([yshift=-.2cm]y.south) {$\mathbf{y}_b$};
\draw [<-] ([yshift=0cm,xshift=-.5cm]llm.south) .. controls +(0, -.95cm) and +(0,+.95cm) .. +(-2cm, -.95cm);
\draw [<-] ([yshift=0cm,xshift=.5cm]llm.south) .. controls +(0, -.95cm) and +(0,+.95cm) .. +(+2cm, -.95cm);
\node at (4.65cm,3.3cm) {$- \log \mathrm{Sigmoid}($\ctext[RGB]{153,204,255}{$R_{\phi}(\mathbf{x},\mathbf{y}_a)$}$-$\ctext[RGB]{255,204,204}{$R_{\phi}(\mathbf{x},\mathbf{y}_b)$}$)$};
\draw [->] (oy last.north) .. controls +(0, 1cm) and +(0, -1cm) .. ([xshift=3.1cm, yshift=1.25cm]oy last.north);
\draw [->] (oy last1.north) .. controls +(0, 1cm) and +(0, -1cm) .. ([xshift=-3.1cm, yshift=1.25cm]oy last1.north);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-ppo-clip.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\begin{scope}
\begin{axis}[
compat=1.3,
anchor=south east,
at={(0,0)},
axis x line=bottom,axis y line=left,
xtick={0,1,1.2,2},
xticklabels={,1,$1+\epsilon$,},
ytick=\empty,
every tick label/.append style={font=\scriptsize},
every axis label/.append style={font=\scriptsize},
xtick distance=.1cm,
ymin=0,
ymax=3,
enlarge x limits=0.0,
width=6.4cm,
height=6.5cm,
clip=false,]
\addplot [mark=none,thick] coordinates {
(0, 0)
(1, 1.5)
(1.2, 1.8)
(2, 1.8)
};
\addplot [mark=none,dashed,line width=0.5pt] coordinates {
(1, 0)
(1, 3)
};
\addplot [mark=none,dashed,lolred,line width=1.5pt] coordinates {
(1.2, 0)
(1.2, 3)
};
\node [anchor=south west] at (0,4.8cm) {$\textrm{Clip}(\cdot)$};
\node [anchor=north] at (4.8cm, 0cm) {\footnotesize{ratio}};
\node [anchor=north] at (0,0) {0};
\node (caption a) at (xticklabel cs:0.5) {};
\end{axis}
\node (tmp) at (0,0) {};
\node [anchor=north] (a) at ([yshift=-.8cm]caption a|-tmp) {\footnotesize{(a) $A_t>0$}};
\begin{axis}[
compat=1.3,
anchor=south west,
at={(+3cm,0)},
axis x line=top,axis y line=left,
xtick={0,0.8,1,2},
xticklabels={,$1-\epsilon$,1,},
ytick=\empty,
every tick label/.append style={font=\scriptsize},
every axis label/.append style={font=\scriptsize},
xtick distance=.1cm,
ymin=0,
ymax=3,
enlarge x limits=0.0,
width=6.4cm,
height=6.5cm,
y dir=reverse,
clip=false,]
\addplot [mark=none,thick] coordinates {
(0, 1.2)
(0.8, 1.2)
(1.0, 1.5)
(1.8, 2.7)
};
\addplot [mark=none,dashed,lolred,line width=1.5pt] coordinates {
(0.8, 0)
(0.8, 3)
};
\addplot [mark=none,dashed,line width=0.5pt] coordinates {
(1, 0)
(1, 3)
};
\node [anchor=south west] at (0,-5.4cm) {$\textrm{Clip}(\cdot)$};
\node [anchor=south] at (4.8cm, 0) {\footnotesize{ratio}};
\node [anchor=south] at (0,0) {0};
\node (caption b) at (xticklabel cs:0.5) {};
\end{axis}
\node [anchor=north] (b) at ([yshift=-.8cm]caption b|-tmp) {\footnotesize{(b) $A_t<0$}};
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-reward-model.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{1.5cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
\begin{scope}
\node [anchor=center] (x1) at (0,0) {\footnotesize{$x_1$}};
\node [anchor=center] (xcdots) at ([xshift=\ssep]x1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (xm) at ([xshift=\ssep]xcdots.center) {\footnotesize{$x_m$}};
\node [anchor=center] (y1) at ([xshift=\ssep]xm.center) {\footnotesize{$y_1$}};
\node [anchor=center] (ycdots) at ([xshift=\ssep]y1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (yt) at ([xshift=\ssep]ycdots.center) {\footnotesize{$y_{T}$}};
\foreach \x in {x1, xm, y1, yt}
\draw [->] ([yshift=0.2cm]\x.center) -- ([yshift=0.55cm]\x.center);
\node [anchor=center] (oyt) at ([yshift=2.6cm]yt.center) {\footnotesize{$\mathbf{h}_{y_T}$}};
\node [anchor=south,draw,thick,minimum width=9cm,minimum height=1.3cm] (llm) at ([yshift=0.4cm]$(xm.north)!.5!(y1.north)$) {\large{Reward Model (LLM)}};
\foreach \x in {oyt}
\draw [<-] ([yshift=-0.25cm]\x.center) -- ([yshift=-0.6cm]\x.center);
\filldraw [fill=red!20,draw=white] ([xshift=-0.5cm,yshift=0.2cm]oyt.north west) -- ([xshift=0.5cm,yshift=0.2cm]oyt.north east) -- ([yshift=1cm,xshift=0]oyt.north east) -- ([yshift=1cm,xshift=0]oyt.north west) -- ([xshift=-0.5cm,yshift=0.2cm]oyt.north west);
\node [anchor=south] (Wr) at ([yshift=0.3cm]oyt.north) {\footnotesize{$\mathbf{W}_r$}};
\node [anchor=south] (output) at ([yshift=0.5cm]Wr.north) {\footnotesize{Reward (Scalar)}};
\node [anchor=west] (linear) at ([xshift=0.5cm]Wr.east) {\scriptsize{Linear Map}};
\node [anchor=west, align=left,xshift=-0.4cm] at (linear.west|-oyt.east) {\scriptsize{Representation}};
\draw [->] ([yshift=0.2cm]Wr.north) -- (output.south);
\draw [->] ([yshift=-0.1cm]oyt.north) -- ([yshift=0.18cm]oyt.north);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-rlhf.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\tikzstyle{model} = [minimum width=2cm, minimum height=1.2cm,align=center,inner sep=2pt,draw,thick,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex},fill=white]
\tikzstyle{label} = [minimum width=2ex, minimum height=1.5ex,draw,scale=.75,fill=white]
\tikzstyle{circled} = [shape=circle,draw,inner sep=2pt]
\begin{scope}
\node [anchor=west] (x) at (-0.2cm,0) {$\mathbf{x}$};
\node [anchor=west,model] (policy model) at ([xshift=.8cm]x) {Policy\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]policy model.north west) {To Learn};
\node [anchor=west] (y) at ([xshift=1.2cm]policy model.east) {$\mathbf{y}$};
\node [anchor=west,model] (reward model) at ([xshift=1.4cm]y.east) {Reward\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]reward model.north west) {Fixed};
\node [anchor=south,model] (reference model) at ([yshift=.4cm]reward model.north) {Reference\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]reference model.north west) {Fixed};
\node [anchor=north,model] (value model) at ([yshift=-.4cm]reward model.south) {Value\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]value model.north west) {To Learn};
\node [anchor=west] (reward) at ([xshift=0.1cm]reward model.east) {$R_\phi(\mathbf{x}, \mathbf{y})$};
\node [anchor=west] (value) at ([xshift=0.1cm]value model.east) {$\{V_{\omega,1}, \cdots, V_{\omega,T}\}$};
\node [anchor=west] (reference) at ([xshift=0.1cm]reference model.east) {Penalty};
\node [anchor=west,align=center,rounded corners=5pt,inner sep=12pt,fill=lightgray] (advantage estimation) at ([xshift=1.5cm]reward) {Advantage\\Estimation};
\node [anchor=west] (advantage estimation out) at (advantage estimation.east) {$\{A_{1}, \cdots, A_{T}\}$};
\node (left bottom) at (x.west|-value model.south) {};
\node (right bottom) at (advantage estimation out.east|-value model.south) {};
\node [anchor=north,minimum width=13.6cm, minimum height=2cm,rounded corners=8pt,draw=lightgray,thick] (note) at ($(left bottom)!.5!(right bottom)+(0, -1cm)$) {};
\matrix [anchor=center, every even column/.style={column sep=1cm}] at (note.center) {
\node {\circled{1}}; & \node [anchor=west, inner sep=0pt] {sample an output $\mathbf{y}$ from $\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot|\mathbf{x})$} ; & \node {\circled{2}}; & \node [anchor=west, inner sep=0pt] {compute the $\mathrm{Penalty}$} ; \\
\node {\circled{3}}; & \node [anchor=west, inner sep=0pt] {compute the reward for $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$ } ; & \node {\circled{4}}; & \node [anchor=west, inner sep=0pt] {predict the value for $\mathrm{seq}_{\mathbf{x}, \mathbf{y}}$} ; \\
\node {\circled{5}}; & \node [anchor=west, inner sep=0pt] {optimize the value model $V_{\omega}(\cdot)$} ; & \node {\circled{6}}; & \node [anchor=west, inner sep=0pt] {optimize the policy model} ; \\
};
\draw [->,thick] (x.east) -- (policy model.west);
\draw [->,thick] (policy model.east) node [xshift=.5cm,yshift=.0cm,above,align=center] {\circled{1}} -- (y.west);
\draw [->,thick] (y.east) .. controls (reward model.west) and ($(y.east |- reference model.west)$) .. (reference model.west);
\draw [->,thick] (y.east) .. controls (reward model.west) and ($(y.east |- value model.west)$) .. (value model.west);
\draw [->,thick] (y.east) -- (reward model.west);
\draw [->,thick] (reference.east) node [above,xshift=.8cm] {\circled{2}} .. controls (reference.east -| advantage estimation.north) .. (advantage estimation.north);
\draw [->,thick] (reward.east) -- node [above,xshift=0.0cm] {\circled{3}} (advantage estimation.west);
\draw [->,thick] (value.east) .. controls (value.east -| advantage estimation.south) .. node [above,xshift=-.2cm] {\circled{4}}(advantage estimation.south);
% update model
\node (policy model bottom) at ($(right bottom -| policy model)+(0,-.6cm)$) {};
\node [anchor=north] (policy model bottom1) at (policy model bottom |- advantage estimation out.north) {};
\node (value model bottom) at ($(right bottom -| value model)+(0,-.6cm)$) {};
\node (advantage estimation out bottom) at ($(advantage estimation out |- right bottom)+(0,-.6cm)$) {};
\draw [->,thick,dashed] (advantage estimation out) -- ($(advantage estimation out)!.5!(advantage estimation out bottom)$) .. controls (advantage estimation out bottom) .. ($(advantage estimation out bottom.center)!.5!(policy model bottom.center)+(5em,0)$) -- ($(advantage estimation out bottom.center)!.5!(policy model bottom.center)-(5em,0)$) .. controls (policy model bottom.center) .. ($(policy model bottom1.center)!.5!(policy model bottom)$) -- node [above,xshift=0.3cm,yshift=-.2cm,solid] {\circled{6}}(policy model.south);
\draw [->,thick,dashed] ($(advantage estimation out bottom.center)!.5!(policy model bottom.center)+(2.89em,0)$) node [above,xshift=-1.5cm,solid] {\circled{5}}.. controls (value model bottom) .. (value model.south);
\end{scope}
\end{tikzpicture}
\end{center}
section3/Figures/figure-update-with-policy-gradient.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{.7cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
\begin{scope}
\node [anchor=west] (x0) at (0,0) {\footnotesize{$x_0$}};
\node [anchor=center] (x cdots) at ([xshift=\ssep]x0.center) {};
\node [anchor=center] (x m) at ([xshift=\ssep]x cdots.center) {};
\node [anchor=center] (y1) at ([xshift=\ssep]x m.center) {\footnotesize{$x_{m}$}};
\node (x real cdots) at ($(x cdots)!.5!(x m)$) {\footnotesize{$\cdots$}};
\node [anchor=center] (y2) at ([xshift=\ssep]y1.center) {};
\node [anchor=center] (y cdots) at ([xshift=\ssep]y2.center) {};
\node [anchor=center] (y n) at ([xshift=\ssep]y cdots.center) {};
\foreach \x in {x0,y1}
\draw [->] ([yshift=0.2cm]\x.center) -- ([yshift=0.55cm]\x.center);
\node [anchor=center] (oy1) at ([yshift=2.8cm]y1.center) {\footnotesize{$y_1$}};
\node [anchor=center] (oy2) at ([yshift=2.8cm]y2.center) {};
\node [anchor=center] (oy cdots) at ([yshift=2.8cm]y cdots.center) {};
\node [anchor=center] (oy last) at ([yshift=2.8cm]y n.center) {\footnotesize{$y_T$}};
\node (oy real cdots) at ($(oy cdots)!.5!(oy2)$) {\footnotesize{$\cdots$}};
\draw [decorate, decoration={brace,amplitude=.2cm},thick] (oy1.north west) -- (oy last.north east);
\node [anchor=center] (oy) at ([yshift=.7cm]$(oy2)!.5!(oy cdots)$) {\footnotesize{$\mathbf{y}$}};
\foreach \x in {oy1, oy last}
\draw [<-] ([yshift=-0.25cm]\x.center) -- ([yshift=-0.8cm]\x.center);
\node [anchor=south,draw,thick,minimum width=5.2cm,minimum height=1.3cm,fill=white,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex}] (llm) at ([yshift=.6cm]y1.center) {Policy (LLM)};
\node [anchor=center] (freeze) at ([xshift=-.25cm,yshift=.25cm]llm.south east) {\textcolor{cyan!60}{ \faSnowflake}};
\node (rside) at ([xshift=0.5cm,yshift=-1.2cm]oy last.north east|-llm.east) {};
\node (lside) at ([yshift=-1.2cm]llm.west|-y n.east) {};
\node at ($(lside)!.5!(rside)$) {\footnotesize{(a)~sample an output $\mathbf{y}$}};
\node [anchor=west] (x0) at (\textwidth/2,0) {\footnotesize{$x_0$}};
\node [anchor=center] (x cdots) at ([xshift=\ssep]x0.center) {};
\node [anchor=center] (x m) at ([xshift=\ssep]x cdots.center) {};
\node [anchor=center] (y1) at ([xshift=\ssep]x m.center) {\footnotesize{$x_{m}$}};
\node (x real cdots) at ($(x cdots)!.5!(x m)$) {\footnotesize{$\cdots$}};
\node [anchor=center] (y2) at ([xshift=\ssep]y1.center) {};
\node [anchor=center] (y cdots) at ([xshift=\ssep]y2.center) {};
\node [anchor=center] (y n) at ([xshift=\ssep]y cdots.center) {$y_{T-1}$};
\node (y real cdots) at ($(y2)!.5!(y cdots)$) {\footnotesize{$\cdots$}};
\foreach \x in {x0,y1, y n}
\draw [->] ([yshift=0.2cm]\x.center) -- ([yshift=0.55cm]\x.center);
\node [anchor=center] (oy1) at ([yshift=2.8cm]y1.center) {};
\node [anchor=center] (oy2) at ([yshift=2.8cm]y2.center) {};
\node [anchor=center] (oy cdots) at ([yshift=2.8cm]y cdots.center) {};
\node [anchor=center] (oy last) at ([yshift=2.8cm]y n.center) {};
\node [anchor=south,draw,thick,minimum width=5.2cm,minimum height=1.3cm,fill=white,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex}] (llm) at ([yshift=.6cm]y1.center) {Policy (LLM)};
\node [anchor=center] (freeze) at ([xshift=-.25cm,yshift=.25cm]llm.south east) {\textcolor{red!60}{ \faFire}};
\node [anchor=east,inner sep=0] (prob) at ([xshift=1em]oy last.east-|llm.east) {\ctext[RGB]{255,204,204}{$\log\left(\textrm{Pr}_\theta (y_1|\textbf{x})\times\cdots\times\textrm{Pr}_\theta (y_T|\textbf{y}_{<T},\textbf{x})\right)$}};
\node [anchor=west,inner sep=0] (reward) at ([xshift=.2cm]prob.east) {\ctext[RGB]{153,204,255}{$R(\mathbf{y})$}};
\node [anchor=center] at ($(prob.east)!.5!(reward.west)$) {$\times$};
\node (rside) at ([xshift=0.5cm,yshift=-1.2cm]oy last.north east|-llm.east) {};
\node (lside) at ([yshift=-1.2cm]llm.west|-y n.east) {};
\node at ($(lside)!.5!(rside)$) {\footnotesize{(b)~train the LLM via policy gradient}};
\draw [<-] ([yshift=-.25cm,xshift=-1.35cm]oy1.center) .. controls ([yshift=-.525cm,xshift=-1.35cm]oy1.center) .. ([yshift=-.525cm,xshift=-1cm]oy1.center) -- ([yshift=-.525cm,xshift=-.35cm]oy1.center) .. controls ([yshift=-.525cm]oy1.center) .. ([yshift=-.8cm]oy1.center);
\draw [<-] ([yshift=-.25cm, xshift=-.7cm]oy last.center) .. controls ([yshift=-.525cm,xshift=-.7cm]oy last.center) .. ([yshift=-.525cm,xshift=-.35cm]oy last.center) .. controls ([yshift=-.525cm]oy last.center) .. ([yshift=-.8cm]oy last.center);
\draw [->,thick] (oy.north) -- +(0,.6cm) -- ([yshift=.6cm]oy.north-|reward) node [midway,above] {compute the reward: \ctext[RGB]{153,204,255}{$R(\mathbf{y})$}} -- ([yshift=.2cm]reward.north);
\end{scope}
\end{tikzpicture}
\end{center}
section3/algorithm.tex 0 → 100644
\begin{algorithm}[t]
\caption{PPO-based Optimization Workflow}
\begin{algorithmic}[1]
\INPUT the optimized reward model $r_{\phi}(\cdot)$, the reference model $\pi_{\theta_{\mathrm{ref}}}(\cdot)$, the initialized value model $V_{\omega}(\cdot)$, the policy model $\pi_{\theta}(\cdot)$, the input-only dataset $S_{x}$;
\OUTPUT the aligned $\pi_{\theta}(\cdot)$;
\State \%\%\% suppose the batch size is set to 1
\For{$\mathbf{x} \in S_x$}
\State $\pi_{\theta_\mathrm{old}(\cdot)} = \mathrm{Clone}(\pi_{\theta}(\cdot))$
\State sample an output from $\pi_{\theta_\mathrm{old}}$: $\mathbf{y} \sim \pi_{\theta_\mathrm{old}}(\cdot|\mathbf{x})$
\State compute the reward for $[\mathbf{x},\mathbf{y}]$: $r_{\phi}(\mathbf{x},\mathbf{y})$
\State \%\%\% roll out
\For{ppo\_epoch\_idx=1 to $\mathrm{PPO\_EPOCH}$}
\State \%\%\% \textit{compute policy model loss and value model loss through advantages and returns}
\State initialize the loss values: $\mathrm{loss}_{p} = 0$, $\mathrm{loss}_{v} = 0$
\For{t=1 to T}
\State compute the penalty through Eq. (\ref{eq:penalty}): $\mathrm{Penalty}_{t}=\log \mathrm{Pr}_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t}) - \log \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t|\mathbf{x},\mathbf{y}_{<t})$
\State \textbf{if} t==T \textbf{then}
\State \hspace{0.5cm} $r_{t} = r_{\phi}(\mathbf{x}, \mathbf{y}) -\beta \mathrm{Penalty}_{t}$
\State \textbf{else}
\State \hspace{0.5cm} $r_{t} = -\beta \mathrm{Penalty}_{t}$
\State \textbf{end if}
\State predict the value $V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State \textbf{if} t==T \textbf{then}
\State \hspace{0.5cm} $V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})=0$
\State \textbf{else}
\State \hspace{0.5cm} predict the next time step value $V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})$
\State \textbf{end if}
\State \%\%\% \textit{compute policy model loss}
\State compute the advantage $A(\mathbf{x},\mathbf{y}_{<t},y_{t})$ through Eq. (\ref{eq:advantage}): $A(\mathbf{x},\mathbf{y}_{<t},y_{t})=r_{t}+\gamma V_{\omega}(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the policy model loss through Eq. (\ref{eq:ppo-loss}) and add it to $\mathrm{loss}_{p}$: $\mathrm{loss}_{p} = \mathrm{loss}_{p} + \mathrm{Clip}\Big( \frac{\pi_{\theta}(y_t|\mathbf{x},\mathbf{y}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|\mathbf{x},\mathbf{y}_{<t})} \Big) A(\mathbf{x},\mathbf{y}_{<t},y_t)$
\State \%\%\% \textit{compute value model loss}
\State compute the return $\mathrm{Return}_{t}$: $\mathrm{Return}_{t}=A(\mathbf{x},\mathbf{y}_{<t},y_t)+V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t})$
\State compute the value model loss through Eq. (\ref{eq:value-loss}) and add it to $\mathrm{loss}_{v}$: $\mathrm{loss}_v = \mathrm{loss}_v + (\mathrm{Return}_{t}-V_{\omega}(\mathbf{x},\mathbf{y}_{<t},y_{t}))^2$
\EndFor
\State update the parameters of $\pi_{\theta}(\cdot)$ through $\mathrm{loss}_{p}$
\State update the parameters of $V_{\omega}(\cdot)$ through $\mathrm{loss}_{v}$
\EndFor
\EndFor
\State return $\pi_{\theta}$
\end{algorithmic}
\label{alg:rlhf}
\end{algorithm}
\ No newline at end of file
section4/Figures/figure-an-overlap-example-dense-reward-model.tex 0 → 100644
\begin{center}
\def\tgtwidth{0.8\textwidth}
\begin{tikzpicture}
\tikzstyle{ynode} = [fill=black!5!white,rounded corners=5pt,minimum width=\tgtwidth,inner sep=4pt,align=left,text width=\tgtwidth]
\begin{scope}
\node (input) at (0,0) {\large{$\mathbf{x}$}};
\node [ynode, anchor=west] (note) at ([xshift=.2cm]input.east) {
\begin{varwidth}{\textwidth}
\footnotesize{Give me three tips to improve my accuracy in solving math problems.}
\end{varwidth}
};
\node [ynode,anchor=north west] (output1) at ([yshift=-.2cm]note.south west) {
\setstretch{2.2}\\\footnotesize{There are three tips for improving accuracy in solving math problems: \\
1.Practice regularly. \\
2.Understand the concepts. \\
3.Double-check your work. \\}
};
\node [anchor=east](y1) at ([xshift=-.2cm]output1.west) {\large{$\mathbf{y}$}};
\node [fill=lolred!50, anchor=south west,minimum height=.45cm,inner sep=1pt] (label1) at ([xshift=6.85cm,yshift=-.65cm]output1.north west) {\scriptsize{Intermediate Reward: +1}};
\draw [-,lolred!50,line width=1.5pt] ([yshift=-.2pt]label1.north west) -- ([yshift=-.90cm]label1.north west);
\node [fill=lolred!50, anchor=south west,minimum height=.45cm,inner sep=1pt] (label1) at ([xshift=4.05cm,yshift=-2.25cm]output1.north west) {\scriptsize{Intermediate Reward: +1}};
\draw [-,line width=1.5pt,lolred!50] ([yshift=-.2pt]label1.north west) -- ([yshift=-.78cm]label1.north west);
\node [fill=lolblue!50, anchor=south west,minimum height=.45cm,inner sep=1pt] (label1) at ([xshift=4.1cm,yshift=-3.05cm]output1.north west) {\scriptsize{Delayed Reward: +10.6}};
\draw [-,line width=1.5pt,lolblue!50] ([yshift=-.2pt]label1.north west) -- ([yshift=-.75cm]label1.north west);
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-dpo.tex 0 → 100644
\begin{center}
\def\tgtwidth{0.5\textwidth}
\begin{tikzpicture}
\tikzstyle{ynode} = [draw,rounded corners=4pt, minimum width=\tgtwidth,inner sep=4pt,align=left,text width=\tgtwidth]
\begin{scope}
\node (input) at (0,0) {$\mathbf{x}$};
\node [ynode, anchor=west] (note) at ([xshift=.2cm]input.east) {
\begin{varwidth}{\textwidth}
\scriptsize{Give me three tips to improve my accuracy in solving math problems.}
\end{varwidth}
};
\node [ynode,anchor=north west] (output1) at ([yshift=-.2cm]note.south west) {
\scriptsize{Improving accuracy in math problem-solving is crucial for success. Here are three tips to help you enhance your accuracy: \\
1. Focus on Understanding Concepts: Before diving into solving problems, ensure you have a solid understanding of the underlying concepts. Break down the problem into smaller parts and relate each part to the concepts you've learned. Understanding the theory behind the problem will help you choose the right approach to solve it accurately. \\
2. Practice Regularly: Like any other skill, math proficiency improves with practice. Work on a variety of problems regularly to build your problem-solving skills and increase your accuracy. Start with easier problems and gradually move on to more challenging ones. Consider using resources like textbooks, online problem sets, or math apps to practice. \\
3. Check and Review Your Work: After solving a problem, always double-check your work for any errors. Pay attention to details such as signs, decimal points, and units. If time permits, try solving the problem using a different method to verify your initial answer. By reviewing your work, you can catch and correct any mistakes, improving your overall accuracy.}\\};
\node [anchor=east](y1) at ([xshift=-.2cm]output1.west) {$\mathbf{y}_a$};
\node [ynode,anchor=north] (output2) at ([yshift=-.2cm]output1.south) {
\scriptsize{There are three tips for improving accuracy in solving math problems: \\
1.Practice regularly. \\
2.Understand the concepts. \\
3.Double-check your work.\\}
};
\node [anchor=east](y2) at ([xshift=-.2cm]output2.west) {$\mathbf{y}_b$};
\begin{axis}[
compat=1.3,
anchor=north west,
at={(note.north east)},
axis x line=bottom,axis y line=left,
xshift=1.4cm,
yshift=-.5cm,
xtick={0,1,2,3,4},
xticklabels={$\mathbf{y}_a$,$\mathbf{y}_1$,$\mathbf{y}_b$,$\mathbf{y}_2$,$\cdots$},
every tick label/.append style={font=\footnotesize},
every axis label/.append style={font=\footnotesize},
xtick distance=.1cm,
ymin=0,
ymax=.6,
ylabel=$\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$,
ylabel shift=-.1cm,
enlarge x limits=0.3,
width=5cm,
height=4cm,
ybar,
bar shift=0pt,]
\addplot[fill=lolred,draw=none] coordinates {
(2, .34)
(3, .24)
};
\addplot[fill=lolblue,draw=none] coordinates {
(0, .19)
(1, .15)
};
\addplot[fill=lightgray,draw=none] coordinates {
(4, .08)
};
\end{axis}
\begin{axis}[
compat=1.3,
anchor=south west,
at={(output2.south east)},
axis x line=bottom,axis y line=left,
xshift=1.4cm,
yshift=.5cm,
xtick={0,1,2,3,4},
xticklabels={$\mathbf{y}_a$,$\mathbf{y}_1$,$\mathbf{y}_b$,$\mathbf{y}_2$,$\cdots$},
every tick label/.append style={font=\footnotesize},
every axis label/.append style={font=\footnotesize},
xtick distance=.1cm,
ymin=0,
ymax=.6,
ylabel=$\mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$,
ylabel shift=-.1cm,
enlarge x limits=0.3,
width=5cm,
height=4cm,
ybar,
bar shift=0pt,]
\addplot[fill=lolred,draw=none] coordinates {
(2, .14)
(3, .04)
};
\addplot[fill=lolblue,draw=none] coordinates {
(0, .39)
(1, .35)
};
\addplot[fill=lightgray,draw=none] coordinates {
(4, .08)
};
\end{axis}
\node at ([xshift=2.35cm,yshift=-2.00cm]note.north east) {\scriptsize{Preferred}};
\node at ([xshift=3.45cm,yshift=-1.4cm]note.north east) {\scriptsize{Dispreferred}};
\node at ([xshift=2.35cm,yshift=2.25cm]output2.south east) {\scriptsize{Preferred}};
\node at ([xshift=3.45cm,yshift=1.22cm]output2.south east) {\scriptsize{Dispreferred}};
\draw [->,thick] ([xshift=3cm,yshift=-2.8cm]output1.north east) -- node [align=center,fill=white,text width=5.5cm,inner sep=2pt] {
\scriptsize{Minizing the negative probability: \\
$- \log\mathrm{Sigmoid} \big( \beta \log \frac{\mathrm{Pr}_{\theta}(\mathbf{y}_{a}|\mathbf{\mathbf{x}})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_{a}|\mathbf{\mathbf{x}})} - \beta \log \frac{\mathrm{Pr}_{\theta}(\mathbf{y}_{b}|\mathbf{\mathbf{x}})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_{b}|\mathbf{\mathbf{x}})} \big)$}
}([xshift=3cm,yshift=3.2cm]output2.south east);
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-examples-of-rule-based-reward-length.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}[background rectangle/.style={fill=black!5!white,rounded corners=5pt,minimum width=\textwidth}, show background rectangle]
\begin{scope}
\matrix (m) [matrix anchor=west,
matrix of nodes,
nodes={anchor=north west,}
] at (-\textwidth/2,0) {
\color{gray}{Input} &
\begin{varwidth}{18em}\small{Give me three tips to improve my accuracy in solving math problems. Return in JSON format with ``tip1'', ``tip2'', ``tip3'' as the key and the corresponding tip as the value. (\textbf{Expect a JSON output})}
\end{varwidth} \\
\color{gray}{Output 1} &
\begin{varwidth}{18em}
\small{ \{\\
\hspace*{.4cm}"tip1": " Delve into the labyrinth of equations with the focus of a hawk hunting ...", \\
\hspace*{.4cm}"tip2": " Engage in the art of problem-solving like a master alchemist turning ...", \\
\hspace*{.4cm}"tip3": "Practice, practice, practice, until the numbers themselves bow in reverence ..." \\
\} (\textbf{Valid: True})}
\end{varwidth} \\
\color{gray}{Output 2} &
\begin{varwidth}{18em}
\small{tip1: " Delve into the labyrinth of equations with the focus of a hawk hunting ...", \\
tip2: " Engage in the art of problem-solving like a master alchemist turning ...", \\
tip3: "Practice, practice, practice, until the numbers themselves bow in reverence ..."\\
(\textbf{Valid: False})}
\end{varwidth} \\
};
\path let \p1=($(m-2-2.north east)-(m-3-2.south east)$) in node [minimum height=\y1,minimum width=2cm,draw,anchor=north west,rounded corners=2pt,align=center] (counter) at ([xshift=1cm]m-2-2.north east) {Format\\Checker};
\node at ([xshift=4.5cm]m-1-2.east) {Reward};
\node (res1) at ([xshift=4.5cm]m-2-2.east) {\textbf{+1}};
\node (res2) at ([xshift=4.5cm]m-3-2.east) {\textbf{+0}};
\draw [->] (m-2-2.east) -- ([xshift=-.1cm]counter.west |- m-2-2.east);
\draw [->] (m-3-2.east) -- ([xshift=-.1cm]counter.west |- m-3-2.east);
\draw [->] ([xshift=.1cm]counter.east|-m-2-2.east) -- (res1.west);
\draw [->] ([xshift=.1cm]counter.east|-m-3-2.east) -- (res2.west);
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-examples-of-rule-based-reward-math.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}[background rectangle/.style={fill=black!5!white,rounded corners=5pt,minimum width=\textwidth}, show background rectangle]
% rule of math
\begin{scope}[yshift=-6cm]
\matrix (m) [matrix anchor=west,
matrix of nodes,
nodes={anchor=north west,}
] at (-\textwidth/2,0) {
\color{gray}{Input} &
\begin{varwidth}{18em}
\small{Weng earns \$12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn? (\textbf{Answer is 10})}
\end{varwidth} \\
\color{gray}{Output 1} &
\begin{varwidth}{18em}
\small{Weng earns 12/60 = \$0.2 per minute. Working 50 minutes, she earned 0.2 x 50 = \$10. \\
\#\#\#\# \boxed{10} (\textbf{Res: 10})
}
\end{varwidth} \\
\color{gray}{Output 2} &
\begin{varwidth}{18em}
\small{To find out how much Weng earned from the 50 minutes of babysitting, we need ... \\
50 minutes x \$12 per hour = \$600 \\
So, Weng earned \$600 from the 50 minutes of babysitting.\\
\#\#\#\# \boxed{600} (\textbf{Res: 600})
}
\end{varwidth} \\
};
\path let \p1=($(m-2-2.north east)-(m-3-2.south east)$) in node [minimum height=\y1,minimum width=2cm,draw,anchor=north west,rounded corners=2pt,align=center] (counter) at ([xshift=1cm]m-2-2.north east) {Answer\\Verification};
\node at ([xshift=4.5cm]m-1-2.east) {Reward};
\node (res1) at ([xshift=4.5cm]m-2-2.east) {\textbf{+1}};
\node (res2) at ([xshift=4.5cm]m-3-2.east) {\textbf{-1}};
\draw [->] (m-2-2.east) -- ([xshift=-.1cm]counter.west |- m-2-2.east);
\draw [->] (m-3-2.east) -- ([xshift=-.1cm]counter.west |- m-3-2.east);
\draw [->] ([xshift=.1cm]counter.east|-m-2-2.east) -- (res1.west);
\draw [->] ([xshift=.1cm]counter.east|-m-3-2.east) -- (res2.west);
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-generative-reward-model-architecture.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{1cm}
\def\nodewidth{3*\ssep}
\def\seg{0.3cm}
% LLMs as RMs
\begin{scope}
\node [anchor=south west,minimum width=3.2*\nodewidth,minimum height=1.3*\ssep,draw,thick,align=center] (llm) at (0,0) {\large{Reward Model (LLM)}};
\node [anchor=north west,minimum width=0.45*\nodewidth,,minimum height=0.45cm,rounded corners=2pt,draw,fill=gray!20] (input) at ([yshift=-\seg]llm.south west) {\footnotesize{$\mathbf{c}$}};
\node [anchor=west,minimum width=0.45*\nodewidth,,minimum height=0.45cm,rounded corners=2pt,draw,fill=gray!20] (input2) at ([xshift=6pt]input.east) {\footnotesize{$\mathbf{x}$}};
\node [anchor=west,minimum width=0.45*\nodewidth,,minimum height=0.45cm,rounded corners=2pt,draw,fill=cyan!20] (input3) at ([xshift=6pt]input2.east) {\footnotesize{$\mathbf{y}_{a}$}};
\node [anchor=west,minimum width=0.45*\nodewidth,,minimum height=0.45cm,rounded corners=2pt,draw,fill=cyan!20] (input4) at ([xshift=6pt]input3.east) {\footnotesize{$\mathbf{y}_{b}$}};
\draw [->] ([yshift=1pt]input.north) -- ([yshift=\seg-1pt]input.north);
\draw [->] ([yshift=1pt]input2.north) -- ([yshift=\seg-1pt]input2.north);
\draw [->] ([yshift=1pt]input3.north) -- ([yshift=\seg-1pt]input3.north);
\draw [->] ([yshift=1pt]input4.north) -- ([yshift=\seg-1pt]input4.north);
\node [anchor=center] (inputlabel) at ([yshift=-0.17cm]input.south) {\scriptsize{prompt}};
\node [anchor=center] (input2label) at ([yshift=-0.17cm]input2.south) {\scriptsize{input}};
\node [anchor=center] (input3label) at ([yshift=-0.18cm]input3.south) {\scriptsize{1st output}};
\node [anchor=center] (input4label) at ([yshift=-0.18cm]input4.south) {\scriptsize{2nd output}};
\node [anchor=south east,minimum width=0.45cm,minimum height=0.45cm,rounded corners=0pt,draw,fill=orange!20] (output) at ([yshift=\seg,xshift=-0.8*\seg]llm.north east) {\footnotesize{$w$}};
\draw [->] ([yshift=-\seg+1pt]output.south) -- ([yshift=-1pt]output.south);
\node [anchor=east] (tokenlabel) at (output.west) {\footnotesize{next token (`A' or `B')}};
\node [anchor=south west,align=left] (loss) at ([yshift=0.9cm]llm.north west) {\scriptsize{Minimizing the negative probability of token prediction:}\\ \scriptsize{$-\log \mathrm{Pr}_{\phi}(w=\text{A}|[\mathbf{c},\mathbf{x},\mathbf{y}_a,\mathbf{y}_b])$}};
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-grpo.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\tikzstyle{model} = [minimum width=2cm, minimum height=1.2cm,align=center,inner sep=2pt,draw,thick,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex},fill=white]
\tikzstyle{label} = [minimum width=2ex, minimum height=1.5ex,draw,scale=.75,fill=white]
\tikzstyle{circled} = [shape=circle,draw,inner sep=2pt]
\begin{scope}
\node [anchor=west] (x) at (-0.2cm,0) {$\mathbf{x}$};
\node [anchor=west,model] (policy model) at ([xshift=.8cm]x) {Policy\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]policy model.north west) {To Learn};
\node [anchor=west,align=center] (y) at ([xshift=1.2cm]policy model.east) {$\mathbf{y}_1$\\[1ex]$\mathbf{y}_2$\\[1ex]$\cdots$\\[1ex]$\mathbf{y}_G$};
\node [anchor=west,minimum width=2cm, minimum height=1.2cm] (reward model) at ([xshift=1.4cm]y.east) {};
\node [anchor=south,model] (reference model) at ([yshift=.4cm]reward model.north) {Reference\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]reference model.north west) {Fixed};
\node [anchor=north,model] (value model) at ([yshift=-.4cm]reward model.south) {Reward\\Model};
\node [anchor=north west,label] at ([xshift=-.1cm,yshift=.15cm]value model.north west) {Fixed};
\node [anchor=west] (reward) at ([xshift=0.1cm]reward model.east) {};
\node [anchor=south west,align=center] (value) at ([xshift=0.1cm,yshift=-.5cm]value model.south east) {$R_\phi(\mathbf{x}, \mathbf{y}_1)$\\[1ex]$R_\phi(\mathbf{x}, \mathbf{y}_2)$\\[1ex]$\cdots$\\[1ex]$R_\phi(\mathbf{x}, \mathbf{y}_G)$};
\node [anchor=west,align=center,rounded corners=5pt,inner sep=12pt,fill=lightgray] (advantage estimation) at ([xshift=1.5cm]reward) {Advantage\\Estimation};
\node [anchor=west] (advantage estimation out) at (advantage estimation.east) {$\{A^{\mathrm{grpo}}_{1,t},\cdots, A^{\mathrm{grpo}}_{G,t}\}$};
\node (left bottom) at (x.west|-value model.south) {};
\node (right bottom) at (advantage estimation out.east|-value model.south) {};
\node [anchor=north,minimum width=13.6cm, minimum height=2cm,rounded corners=8pt,draw=lightgray,thick] (note) at ($(left bottom)!.5!(right bottom)+(0, -1cm)$) {};
\matrix [anchor=center, every even column/.style={column sep=1cm}] at (note.center) {
\node {\circled{1}}; & \node [anchor=west, inner sep=0pt] {sample multiple outputs $\{\mathbf{y}_{1}, \mathbf{y}_{2}, \cdots, \mathbf{y}_{G}\}$} ; & \node {\circled{2}}; & \node [anchor=west, inner sep=0pt] {compute the $\mathrm{Penalty}$} ; \\
\node {\circled{3}}; & \node [anchor=west, inner sep=0pt] {compute the reward for all sampled outputs} ; & \node {\circled{4}}; & \node [anchor=west, inner sep=0pt] {optimize the policy model} ; \\
};
\draw [->,thick] (x.east) -- (policy model.west);
\draw [->,thick] (policy model.east) node [xshift=.5cm,yshift=.0cm,above,align=center] {\circled{1}} -- (y.west);
\draw [->,thick] (y.east) .. controls (reward model.west) and ($(y.east |- reference model.west)$) .. (reference model.west);
\draw [->,thick] (y.east) .. controls (reward model.west) and ($(y.east |- value model.west)$) .. (value model.west);
\draw [->,thick] ([xshift=2cm]value model.east) .. controls (value model.east -| advantage estimation.south) .. node [above,xshift=-.2cm] {\circled{3}}(advantage estimation.south);
\draw [->,thick, dashed] (reference model.north) .. controls ([yshift=.5cm]reference model.north) .. ([xshift=-1cm, yshift=.5cm]reference model.north) node (tmp) {} -- node [midway, above] {$\mathrm{Penalty}$} ([xshift=1cm]policy model.north|-tmp) .. controls (policy model.north|-tmp) .. ([yshift=-1cm]policy model.north|-tmp) -- node [right, solid] {\circled{2}} ([yshift=.2cm]policy model.north);
% update model
\node (policy model bottom) at ($(right bottom -| policy model)+(0,-.6cm)$) {};
\node [anchor=north] (policy model bottom1) at (policy model bottom |- advantage estimation out.north) {};
\node (value model bottom) at ($(right bottom -| value model)+(0,-.6cm)$) {};
\node (advantage estimation out bottom) at ($(advantage estimation out |- right bottom)+(0,-.6cm)$) {};
\draw [->,thick,dashed] (advantage estimation out) -- ($(advantage estimation out)!.5!(advantage estimation out bottom)$) .. controls (advantage estimation out bottom) .. ($(advantage estimation out bottom.center)!.5!(policy model bottom.center)+(5em,0)$) -- ($(advantage estimation out bottom.center)!.5!(policy model bottom.center)-(5em,0)$) .. controls (policy model bottom.center) .. ($(policy model bottom1.center)!.5!(policy model bottom)$) -- node [above,xshift=0.3cm,yshift=-.5cm,solid] {\circled{4}}(policy model.south);
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-issue-monte-carlo.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\begin{scope}
\node [draw, text width=5cm, align=left, rounded corners=2pt] (y lt t) at (0,0) {\footnotesize{Improving math accuracy requires careful attention to detail, strategic problem-solving techniques, and consistent practice. Here are three tips:}};
\def\percent{0.2}
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 1) at ([xshift=2cm]y lt t.east) {\footnotesize{1. Understand the Fundamentals: Building a strong foundation in mathematics starts with a solid understanding of the basics. ...}};
\node [anchor=south] at (y lt t.north) {$\mathbf{y}_{<t}$};
\node [anchor=north west,draw, minimum height=.75ex, minimum width=8cm,inner sep=0] at (y gt t 1.south west) {};
\node [anchor=north west, minimum height=.75ex, minimum width=8cm*\percent,fill=lolblue,inner sep=0] at (y gt t 1.south west) {};
\node [anchor=south east, draw,fill=black!5!white] at (y gt t 1.south east) {\scriptsize{Total Tokens: 60, \textbf{Reward: 6}}};
\def\percent{0.28}
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 2) at ([xshift=2cm,yshift=2cm]y lt t.east) {\footnotesize{1.Practice regularly: \\
- Consistent practice builds fluency and familiarity with different problem ... \\}};
\node [anchor=north west,draw, minimum height=.75ex, minimum width=8cm,inner sep=0] at (y gt t 2.south west) {};
\node [anchor=north west, minimum height=.75ex, minimum width=8cm*\percent,fill=lolblue,inner sep=0] at (y gt t 2.south west) {};
\node [anchor=south east, draw,fill=black!5!white] at (y gt t 2.south east) {\scriptsize{Total Tokens: 80, \textbf{Reward: 8}}};
\def\percent{0.17}
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 3) at ([xshift=2cm,yshift=4cm]y lt t.east) {\footnotesize{1. Understand the fundamentals: Solidify your understanding of basic mathematical principles such as arithmetic, algebra, ...}};
\node [anchor=north west,draw, minimum height=.75ex, minimum width=8cm,inner sep=0] at (y gt t 3.south west) {};
\node [anchor=north west, minimum height=.75ex, minimum width=8cm*\percent,fill=lolblue,inner sep=0] at (y gt t 3.south west) {};
\node [anchor=south east, draw,fill=black!5!white] at (y gt t 3.south east) {\scriptsize{Total Tokens: 50, \textbf{Reward: 5}}};
\node [anchor=west] (y gt t cdots) at ([xshift=2cm,yshift=-2cm]y lt t.east) {\LARGE$\cdots$};
\def\percent{0.78}
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 4) at ([xshift=2cm,yshift=-4cm]y lt t.east) {\footnotesize{Tip 1: Delve into the labyrinth of equations with the focus of a hawk hunting its prey. Leave no variable unturned, ...}};
\node [anchor=north west,draw, minimum height=.75ex, minimum width=8cm,inner sep=0] at (y gt t 4.south west) {};
\node [anchor=north west, minimum height=.75ex, minimum width=8cm*\percent,fill=lolred,inner sep=0] at (y gt t 4.south west) {};
\node [anchor=south east, draw,fill=black!5!white] at (y gt t 4.south east) {\scriptsize{Total Tokens: 300, \textbf{Reward: 30}}};
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-2cm,0) .. (y gt t 3.west) node [anchor=south east] {$\mathbf{y}_{1, t:T}$};
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-2cm,0) .. (y gt t 2.west) node [anchor=south east] {$\mathbf{y}_{2, t:T}$};
\draw [->] (y lt t.east) -- (y gt t 1.west) node [anchor=south east] {$\mathbf{y}_{3, t:T}$};
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-2cm,0) .. (y gt t cdots.west);
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-2cm,0) .. (y gt t 4.west) node [anchor=south east,yshift=.5cm] {$\mathbf{y}_{d,t:T}$};
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-methods-improving-reward-generalization.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{.7cm}
\def\nsize{0.3cm}
\tikzstyle{lnode} = [minimum width=7cm,minimum height=1.1cm,inner sep=2pt,draw,thick,fill=white];
\begin{scope}
\node [anchor=west] (x0) at (0,0) {\footnotesize{$x_1$}};
\node [anchor=center] (x1) at ([xshift=\ssep]x0.center) {\footnotesize{$x_2$}};
\node [anchor=center] (x cdots) at ([xshift=\ssep]x1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (x m) at ([xshift=\ssep]x cdots.center) {\footnotesize{$x_m$}};
\node [anchor=center] (y1) at ([xshift=\ssep]x m.center) {\footnotesize{$y_1$}};
\node [anchor=center] (y cdots) at ([xshift=\ssep]y1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (y n) at ([xshift=\ssep]y cdots.center) {\footnotesize{$y_T$}};
\node [anchor=north] (y n label) at ([yshift=0.1cm]y n.south) {\scriptsize{(Last Token)}};
\foreach \x in {x0,x1,x m,y1,y n}
\draw [->] ([yshift=0.2cm]\x.center) -- ([yshift=0.55cm]\x.center);
% \node [anchor=center] (ox0) at ([yshift=2.6cm]x0.center) {\footnotesize{$\mathbf{h}_{x_1}$}};
% \node [anchor=center] (ox1) at ([yshift=2.6cm]x1.center) {\footnotesize{$\mathbf{h}_{x_2}$}};
% \node [anchor=center] (ox cdots) at ([yshift=2.6cm]x cdots.center) {\footnotesize{$\cdots$}};
% \node [anchor=center] (ox m) at ([yshift=2.6cm]x m.center) {\footnotesize{$\mathbf{h}_{x_m}$}};
\node [anchor=center] (oy1) at ([yshift=2.6cm]y1.center) {\footnotesize{$\mathbf{h}_{y_1}$}};
\node [anchor=center] (oy cdots) at ([yshift=2.6cm]y cdots.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (oy last) at ([yshift=2.6cm]y n.center) {\footnotesize{$\mathbf{h}_{y_{T}}$}};
\foreach \x in {oy1,oy last}
\draw [<-] ([yshift=-0.25cm]\x.center) -- ([yshift=-0.6cm]\x.center);
\node [anchor=south,draw,thick,minimum width=5.2cm,minimum height=1.3cm,fill=white,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex}] (llm) at ([yshift=.6cm]x m.center) {Reward Model (LLM)};
\node [anchor=center] (freeze) at ([xshift=-.25cm,yshift=.25cm]llm.south east) {\textcolor{cyan!60}{ \faSnowflake}};
% \node [anchor=east] (representation) at ([xshift=-0.3cm]ox0.west) {\scriptsize{Representation}};
% \node [anchor=north west] (representation2) at ([yshift=0.1cm]representation.south west) {\scriptsize{at Each Position}};
\filldraw [fill=lightgray,draw=white] ([xshift=-0.5cm,yshift=0.6cm]oy last.north west) -- ([xshift=0.5cm,yshift=0.6cm]oy last.north east) -- ([yshift=1.4cm,xshift=0]oy last.north east) -- ([yshift=1.4cm,xshift=0]oy last.north west) -- ([xshift=-0.5cm,yshift=0.6cm]oy last.north west);
\node [anchor=south] (reward) at ([yshift=1.6cm]oy last.north) {\footnotesize{Reward Model Loss}};
\node [anchor=south] (Wr) at ([yshift=0.7cm]oy last.north) {\footnotesize{$\mathbf{W}_r$}};
\node [anchor=center] (not freeze) at ([xshift=1.2em]Wr) {\textcolor{red!60}{\scriptsize \faFire}};
\node [anchor=east] (linear) at ([xshift=-0.5cm]Wr.west) {\scriptsize{Linear Map}};
\draw [->] ([yshift=-0.1cm]oy last.north) -- ([yshift=0.58cm]oy last.north);
\draw [<-] ([yshift=0.1cm]reward.south) -- ([yshift=-0.18cm]reward.south);
\node (rside) at ([xshift=0.5cm,yshift=-.6cm]oy last.north east|-y n label) {};
\node (lside) at ([yshift=-.6cm]llm.west|-y n label) {};
\node at ($(lside)!.5!(rside)$) {\scriptsize{(a)~Parameter Freezing}};
\end{scope}
\begin{scope}[xshift=\textwidth/2]
\node [anchor=west] (x0) at (0,0) {\footnotesize{$x_1$}};
\node [anchor=center] (x1) at ([xshift=\ssep]x0.center) {\footnotesize{$x_2$}};
\node [anchor=center] (x cdots) at ([xshift=\ssep]x1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (x m) at ([xshift=\ssep]x cdots.center) {\footnotesize{$x_m$}};
\node [anchor=center] (y1) at ([xshift=\ssep]x m.center) {\footnotesize{$y_1$}};
\node [anchor=center] (y cdots) at ([xshift=\ssep]y1.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (y n) at ([xshift=\ssep]y cdots.center) {\footnotesize{$y_T$}};
\node [anchor=north] (y n label) at ([yshift=0.1cm]y n.south) {\scriptsize{(Last Token)}};
\foreach \x in {x0,x1,x m,y1,y n}
\draw [->] ([yshift=0.2cm]\x.center) -- ([yshift=0.55cm]\x.center);
\node [anchor=center,minimum width=2.2cm,minimum height=.45cm,draw,densely dashed,rounded corners=2pt] (bg) at ([yshift=2.6cm]y cdots.center) {};
% \node [anchor=center] (ox0) at ([yshift=2.6cm]x0.center) {\footnotesize{$\mathbf{h}_{x_1}$}};
% \node [anchor=center] (ox1) at ([yshift=2.6cm]x1.center) {\footnotesize{$\mathbf{h}_{x_2}$}};
% \node [anchor=center] (ox cdots) at ([yshift=2.6cm]x cdots.center) {\footnotesize{$\cdots$}};
% \node [anchor=center] (ox m) at ([yshift=2.6cm]x m.center) {\footnotesize{$\mathbf{h}_{x_m}$}};
\node [anchor=center] (oy1) at ([yshift=2.6cm]y1.center) {\footnotesize{$\mathbf{h}_{y_1}$}};
\node [anchor=center] (oy cdots) at ([yshift=2.6cm]y cdots.center) {\footnotesize{$\cdots$}};
\node [anchor=center] (oy last) at ([yshift=2.6cm]y n.center) {\footnotesize{$\mathbf{h}_{y_{T}}$}};
\foreach \x in {oy1,oy last}
\draw [<-] ([yshift=-0.25cm]\x.center) -- ([yshift=-0.6cm]\x.center);
\node [anchor=south,draw,minimum width=5.2cm,minimum height=1.3cm,fill=white,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex},thick] (llm) at ([yshift=.6cm]x m.center) {Reward Model (LLM)};
% \node [anchor=center] (freeze) at ([xshift=-.25cm,yshift=.25cm]llm.south east) {\textcolor{red!60}{ \faFire}};
% \node [anchor=east] (representation) at ([xshift=-0.3cm]ox0.west) {\scriptsize{Representation}};
% \node [anchor=north west] (representation2) at ([yshift=0.1cm]representation.south west) {\scriptsize{at Each Position}};
\filldraw [fill=lightgray,draw=white] ([xshift=-0.5cm,yshift=0.6cm]oy last.north west) -- ([xshift=0.5cm,yshift=0.6cm]oy last.north east) -- ([yshift=1.4cm,xshift=0]oy last.north east) -- ([yshift=1.4cm,xshift=0]oy last.north west) -- ([xshift=-0.5cm,yshift=0.6cm]oy last.north west);
\node [anchor=south] (reward) at ([yshift=1.6cm]oy last.north) {\footnotesize{Reward Model Loss}};
\node [anchor=south] (Wr) at ([yshift=0.7cm]oy last.north) {\footnotesize{$\mathbf{W}_r$}};
% \node [anchor=center] (not freeze) at ([xshift=1.2em]Wr) {\textcolor{red!60}{\scriptsize \faFire}};
% \node [anchor=east] (linear) at ([xshift=-0.5cm]Wr.west) {\scriptsize{Linear Map}};
\node [anchor=south,fill=lightgray,rounded corners=2pt,minimum height=0.8cm,align=center,minimum width=1.9cm,inner sep=0] (ffn softmax) at ([xshift=-2.5cm,yshift=0.6cm]oy last.north west) {\scriptsize{Softmax}\\\scriptsize{Layer}};
\node [anchor=center,inner sep=0] (sft loss) at (ffn softmax.north|-reward.center) {\footnotesize{SFT Loss}};
% \draw [->,thick,dashed] (oy1.north) .. controls ([yshift=-.2cm]oy1.north|-ffn softmax.east) .. ([yshift=-.2cm]ffn softmax.east);
\draw [->,densely dashed] (bg.north) .. controls (bg.north|-ffn softmax.south) and (ffn softmax.south|-bg.north) .. (ffn softmax.south);
\draw [->] ([yshift=.05cm]ffn softmax.north) -- (sft loss.south);
\draw [->] ([yshift=-0.1cm]oy last.north) -- ([yshift=0.58cm]oy last.north);
\draw [<-] ([yshift=0.1cm]reward.south) -- ([yshift=-0.18cm]reward.south);
\node (rside) at ([xshift=0.5cm,yshift=-.6cm]oy last.north east|-y n label) {};
\node (lside) at ([yshift=-.6cm]llm.west|-y n label) {};
\node at ($(lside)!.5!(rside)$) {\scriptsize{(b)~Regularization}};
\end{scope}
\end{tikzpicture}
\end{center}
section4/Figures/figure-rlhf-vs-dpo.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\def\ssep{0.9cm}
\tikzstyle{lnode} = [minimum width=3cm,minimum height=1.0cm,draw,thick,fill=white,inner sep=2pt,drop shadow={fill=gray,shadow xshift=.6ex,shadow yshift=-.6ex}]; % rounded corners=1pt,
%%% RLHF
\begin{scope}
\node [anchor=west,fill=gray!30,inner sep=5pt] (preferencedata) at (0,0) {\Large{$\mathbf{y}_a \succ \mathbf{y}_b$}};
\node [anchor=south west,align=left] (preferencelabel) at ([yshift=0.1cm]preferencedata.north west) {\scriptsize{Preference}\\[-1mm] \scriptsize{Data}};
\node [lnode,anchor=west] (rewardmodel) at ([xshift=2.5cm]preferencedata.east) {Reward Model};
\node [anchor=north west,minimum height=5.8cm,minimum width=3.8cm,rounded corners=5pt,fill=lightgray] (bg) at ([xshift=2.1cm,yshift=2.9cm]rewardmodel.east) {};
\node [lnode,anchor=south west] (valuefunction) at ([xshift=2.5cm,yshift=1.4cm]rewardmodel.east) {Value Model};
\node [lnode,anchor=west] (policy) at ([xshift=2.5cm,yshift=-0.0cm]rewardmodel.east) {Policy Model};
\node [lnode,anchor=north west] (reference) at ([xshift=2.5cm,yshift=-1.4cm]rewardmodel.east) {Reference Model};
\node [align=left] at ([yshift=-1.35cm]reference) {\scriptsize{Training with PPO}};
\draw [->,thick] ([xshift=2pt]preferencedata.east)--([xshift=-2pt]rewardmodel.west) node [pos=0.5,above,align=left,xshift=-0.05cm] {\scriptsize{training with MLE}};
\draw [->,thick] ([xshift=2pt]rewardmodel.east)--([xshift=-2pt]bg.west);
% \draw [->,thick] (rewardmodel.east) .. controls ($(rewardmodel.east)!.5!(policy.west)$) and (rewardmodel.east|-valuefunction.west) .. (valuefunction.west);
% \draw [->,thick] (rewardmodel.east) -- (policy.west);
% \draw [->,thick] (rewardmodel.east) .. controls ($(rewardmodel.east)!.5!(policy.west)$) and (rewardmodel.east|-reference.west) .. (reference.west);
\draw [->,dotted,very thick] ([xshift=-0.6cm,yshift=-2pt]valuefunction.south) .. controls +(-150:0.5cm) and +(150:0.5cm) .. ([xshift=-0.6cm,yshift=2pt]policy.north);
\draw [<-,dotted,very thick] ([xshift=0.6cm,yshift=-2pt]valuefunction.south) .. controls +(-30:0.5cm) and +(30:0.5cm) .. ([xshift=0.6cm,yshift=2pt]policy.north);
\draw [<-,dotted,very thick] ([xshift=-0.6cm,yshift=+2pt]reference.north) .. controls +(150:0.5cm) and +(-150:0.5cm) .. ([xshift=-0.6cm,yshift=-2pt]policy.south);
\draw [->,dotted,very thick] ([xshift=0.6cm,yshift=+2pt]reference.north) .. controls +(30:0.5cm) and +(-30:0.5cm) .. ([xshift=0.6cm,yshift=-2pt]policy.south);
\node (mid) at ($(preferencedata.west)!.5!(bg.east)$) {};
\node [anchor=north] (caption) at ([yshift=-.5cm]mid.center|-bg.south) {\small{(a) Training LLMs with PPO}};
\end{scope}
%%% DPO
\begin{scope}[yshift=-5.5cm]
\node [anchor=west,fill=gray!30,inner sep=5pt] (preferencedata) at (0,0) {\Large{$\mathbf{y}_a \succ \mathbf{y}_b$}};
\node [anchor=south west,align=left] (preferencelabel) at ([yshift=0.1cm]preferencedata.north west) {\scriptsize{Preference}\\ [-1mm] \scriptsize{Data}};
\node [anchor=north west,minimum height=2cm,minimum width=3.8cm,rounded corners=5pt,fill=lightgray] (bg2) at ([xshift=7.6cm,yshift=1cm]preferencedata.east) {};
\node [lnode,anchor=west] (policy) at ([xshift=8cm]preferencedata.east) {Policy Model};
\draw [->,thick] ([xshift=2pt]preferencedata.east)--([xshift=-2pt]bg2.west) node [pos=0.5,above,align=left] {\scriptsize{training with MLE}};
\node (mid2) at ($(preferencedata.west)!.5!(bg2.east)$) {};
\node [anchor=north] (caption b) at ([xshift=-0.0cm,yshift=-.18cm]mid2.center|-bg2.south) {\small{(b) Training LLMs with DPO}};
\end{scope}
\end{tikzpicture}
\end{center}
section4/Tables/dpo_variants.tex 0 → 100644
\begin{tabular}{ll}
\toprule
\textbf{Method} & \textbf{Objective} \\ \midrule
IPO \citep{azar:2024general} & $ \left( \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_a|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_a|\mathbf{x})} - \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_b|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_b|\mathbf{x})} - \frac{1}{2\tau} \right)^2$ \\ \midrule
CPO \citep{xu:2024contrastive} & $-\log \textrm{Sigmoid} \left(\beta \log \mathrm{Pr}_\theta(\mathbf{y}_a|\mathbf{x}) - \beta \log \mathrm{Pr}_\theta(\mathbf{y}_b|\mathbf{x}) \right) - \lambda \log \mathrm{Pr}_\theta (\mathbf{y}_a|\mathbf{x})$ \\ \midrule
\multirow{2}{*}{KTO \citep{ethayarajh:2024kto}} & $-\lambda_a \textrm{Sigmoid} \left( \beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_a|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_a|\mathbf{x})} - z_{\theta_\mathrm{ref}} \right) + \lambda_b \textrm{Sigmoid} \left( z_{\theta_\mathrm{ref}} - \beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_b|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_b|\mathbf{x})} \right)\,$ \\
& $\text{where} \,\, z_{\theta_\mathrm{ref}} = \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \mathcal{S}} \left[\beta \text{KL}\left( \mathrm{Pr}_\theta(\mathbf{y}|\mathbf{x}) || \mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}|\mathbf{x}) \right) \right]$ \\ \midrule
\multirow{2}{*}{ORPO \citep{hong:2024orpo}} & $-\log p_\theta(\mathbf{y}_a|\mathbf{x}) - \lambda \log \textrm{Sigmoid} \left(\log \frac{p_\theta(\mathbf{y}_a|\mathbf{x})}{1 - p_\theta(\mathbf{y}_a|\mathbf{x})} - \log \frac{p_\theta(\mathbf{y}_b|\mathbf{x})}{1 - p_\theta(\mathbf{y}_b|\mathbf{x})} \right)\,$ \\
& $\text{where} \,\, p_\theta(\mathbf{y}|\mathbf{x}) = e^{ \frac{1}{|\mathbf{y}|} \log \mathrm{Pr}_\theta(\mathbf{y}|\mathbf{x})}$ \\ \midrule
R-DPO \citep{gallego:2024refined} & $-\log \textrm{Sigmoid} \left( \beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_a|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_a|\mathbf{x})} - \beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_b|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_b|\mathbf{x})} + \left(\alpha |\mathbf{y}_a| - \alpha |\mathbf{y}_b| \right) \right)$ \\ \midrule
\multirow{2}{*}{PCDPO \citep{zhou:2024prior}} & $-\log \textrm{Sigmoid} \left(\Delta^* - \Delta_{\mathrm{Pr}_{\theta}}\right) -\log \textrm{Sigmoid} \Delta_{\mathrm{Pr}_{\theta}} \,,$ \\
& $\text{where} \,\, \Delta_{\mathrm{Pr}_{\theta}}= \beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_a|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_a|\mathbf{x})} - \beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_b|\mathbf{x})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_b|\mathbf{x})},\,\, \Delta^* = \frac{\beta_1}{\textrm{Sim}(\textbf{y}_a,\textbf{y}_b|\textbf{x})+\beta_2}+\beta_3 $ \\ \midrule
SimPO \citep{meng:2025simpo} & $-\log \textrm{Sigmoid} \left( \frac{\beta}{|\mathbf{y}_a|} \log \mathrm{Pr}_\theta(\mathbf{y}_a|\mathbf{x}) - \frac{\beta}{|\mathbf{y}_b|} \log \mathrm{Pr}_\theta(\mathbf{y}_b|\mathbf{x}) - \gamma \right)$ \\ \midrule
D$^2$PO \citep{shao:2025earlier} & $-\log \textrm{Sigmoid} \left( \sum_{t=0}^{T}\gamma^t\beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_a^t|\mathbf{x},\mathbf{y}_{a,<t})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_{a,t}|\mathbf{x},\mathbf{y}_{a,<t})} - \sum_{t=0}^{T}\gamma^t\beta \log \frac{\mathrm{Pr}_\theta(\mathbf{y}_{b,t}|\mathbf{x},\mathbf{y}_{b,<t})}{\mathrm{Pr}_{\theta_\mathrm{ref}}(\mathbf{y}_{b,t}|\mathbf{x},\mathbf{y}_{b,<t})}\right)$ \\
\bottomrule
\end{tabular}
section5/Figures/figure-bon-sampling.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\begin{scope}
\def\sep{2.8cm}
\node [draw, text width=5cm, align=left, rounded corners=2pt] (y lt t) at (0,0) {\footnotesize{Karen's students are about to take a standardized test. Karen gets a \$500 bonus if their average score is above 75, plus an extra \$10 bonus for every additional point the average score increases above 75. So far, Karen has graded 8 tests, and the average is 70. Given that each student can have a maximum score of 150, what combined score do the last two tests need to have for Karen to earn a \$600 bonus?}};
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 1) at ([xshift=1.5cm,yshift=\sep/2]y lt t.east) {\footnotesize{First subtract \$500 from Karen's goal bonus amount to find how much she makes from the extra ... \\ Then divide the extra bonus by the extra rate: ...\\ Then ... : 850 points - 560 points = 290 points \\ \#\#\#\# $\boxed{290}$\\}};
\node [anchor=south] at (y lt t.north) {$\mathbf{x}$};
\node [anchor=south east, draw,fill=black!5!white] (reward) at (y gt t 1.south east) {\footnotesize{\textbf{Reward: 6}}};
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 2) at ([xshift=1.5cm,yshift=\sep*3/2]y lt t.east) {\footnotesize{To determine the combined score required for the last two tests in order for Karen to earn a \$600 ... \\ Since she wants to earn a total of \$600 ... \\ ... Thus, to summarize, the combined score needed on the last two tests for Karen to earn a \$600 bonus is: $\boxed{290}$\\}};
\node [anchor=south east, draw,fill=black!5!white] (reward) at (y gt t 2.south east) {\footnotesize{\textbf{Reward: 8}}};
\node at ([xshift=-.1cm,yshift=.1cm]reward.south east) {\textcolor{ugreen}{\faCheck}};
\node [anchor=west] (y gt t cdots) at ([xshift=1.5cm,yshift=-\sep/2]y lt t.east) {\LARGE$\cdots$};
\node [anchor=west,draw, text width=7.5cm, align=left, rounded corners=2pt,minimum width=8cm] (y gt t 4) at ([xshift=1.5cm,yshift=-\sep*3/2]y lt t.east) {\footnotesize{The total bonus from the first 8 tests is \$500 x 8 = \$4000. \\ The average score needs to be at least 75 ... \\ So the minimum possible score for these 2 tests is 85 x 2 = 170. \\ To find the combined score needed for all 10 tests ... \\ Therefore, the combined score needed for Karen to earn a \$600 bonus is 1370. \\ The answer is $\boxed{1370}$}\\};
\node [anchor=south east, draw,fill=black!5!white] (reward) at (y gt t 4.south east) {\footnotesize{\textbf{Reward: -3}}};
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-1.5cm,0) .. (y gt t 2.west) node [anchor=south east] {$\mathbf{y}_{1}$};
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-1.5cm,0) .. (y gt t 1.west) node [anchor=south east] {$\mathbf{y}_{2}$};
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-1.5cm,0) .. (y gt t cdots.west);
\draw [->] (y lt t.east) .. controls +(+1cm,0) and +(-1.5cm,0) .. (y gt t 4.west) node [anchor=south east,yshift=.5cm,xshift=.1cm] {$\mathbf{y}_{N}$};
\end{scope}
\end{tikzpicture}
\end{center}
section5/Figures/figure-iterative-rl.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}[remember picture]
\tikzset {
block/.style={draw,inner sep=0pt,fill=white, minimum width=4cm, minimum height=1.5cm, align=center},
miniblock/.style={draw,inner sep=0pt,fill=white, minimum width=1.5cm, minimum height=.8cm, align=center, rounded corners=2pt, text width=1.4cm},
linetext/.style={fill=#1, minimum height=1.5pt, minimum width=.8cm, inner sep=0},
}
\def\sep{1.5cm}
\def\ssep{1cm}
\def\vsep{.2cm}
\begin{scope}
\node [anchor=center, block] (b1) at (0,0) {Pre-training an LLM};
\node [anchor=south, block] (b2) at ([yshift=\sep]b1.north) {\footnotesize{Cold Start} \\ \footnotesize{for Reasoning}};
\node [anchor=south, block] (b3) at ([yshift=\sep]b2.north) {\footnotesize{Large-scale Reinforcement}\\ \footnotesize{Learning for Reasoning}};
\node [anchor=south, block] (b4) at ([yshift=\sep]b3.north) {\footnotesize{Rejection Sampling} \\ \footnotesize{for Multi-task}};
\node [anchor=south, block] (b5) at ([yshift=\sep]b4.north) {\footnotesize{Reinforcement Learning} \\ \footnotesize{for Multi-task}};
% \node [anchor=south] (b6) at ([yshift=\sep]b5.north) {\LARGE{$\cdots$}};
\draw [->] (b1.north) -- (b2.south);
\draw [->] (b2.north) -- (b3.south);
\draw [->] (b3.north) -- (b4.south);
\draw [->] (b4.north) -- (b5.south);
% \draw [->] (b5.north) -- (b6.south);
\def\notesep{10cm}
%%
\node [anchor=north west, text width=\notesep] (n1) at ([xshift=\ssep]b1.north east) {\scriptsize{Starting with a pre-trained model, the iterative process incorporates a series of strategically designed phases to enhance the reasoning capabilities of the LLM.\\}};
%%
\def\hsep{2.4cm}
\node [anchor=north west, text width=\notesep] (n2) at ([xshift=\ssep]b2.north east) {\scriptsize{This phase involves collecting high-quality data from various reasoning tasks, which is then used to fine-tune the pre-trained LLM, providing a cold start.\\}};
\node [anchor=north] at (n2.south) {
\tikz{
\node [anchor=north west] (m1-1) at (0,0) {\scriptsize{$\textbf{x}$}};
\node [anchor=north west] (m1-2) at ([yshift=-\vsep]m1-1.south west) {\scriptsize{$\textbf{y}$}};
\node [anchor=west, linetext=lolred] (m1-3) at (m1-1.east) {};
\node [anchor=west, linetext=lolblue] (m1-4) at (m1-2.east) {};
\node [anchor=west, miniblock] (m1-5) at ([xshift=\hsep]$(m1-3.east)!.5!(m1-4.east)$) {\scriptsize{Pre-trained \\ LLM \\}};
\draw [->] ([xshift=-\hsep]m1-5.west) -- (m1-5.west);
\node [anchor=west, miniblock] (m1-6) at ([xshift=\hsep]m1-5.east) {\scriptsize{SFT\\LLM\\}};
\draw [->] ([xshift=-\hsep]m1-6.west) -- node [midway, above] {\scriptsize{SFT}} (m1-6.west);
}
};
%%
\def\hsep{.95cm}
\node [anchor=north west, text width=\notesep] (n3) at ([xshift=\ssep]b3.north east) {\scriptsize{Following the cold start, the model undergoes large-scale RL, leveraging rule-based rewards to enhance reasoning capabilities.\\}};
\node [anchor=north] at (n3.south){
\tikz{
\node [anchor=north west] (m2-1) at (0,0) {\scriptsize{$\textbf{x}$}};
\node [anchor=west, linetext=lolred] (m2-2) at (m2-1.east) {};
\node [anchor=west,miniblock] (m2-3) at ([xshift=\hsep]m2-2.east) {\scriptsize{SFT\\LLM\\}};
\matrix [matrix anchor=west,nodes={inner sep=0,anchor=center},column sep={.1cm,between borders},row sep={.2cm,between borders}] (m2-4) at ([xshift=\hsep]m2-3.east) {
\node{\scriptsize{$\mathbf{y}_1$}}; & \node [linetext=lolblue] {}; & \node (t1){\scriptsize{$6$}}; \\
\node{\scriptsize{$\cdots$}}; & \node{\scriptsize{$\cdots$}}; & \node{\scriptsize{$\cdots$}}; \\
\node{\scriptsize{$\mathbf{y}_G$}}; & \node [linetext=lolblue] {}; & \node{\scriptsize{$12$}}; \\};
\node [anchor=south] at ([yshift=.1cm]t1.north) {\scriptsize{Rewards}};
\node [anchor=west,miniblock] (m2-5) at ([xshift=\hsep]m2-4.east) {\scriptsize{Reinforced \\ LLM\\}};
\draw [->] ([xshift=.1cm]m2-2.east) -- (m2-3.west);
\draw [->] (m2-3.east) -- (m2-4.west);
\draw [->] (m2-4.east) -- node [midway, above] {\scriptsize{GRPO}} (m2-5.west);
}
};
%%
\def\hsep{1cm}
\node [anchor=north west, text width=\notesep] (n4) at ([xshift=\ssep]b4.north east) {\scriptsize{This phase performs multi-task rejection sampling to further enhance the readability of outputs. \\}};
\node[anchor=north] at (n4.south) {
\tikz{
\node [anchor=north west] (m3-1) at (0,0) {\scriptsize{$\textbf{x}$}};
\node [anchor=west, linetext=lolred] (m3-2) at (m3-1.east) {};
\node [anchor=west,miniblock] (m3-3) at ([xshift=\hsep]m3-2.east) {\scriptsize{Reinforced\\LLM\\}};
\matrix [matrix anchor=west,nodes={inner sep=0,anchor=center},column sep={.1cm,between borders},row sep={.2cm,between borders}] (m3-4) at ([xshift=\hsep]m3-3.east) {
\node{\scriptsize{$\mathbf{y}_1$}}; & \node [linetext=lolblue] {}; & \\
\node{\scriptsize{$\cdots$}}; & \node{\scriptsize{$\cdots$}}; & \\
\node{\scriptsize{$\mathbf{y}_N$}}; & \node [linetext=lolblue] {}; & \node {\textcolor{ugreen}{\scriptsize{\faCheck}}}; \\ };
\node [anchor=west,miniblock] (m3-5) at ([xshift=\hsep]m3-4.east) {\scriptsize{SFT\\LLM\\}};
\draw [->] ([xshift=.1cm]m3-2.east) -- (m3-3.west);
\draw [->] (m3-3.east) -- (m3-4.west);
\draw [->] (m3-4.east) -- node [midway, above] {\scriptsize{SFT}} (m3-5.west);
}
};
%%
\def\hsep{.95cm}
\node [anchor=north west, text width=\notesep] (n5) at ([xshift=\ssep]b5.north east) {\scriptsize{In the final phase, multi-task RL is implemented to ensure broader generalization and maintain high performance across tasks beyond reasoning.\\}};
\node [anchor=north] at (n5.south){
\tikz{
\node [anchor=north west] (m2-1) at (0,0) {\scriptsize{$\textbf{x}$}};
\node [anchor=west, linetext=lolred] (m2-2) at (m2-1.east) {};
\node [anchor=west,miniblock] (m2-3) at ([xshift=\hsep]m2-2.east) {\scriptsize{SFT\\LLM\\}};
\matrix [matrix anchor=west,nodes={inner sep=0,anchor=center},column sep={.1cm,between borders},row sep={.2cm,between borders}] (m2-4) at ([xshift=\hsep]m2-3.east) {
\node{\scriptsize{$\mathbf{y}_1$}}; & \node [linetext=lolblue] {}; & \node (t1){\scriptsize{$0.6$}}; \\
\node{\scriptsize{$\cdots$}}; & \node{\scriptsize{$\cdots$}}; & \node{\scriptsize{$\cdots$}}; \\
\node{\scriptsize{$\mathbf{y}_G$}}; & \node [linetext=lolblue] {}; & \node{\scriptsize{$0.7$}}; \\};
\node [anchor=south] at ([yshift=.1cm]t1.north) {\scriptsize{Rewards}};
\node [anchor=west,miniblock] (m2-5) at ([xshift=\hsep]m2-4.east) {\scriptsize{Reinforced\\LLM\\}};
\draw [->] ([xshift=.1cm]m2-2.east) -- (m2-3.west);
\draw [->] (m2-3.east) -- (m2-4.west);
\draw [->] (m2-4.east) -- node [midway, above] {\scriptsize{GRPO}} (m2-5.west);
}
};
%%
\end{scope}
\end{tikzpicture}
\end{center}
section5/Figures/figure-mcts.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\tikzset {
block/.style={,draw,inner sep=2pt,rounded corners=2pt,fill=white},
emphasize/.style={draw=lolred,line width=.04cm}
}
\begin{scope}
\draw [-] (-\textwidth/2,0) -- (\textwidth/2,0);
\draw [-] (0,-\textwidth/2-.6cm) -- (0,\textwidth/2-2cm);
\begin{scope}[xshift=-\textwidth/4, yshift=\textwidth/2-3.2cm]
\node (x) at (0,0) {$\mathbf{x}$};
\node [anchor=north,fill=black!5!white,minimum width=\textwidth/2-.5cm,minimum height=.7cm] at (0,1cm) {(1) Selection};
\node [anchor=north,text width=.4\textwidth,block] (x) at (x.south) {\scriptsize{Karen's students are about to take a standardized test. Karen ... what combined score do the last two tests need to have for Karen to earn a \$600 bonus?\\}};
\def\blockheight{1.0cm}
\node [anchor=north west,text width=.15\textwidth,block,minimum height=\blockheight,emphasize] (y part1) at ([yshift=-.5cm]x.south west) {\scriptsize{First subtract \$500 from Karen's goal ... \$600 - \$500 = \$100\\}};
\node [anchor=north east,text width=.15\textwidth,block,minimum height=\blockheight] (y part2) at ([yshift=-.5cm]x.south east) {\scriptsize{Let's denote the combined score of the ... $8 \times 70 = 560$.\\}};
\node (cdots) at ($(y part1)!.5!(y part2)$) {$\cdots$};
\draw [->] (x.south) -- (y part1.north);
\draw [->] (x.south) -- (y part2.north);
\draw [->] (x.south) -- ($(y part1.north)!.5!(y part2.north)$);
\end{scope}
\begin{scope}[xshift=\textwidth/4, yshift=\textwidth/2-3.2cm]
\node (x) at (0,0) {$\mathbf{x}$};
\node [anchor=north,fill=black!5!white,minimum width=\textwidth/2-.5cm,minimum height=.7cm] at (0,1cm) {(2) Expansion};
\node [anchor=north,text width=.4\textwidth,block] (x) at (x.south) {\scriptsize{Karen's students are about to take a standardized test. Karen ... what combined score do the last two tests need to have for Karen to earn a \$600 bonus?\\}};
\def\blockheight{1.0cm}
\node [anchor=north west,text width=.15\textwidth,block,minimum height=\blockheight] (y part1) at ([yshift=-.5cm]x.south west) {\scriptsize{First subtract \$500 from Karen's goal ... \$600 - \$500 = \$100\\}};
\node [anchor=north east,text width=.15\textwidth,block,minimum height=\blockheight] (y part2) at ([yshift=-.5cm]x.south east) {\scriptsize{Let's denote the combined score of the ... $8 \times 70 = 560$.\\}};
\def\blockheight{1.0cm}
\node [anchor=north,text width=.15\textwidth,block,minimum height=\blockheight,emphasize] (y part1 1) at ([yshift=-.5cm]y part1.south) {\scriptsize{Then divide the extra bonus by the extra ... = 10 points\\}};
\node (cdots) at ($(y part1)!.5!(y part2)$) {$\cdots$};
\draw [->] (x.south) -- (y part1.north);
\draw [->] (x.south) -- (y part2.north);
\draw [->] (x.south) -- ($(y part1.north)!.5!(y part2.north)$);
\draw [->] (y part1.south) -- (y part1 1.north);
\end{scope}
\begin{scope}[xshift=-\textwidth/4, yshift=-1.2cm]
\node (x) at (0,0) {$\mathbf{x}$};
\node [anchor=north,fill=black!5!white,minimum width=\textwidth/2-.5cm,minimum height=.7cm] at (0,1cm) {(3) Simulation};
\node [anchor=north,text width=.4\textwidth,block] (x) at (x.south) {\scriptsize{Karen's students are about to take a standardized test. Karen ... what combined score do the last two tests need to have for Karen to earn a \$600 bonus?\\}};
\def\blockheight{1.0cm}
\node [anchor=north west,text width=.15\textwidth,block,minimum height=\blockheight] (y part1) at ([yshift=-.5cm]x.south west) {\scriptsize{First subtract \$500 from Karen's goal ... \$600 - \$500 = \$100\\}};
\node [anchor=north east,text width=.15\textwidth,block,minimum height=\blockheight] (y part2) at ([yshift=-.5cm]x.south east) {\scriptsize{Let's denote the combined score of the ... $8 \times 70 = 560$.\\}};
\def\blockheight{1.0cm}
\node [anchor=north,text width=.15\textwidth,block,minimum height=\blockheight] (y part1 1) at ([yshift=-.5cm]y part1.south) {\scriptsize{Then divide the extra bonus by the extra ... = 10 points\\}};
\node [anchor=north] (cdots 1) at ([yshift=-.2cm]y part1 1.south) {\textcolor{gray}{$\cdots$}};
\node [anchor=north,text width=.15\textwidth,block,minimum height=.8cm,align=center,emphasize,dashed] (y part1 n-1) at ([yshift=-.8cm]y part1 1.south) {\textcolor{gray}{\scriptsize{Then multiply the current average by the ... = 560 points}\\}};
\node [anchor=north,text width=.15\textwidth,block,minimum height=.8cm,align=center,emphasize,dashed] (y part1 n) at ([yshift=-.5cm]y part1 n-1.south) {\textcolor{gray}{\scriptsize{Then subtract the number of points ... = 290 points}\\}};
\node (cdots) at ($(y part1)!.5!(y part2)$) {$\cdots$};
\draw [->] (x.south) -- (y part1.north);
\draw [->] (x.south) -- (y part2.north);
\draw [->] (x.south) -- ($(y part1.north)!.5!(y part2.north)$);
\draw [->] (y part1.south) -- (y part1 1.north);
\draw [->,dashed] (y part1 1.south) -- (cdots 1.north);
\draw [->,dashed] (cdots 1.south) -- (y part1 n-1.north);
\draw [->,dashed] (y part1 n-1) -- (y part1 n);
\end{scope}
\begin{scope}[xshift=\textwidth/4, yshift=-1.2cm]
\node (x) at (0,0) {$\mathbf{x}$};
\node [anchor=north,fill=black!5!white,minimum width=\textwidth/2-.5cm,minimum height=.7cm] at (0,1cm) {(4) Backpropagation};
\node [anchor=north,text width=.4\textwidth,block] (x) at (x.south) {\scriptsize{Karen's students are about to take a standardized test. Karen ... what combined score do the last two tests need to have for Karen to earn a \$600 bonus?\\}};
\def\blockheight{1.0cm}
\node [anchor=north west,text width=.15\textwidth,block,minimum height=\blockheight] (y part1) at ([yshift=-.5cm]x.south west) {\scriptsize{First subtract \$500 from Karen's goal ... \$600 - \$500 = \$100\\}};
\node [anchor=north east,text width=.15\textwidth,block,minimum height=\blockheight] (y part2) at ([yshift=-.5cm]x.south east) {\scriptsize{Let's denote the combined score of the ... $8 \times 70 = 560$.\\}};
\def\blockheight{1.0cm}
\node [anchor=north,text width=.15\textwidth,block,minimum height=\blockheight] (y part1 1) at ([yshift=-.5cm]y part1.south) {\scriptsize{Then divide the extra bonus by the extra ... = 10 points\\}};
\node (cdots) at ($(y part1)!.5!(y part2)$) {$\cdots$};
\node [anchor=north] (cdots 1) at ([yshift=-.2cm]y part1 1.south) {\textcolor{gray}{$\cdots$}};
\node [anchor=north,text width=.15\textwidth,block,minimum height=.8cm,align=center,draw=gray,opacity=.6] (y part1 n-1) at ([yshift=-.8cm]y part1 1.south) {\textcolor{gray}{\scriptsize{Then multiply the current average by the ... = 560 points}\\}};
\node [anchor=north,text width=.15\textwidth,block,minimum height=.8cm,align=center] (y part1 n) at ([yshift=-.5cm]y part1 n-1.south) {\scriptsize{Then subtract the number of points ... = 290 points}\\};
\draw [->] (x.south) -- (y part1.north);
\draw [->] (x.south) -- (y part2.north);
\draw [->] (x.south) -- ($(y part1.north)!.5!(y part2.north)$);
\draw [->] (y part1.south) -- (y part1 1.north);
\draw [->,gray,opacity=.6] (y part1 1.south) -- (cdots 1.north);
\draw [->,gray,opacity=.6] (cdots 1.south) -- (y part1 n-1.north);
\draw [->,gray,opacity=.6] (y part1 n-1) -- (y part1 n);
\draw [->,emphasize] ([xshift=-.5cm]y part1 n.north) .. controls +(-.5cm,+.5cm) and +(-.5cm,-.5cm) .. ([xshift=-.5cm]y part1 1.south);
\draw [->,emphasize] ([xshift=-.5cm]y part1 1.north) .. controls +(-.5cm,+.1cm) and +(-.5cm,-.1cm) .. ([xshift=-.5cm]y part1.south);
\draw [->,emphasize] ([xshift=-.5cm]y part1.north) .. controls +(-.5cm,+.1cm) and +(-.5cm,-.1cm) .. ([xshift=-.5cm]y part1.north|-x.south);
\end{scope}
\end{scope}
\end{tikzpicture}
\end{center}
section5/Figures/figure-step-by-step-with-prm.tex 0 → 100644
% !Mode:: "TeX:UTF-8"
% !TEX encoding = UTF-8 Unicode
\begin{center}
\begin{tikzpicture}
\tikzset {
block/.style={,draw,inner sep=2pt,rounded corners=2pt,fill=white}
}
\begin{scope}
\node (x) at (0,0) {$\mathbf{x}$};
\node [anchor=north,text width=.75\textwidth,block] (x) at (x.south) {\footnotesize{Karen's students are about to take a standardized test. Karen gets a \$500 bonus if their average score is above 75, plus an extra \$10 bonus for every additional point the average score increases above 75. So far, Karen has graded 8 tests, and the average is 70. Given that each student can have a maximum score of 150, what combined score do the last two tests need to have for Karen to earn a \$600 bonus?\\}};
\def\blockheight{2.5cm}
\node [anchor=north,text width=.25\textwidth,block,minimum height=\blockheight] (y part1) at ([yshift=-1cm]x.south) {\footnotesize{Let's denote the combined score of the last two tests as $x$. Given that Karen has graded 8 tests and the average is 70, the sum of the scores of the first 8 tests is $8 \times 70 = 560$.\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part1.south east) {\scriptsize{\textbf{Reward: 7}}};
\node [anchor=east,text width=.25\textwidth,block,minimum height=\blockheight] (y part2) at ([xshift=-.5cm]y part1.west) {\footnotesize{The first step is to calculate how much the average score needs to be above 75 for that extra \$100:\\ Extra points required = $\frac{100}{10}$ = 10\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part2.south east) {\scriptsize{\textbf{Reward: 1}}};
\node [anchor=east,inner sep=0] (lcdots) at ([xshift=-.1cm]y part2.west) {\LARGE{$\cdots$}};
\node [anchor=west,text width=.25\textwidth,block,minimum height=\blockheight,draw=lolred,line width=.05cm] (y part3) at ([xshift=.5cm]y part1.east) {\footnotesize{First subtract \$500 from Karen's goal bonus amount to find how much she makes from the extra \$10/point bonus: \$600 - \$500 = \$100\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part3.south east) {\scriptsize{\textbf{Reward: 8}}};
\node [anchor=west,inner sep=0] (rcdots) at ([xshift=.1cm]y part3.east) {\LARGE{$\cdots$}};
\begin{scope}[on background layer]
\node [anchor=north,fill=black!5!white, rounded corners=2pt, inner sep=8pt, fit=(lcdots)(y part1)(y part2)(y part3)(rcdots)] (y) {};
\end{scope}
\def\blockheight{2.5cm}
\node [anchor=north,text width=.25\textwidth,block,minimum height=\blockheight,draw=lolred,line width=.05cm] (y part1 1) at ([yshift=-1cm]y part1.south) {\footnotesize{Then divide the extra bonus by the extra rate: \$100 / \$10/point = 10 points\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part1 1.south east) {\scriptsize{\textbf{Reward: 9}}};
\node [anchor=east,text width=.25\textwidth,block,minimum height=\blockheight] (y part2 1) at ([xshift=-.5cm]y part1 1.west) {\footnotesize{Then, divide the remaining bonus amount by the extra bonus amount per point to find out how many additional points Karen needs: \$100 / \$10 = \$10\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part2 1.south east) {\scriptsize{\textbf{Reward: 2}}};
\node [anchor=east,inner sep=0] (lcdots) at ([xshift=-.1cm]y part2 1.west) {\LARGE{$\cdots$}};
\node [anchor=west,text width=.25\textwidth,block,minimum height=\blockheight] (y part3 1) at ([xshift=.5cm]y part1 1.east) {\footnotesize{Then, divide the \$100 bonus by \$10/point to find the additional average score Karen needs: \$100 / \$10/point = 10 additional points.\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part3 1.south east) {\scriptsize{\textbf{Reward: -1}}};
\node [anchor=west,inner sep=0] (rcdots) at ([xshift=.1cm]y part3 1.east) {\LARGE{$\cdots$}};
\begin{scope}[on background layer]
\node [anchor=north,fill=black!5!white, rounded corners=2pt, inner sep=8pt, fit=(lcdots)(y part1 1)(y part2 1)(y part3 1)(rcdots)] (y 1) {};
\end{scope}
\def\blockheight{3.5cm}
\node [anchor=north,text width=.25\textwidth,block,minimum height=\blockheight] (y part1 2) at ([yshift=-1cm]y part1 1.south) {\footnotesize{Then, multiply the number of additional points needed by the number of tests Karen has yet to grade, which is 2: 10 points * 2 tests = 20 points.\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part1 2.south east) {\scriptsize{\textbf{Reward: -4}}};
\node [anchor=east,text width=.25\textwidth,block,minimum height=\blockheight] (y part2 2) at ([xshift=-.5cm]y part1 2.west) {\footnotesize{Then, Karen needs the combined score of the last two tests to be 10 points higher than the average of 75. Since there are two tests, the combined score for the last two tests needs to be 20 points higher than the average.\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part2 2.south east) {\scriptsize{\textbf{Reward: 3}}};
\node [anchor=east,inner sep=0] (lcdots) at ([xshift=-.1cm]y part2 2.west) {\LARGE{$\cdots$}};
\node [anchor=west,text width=.25\textwidth,block,minimum height=\blockheight,draw=lolred,line width=.05cm] (y part3 2) at ([xshift=.5cm]y part1 2.east) {\footnotesize{Then add the 10 extra points to the baseline 75 point goal to find the students' average test score: 10 points + 75 points = 85 points\\}};
\node [anchor=south east, draw,fill=black!5!white] at (y part3 2.south east) {\scriptsize{\textbf{Reward: 10}}};
\node [anchor=west,inner sep=0] (rcdots) at ([xshift=.1cm]y part3 2.east) {\LARGE{$\cdots$}};
\begin{scope}[on background layer]
\node [anchor=north,fill=black!5!white, rounded corners=2pt, inner sep=8pt, fit=(lcdots)(y part1 2)(y part2 2)(y part3 2)(rcdots)] (y 2) {};
\end{scope}
\draw [->] (x.south) -- ([yshift=.28cm]y part1.north);
\draw [->] (y part3.south) .. controls +(0,-.8cm) and +(0,+.8cm) .. ([yshift=.28cm]y part1 1.north);
\draw [->] (y part1 1.south) -- ([yshift=.28cm]y part1 2.north);
\draw [->,lolblue] (y part3.south) node [xshift=1.0cm,yshift=-.5cm,right] {\footnotesize{\textbf{Self-refinement}}} arc [start angle=225, end angle=495, radius=1.95cm];
\node [anchor=north] (cdots) at ([yshift=-1cm]y part1 2.south) {\LARGE{$\cdots$}};
\draw [->] (y part3 2.south) .. controls +(0,-.8cm) and +(0,+.8cm) .. (cdots.north);
\end{scope}
\end{tikzpicture}
\end{center}
section6/section6.tex 0 → 100644
\section{Reinforcement Learning for Multimodal Models}
Multimodal learning involves models that process and relate information from multiple data modalities (e.g. vision, text, speech), enabling more comprehensive understanding than single-modality systems. By combining modalities, models can make more robust predictions and capture complementary information that one modality alone might miss. Such multimodal approaches have benefits across various applications, such as image caption and image generation. Given these advantages, multimodal learning has become increasingly significant as AI systems aim to perceive and reason more like humans, who naturally integrate sight, sound, and language \citep{baltruvsaitis-etal:2018multimodal}.
In the era of LLM, we can extend their capabilities into the multimodal domain by training them with diverse data types. For example, we can develop a visual language model by training an LLM with image-text pairs using SFT. Here, we return to the issue of aligning models with human preferences. Ideally, once aligned with human preferences through RL, the LLM would also generalize the target modality well by the SFT. However, the reality does not always meet expectations. Although an LLM may align well with human preferences in one aspect, it often struggles to generalize this alignment across a different modality. Thus, to align multimodal models with human preferences, we perform RL to enhance their performance further within the specific modality.
In this section, we use the visual language, speech generation, and diffusion models to discuss the application of RL in multimodal models.
\subsection{Visual Language Models}
While RL is commonly used to train LLMs, its application to other domains has been a prominent research topic. In multimodal language models\footnote{A multimodal language model is defined as a model that integrates an LLM with a multimodal encoder, such as CLIP \citep{radford-etal:2021learning}, allowing the LLM to process inputs beyond text, such as images. Recent literature has also introduced the use of LLMs for generating outputs in non-text modalities, such as images and speech \citep{xu-etal:2025qwen2,zhang-etal:2024mm}. However, in this section, we focus on the former definition of multimodal language models.}, for example, a notable trend is to perform RL training to improve their trustworthiness and helpfulness. This section considers Visual Language Models (VLMs), which connect a visual encoder to an LLM through a linear projector, facilitating general-purpose visual and language understanding. VLMs are currently the most explored extension in multimodal language research, and they form the foundation for many open-source multimodal language models, such as Qwen2.5-VL \citep{qwenTeam:2025qwen2.5-VL} and LLaMA-3.2-11B-Vision \citep{grattafiori-etal:2024llama}.
Training LLMs and VLMs with RL exhibits only minimal differences, primarily related to the input content. Unlike the textual input used for LLMs, the input for VLMs typically comprises a combination of one or multiple images and an instruction, denoted by $(\mathrm{\mathbf{I}}, \mathrm{\mathbf{x}})$, where $\mathrm{\mathbf{I}}$ represents the input images. These images are encoded into representations that are either concatenated with instruction embeddings or integrated through cross-attention mechanisms into the LLM. In practice, this subtle difference does not significantly affect the applicability of RL algorithms to VLMs. As a result, the RL training process for LLMs, as described in Section \ref{sec:example-using-rl-training-llms}, can be seamlessly adapted to train VLMs without major improvements \citep{yu-etal:2024rlhf,wang-etal:2024rovrm,zang2025internlm,ji2025safe}.
\begin{figure*}[!t]
\centering
\caption{RoVRM}
\label{fig:preference-transfer}
\end{figure*}
However, training VLMs with RL is not a low-hanging fruit in practical applications. This is because it typically encounters the challenge of training a visual reward model due to the scarcity of high-quality visual preference data. One straightforward approach to address this issue is to generate visual preference data through the automatic preference data generation method described in Section \ref{sec:automatic-preference-data-generation} \citep{yu-etal:2024rlaif}. Another alternative is a multi-stage training approach for the visual reward model, motivated by a simple idea: human preferences are well captured in text, and these preferences can be transferred across modalities \citep{wang-etal:2024rovrm}. By leveraging textual preference data, this approach reduces the dependence on visual preference data in training a visual reward model. More specifically, as illustrated in Figure \ref{fig:preference-transfer}, we can train a visual reward model in the following three stages:
\begin{itemize}
\item Stage 1: pre-training with large-scale textual preference data. Given the transferability of human preferences across different modalities, we can begin by using large-scale textual preference data to pre-train the visual reward model. Note that the projector parameters are frozen without images. This stage can be considered as providing a stronger starting point for training the visual reward model, as it enables the model to pre-learn general human preferences.
\item Stage 2: fine-tuning with image caption-based preference data. Pre-learned human preferences cannot be directly applied to vision tasks due to both \textit{task gap} and \textit{modality gap}. To bridge the task gap, we fine-tune the reward model using image caption-based preference data in this stage. The rationale behind this approach is that general textual preference data does not cover vision-specific tasks, such as ``Please describe the content in this image''. We use such data to fine-tune the model, adapting it to vision-specific tasks. The projector parameters are frozen at this stage.
\item Stage 3: fine-tuning with small-scale visual preference data. To further bridge the modality gap, we use visual preference data in the final stage to train the model. Note that in this phase, we also train the projector parameters.
\end{itemize}
In this process, not all preference data may align with the preferences used in subsequent phases, potentially leading to preference conflicts. We can enhance the visual reward model through data selection techniques, such as LESS \citep{xia-etal:2024less} to address this. In fact, preference transfer is effective across modalities and has also been shown to work across different tasks and languages \citep{cheng-etal:2023everyone,wu-etal:2024reuse}. Interested readers can refer to these papers for more detailed discussions of these topics.
Apart from preference data, another approach to improving the visual model is to integrate additional image content, such as image captions, into the reward model \citep{sun-etal:2023aligning}. This approach aims to achieve factually augmented reward prediction, addressing reward hacking. Specifically, in the original setup, the reward model predicts a reward based solely on the image, input, and output; that is, the reward model’s input is $[\mathbf{I}, \mathbf{x}, \mathbf{y}]$. In the factually augmented setup, the reward model also receives additional input in the form of the textual image caption $\mathbf{C}$, resulting in an input of $[\mathbf{I}, \mathbf{C}, \mathbf{x}, \mathbf{y}]$. The basic idea is that the backbone of the visual reward model remains a well-trained LLM with an in-context learning ability. With this ability, we can provide additional content to help the model predict rewards more accurately.
While our discussion primarily focuses on models for visual inputs in the context of visual language models, the RL techniques are adaptable across inputs of various modalities, such as video and audio.
% visual language model as reward model
% 视觉奖励模型训练大致思路
% image reward
% 融合文本数据进行提升
% 视觉奖励模型应用场景---对齐图生文模型 文生图评估模型
% 视觉语言模型的定义
% 框架
% Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback
% Aligning Large Multimodal Models with Factually Augmented RLHF
% 偏好数据
% RLAIF-V
% Silkie: Preference Distillation for Large Visual Language Models
% RoVRM
\subsection{Speech Generation Models}
% SpeechAlign: Aligning Speech Generation to Human Preferences
\subsection{Diffusion Models}
% Image Reward
% 为什么要进行强化学习
% 大体训练思路和流程
% DDPO (denoising diffusion preference optimization model)
section7/section7.tex 0 → 100644
\section{Summary}
% future work
% 强化学习作为一种预训练方式
% 高效强化学习方法
% 全模态RL
\ No newline at end of file
section8/section8.tex 0 → 100644
\clearpage
\section{Systems and Datasets}
\begin{table}[h]
\centering
\scalebox{0.88}{
\input{section8/tables/dataset}}
\caption{datasets}
\label{tab:dataset}
\end{table}
\begin{table}[h]
\centering
\scalebox{0.88}{
\input{section8/tables/systems}}
\caption{systems}
\label{tab:system}
\end{table}
\ No newline at end of file
section8/tables/dataset.tex 0 → 100644
\begin{tabular}{lccccc}
\toprule[1.1pt]
\multirow{2}{*}{Dataset Name} & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Sample\\Size\end{tabular}} & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Response\\Size\end{tabular}} & \multirow{2}{*}{Modality} & \multicolumn{2}{c}{Feedback} \\ \cmidrule(l){5-6}
& & & & Source & Category \\ \midrule
\href{https://url}{HuggingFaceH4/stack-exchange-preferences}
&10,000 &1 & \faIcon{file-alt} & Human & Score \\
\href{https://url}{Skywork/Skywork-Reward-Preference-80K-v0.2} &10,000 &2 & \faIcon{images} & AI & Ranking \\
\href{https://huggingface.co/datasets/openbmb/UltraFeedback}{openbmb/UltraFeedback} &10,000 &3 & \faIcon{video} & Rule & \\
&10,000 &4 & \faIcon{volume-up} & & \\
& & &\faIcon{file-alt} \faIcon{images} & & \\
& & & & & \\
& & & & & \\
& & & & & \\
\bottomrule[1.1pt]
\end{tabular}
\ No newline at end of file
section8/tables/systems.tex 0 → 100644
\begin{tabular}{lccccc}
\toprule[1.1pt]
\multirow{2}{*}{System Name} & \multicolumn{4}{c}{Supported Modality} & \multirow{2}{*}{Supported Training Approaches} \\ \cmidrule(r){2-5}
& Text & Image & Video & Audio & \\ \midrule
\href{https://github.com/huggingface/trl}{TRL} & \CheckmarkBold & \CheckmarkBold &\XSolidBrush & \CheckmarkBold & Reward Modeling, PPO, GRPO, DPO, Online-DPO, etc. \\
\href{https://github.com/OpenRLHF/OpenRLHF}{OpenRLHF}& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush & Sft, Reject Sampling, PPO, GRPO, DPO, KTO, etc.\\
\href{https://github.com/hiyouga/EasyR1}{EasyR1}& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush & GRPO, Reinforce++, Remax, RLOO, etc. \\
\href{https://github.com/volcengine/verl}{veRL}& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush & GRPO, PPO, Remax, RLOO, SFT, etc. \\
\href{https://github.com/OpenRLHF/OpenRLHF-M}{OpenRLHF-M}& \XSolidBrush & \CheckmarkBold & \XSolidBrush & \XSolidBrush & PPO, GRPO, RLOO, Online-RLHF, Reject-Sampling, etc. \\
\href{https://github.com/PKU-Alignment/align-anything}{Align-Anything}& \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold &PPO, GRPO, DPO,KTO, ORPO, etc. \\
\bottomrule[1.1pt]
\end{tabular}
\ No newline at end of file
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论