Skip to content

R
rl-introduction
Commit 74c03485 by wangchenglong
Options

update.

parent 3e338e12
正在显示 包含 154 行增加 和 7 行删除
appendix/appendix.tex 0 → 100644
\clearpage
\section*{Appendix A: Useful Systems and Datasets}
\phantomsection
\addcontentsline{toc}{section}{Appendix A: Useful Systems and Datasets}
\begin{table}[h]
\centering
\resizebox{0.98\linewidth}{!}{
\input{appendix/tables/dataset}}
\caption{Preference, reasoning, and trainable agentic RL datasets and environments for LLM. Here, RM denotes reward modeling, Rsn. denotes reasoning, Env. denotes environment-based feedback, Env. gen. denotes procedurally generated environments, and TIR denotes tool-integrated reasoning.}
\label{tab:dataset}
\end{table}
\clearpage
% \section*{Appendix B: Useful Datasets}
% \phantomsection
% \addcontentsline{toc}{section}{Appendix B: Useful Datasets}
\begin{table}[h]
\centering
\resizebox{0.98\linewidth}{!}{
\input{appendix/tables/systems}}
\caption{RL systems for LLM and multimodal post-training, including their supported modalities and corresponding training approaches.}
\label{tab:system}
\end{table}
appendix/tables/dataset.tex 0 → 100644
\scriptsize
\renewcommand{\arraystretch}{1.02}
\begin{tabular}{@{}>{\raggedright\arraybackslash}p{3.3cm}ccc>{\raggedright\arraybackslash}p{1.6cm}c>{\raggedright\arraybackslash}p{1.7cm}>{\raggedright\arraybackslash}p{2.9cm}@{}}
\toprule[1.1pt]
\multirow{2}{*}{Dataset Name} & \multicolumn{3}{c}{Use Case} & \multirow{2}{*}{Scale} & \multirow{2}{*}{Modality} & \multicolumn{2}{c}{Feedback / Reward Signal} \\ \cmidrule(r){2-4}\cmidrule(l){7-8}
& \begin{tabular}[c]{@{}c@{}}\faIcon{star}\\RM\end{tabular}
& \begin{tabular}[c]{@{}c@{}}\faIcon{calculator}\\Rsn.\end{tabular}
& \begin{tabular}[c]{@{}c@{}}\faIcon{robot}\\Agent\end{tabular}
& & & Source & Category \\ \midrule
\href{https://huggingface.co/datasets/Anthropic/hh-rlhf}{Anthropic/hh-rlhf}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 169K & \faIcon{file-alt} & Human & Pairwise preference \\
\href{https://huggingface.co/datasets/stanfordnlp/SHP-2}{stanfordnlp/SHP-2}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 4.8M & \faIcon{file-alt} & Human / crowd & Pairwise preference \\
\href{https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences}{H4/stack-exchange-preferences}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 10.7M & \faIcon{file-alt} & Crowd score & Score / ranking \\
\href{https://huggingface.co/datasets/openai/summarize_from_feedback}{Summarize from Feedback}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 64.8K & \faIcon{file-alt} & Human & Pairwise preference \\
\href{https://huggingface.co/datasets/openbmb/UltraFeedback}{UltraFeedback}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 64K / 340K & \faIcon{file-alt} & GPT-4 & Scores, critiques, pairs \\
\href{https://huggingface.co/datasets/berkeley-nest/Nectar}{berkeley-nest/Nectar}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 183K / 3.8M & \faIcon{file-alt} & GPT-4 & 7-way ranking \\
\href{https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2}{Skywork-Reward-Preference-80K}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 80K & \faIcon{file-alt} & Mixed / curated & Pairwise preference \\
\href{https://huggingface.co/datasets/nvidia/HelpSteer}{nvidia/HelpSteer}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 37K & \faIcon{file-alt} & Human & Attribute scores \\
\href{https://huggingface.co/datasets/nvidia/HelpSteer2}{nvidia/HelpSteer2}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 10K pairs & \faIcon{file-alt} & Human & Attribute scores / preference \\
\href{https://huggingface.co/datasets/nvidia/HelpSteer3}{nvidia/HelpSteer3}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 40K pref. & \faIcon{file-alt} & Human & Preference, feedback, edits \\
\href{https://huggingface.co/datasets/lmarena-ai/arena-human-preference-55k}{Arena Human Preference 55K}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 55K & \faIcon{file-alt} & Human users & Arena battle preference \\
\href{https://huggingface.co/datasets/MMInstruction/VLFeedback}{VLFeedback}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 80K / 380K & \faIcon{file-alt} \faIcon{images} & GPT-4V & Multimodal preference \\
\href{https://github.com/RLHF-V/RLHF-V}{RLHF-V-Dataset}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 5.7K & \faIcon{file-alt} \faIcon{images} & Human & Fine-grained correction \\
\href{https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset}{openbmb/RLAIF-V-Dataset}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & 83K & \faIcon{file-alt} \faIcon{images} & AI feedback & Pairwise preference \\
\href{https://huggingface.co/datasets/OpenGVLab/MMPR-v1.2}{OpenGVLab/MMPR-v1.2}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & 3M & \faIcon{file-alt} \faIcon{images} & AI / verifier & Multimodal reasoning preference \\
\href{https://huggingface.co/datasets/open-r1/OpenR1-Math-220k}{open-r1/OpenR1-Math-220k}
& \XSolidBrush & \CheckmarkBold & \XSolidBrush & 220K & \faIcon{file-alt} & Verifier / judge & Math reasoning traces \\
\href{https://huggingface.co/collections/AI-MO/numinamath}{AI-MO/NuminaMath}
& \XSolidBrush & \CheckmarkBold & \XSolidBrush & 900K & \faIcon{file-alt} & Rule / GPT-4 & Math CoT / TIR \\
\href{https://huggingface.co/datasets/PRIME-RL/Eurus-2-RL-Data}{PRIME-RL/Eurus-2-RL-Data}
& \XSolidBrush & \CheckmarkBold & \XSolidBrush & 482K & \faIcon{file-alt} & Verifier & Math/code outcome reward \\
\href{https://huggingface.co/datasets/AgentGym/AgentGym-RL-Data-ID}{AgentGym-RL-Data}
& \XSolidBrush & \CheckmarkBold & \CheckmarkBold & 184K & \faIcon{file-alt} & Env. & Multi-turn agent reward \\
\href{https://github.com/alfworld/alfworld}{ALFWorld}
& \XSolidBrush & \XSolidBrush & \CheckmarkBold & 3.5K train & \faIcon{file-alt} & Env. & Text-world task reward \\
\href{https://github.com/princeton-nlp/WebShop}{WebShop}
& \XSolidBrush & \XSolidBrush & \CheckmarkBold & 12K tasks & \faIcon{file-alt} & Env. & Web shopping reward \\
\href{https://github.com/PeterGriffinJin/Search-R1}{Search-R1 Data}
& \XSolidBrush & \CheckmarkBold & \CheckmarkBold & 170K train & \faIcon{file-alt} & Retriever / rule & Search-tool QA reward \\
\href{https://github.com/langfengQ/verl-agent}{Sokoban}
& \XSolidBrush & \CheckmarkBold & \CheckmarkBold & Env. gen. & \faIcon{file-alt} \faIcon{images} & Env. & Puzzle-solving reward \\
\href{https://github.com/langfengQ/verl-agent}{Gym Cards}
& \XSolidBrush & \CheckmarkBold & \CheckmarkBold & Env. gen. & \faIcon{file-alt} \faIcon{images} & Env. & Logic-game reward \\
\href{https://github.com/OpenBMB/ToolBench}{ToolBench}
& \XSolidBrush & \CheckmarkBold & \CheckmarkBold & 126K & \faIcon{file-alt} & API execution & Tool-use trajectories \\
\bottomrule[1.1pt]
\end{tabular}
appendix/tables/systems.tex 0 → 100644
\renewcommand{\arraystretch}{1.15}
\begin{tabular}{@{}>{\raggedright\arraybackslash}p{2.7cm}cccc>{\raggedright\arraybackslash}p{9.5cm}@{}}
\toprule[1.1pt]
\multirow{2}{*}{System Name} & \multicolumn{4}{c}{Supported Modality} & \multirow{2}{*}{Supported Training Approaches} \\ \cmidrule(r){2-5}
& Text & Image & Video & Audio & \\ \midrule
\href{https://github.com/huggingface/trl}{TRL}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& SFT, Reward Modeling, PPO, GRPO, RLOO, DPO, KTO, ORPO, Online-DPO; Agentic RL via OpenEnv/Harbor. \\
\href{https://github.com/OpenRLHF/OpenRLHF}{OpenRLHF}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& SFT, RM, PPO, REINFORCE++, GRPO, RLOO, Dr.GRPO, DPO, KTO; Agentic RL with single-turn/multi-turn executors.\\
\href{https://github.com/verl-project/verl}{verl}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& SFT, PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, DAPO, Dr.GRPO; Agentic RL via multi-turn tool calling. \\
\href{https://github.com/langfengQ/verl-agent}{verl-agent}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& GiGPO, GRPO, PPO, DAPO, GSPO, RLOO, REINFORCE++, LoRA; purpose-built Agentic RL for long-horizon LLM/VLM agents. \\
\href{https://github.com/hiyouga/EasyR1}{EasyR1}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& GRPO, DAPO, REINFORCE++, ReMax, RLOO, GSPO, CISPO; no native Agentic RL. \\
\href{https://github.com/hiyouga/LLaMA-Factory}{LLaMA-Factory}
& \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold
& Pre-training, SFT, RM, PPO, DPO, KTO, ORPO, SimPO; no native Agentic RL. \\
\href{https://github.com/ByteDance-Seed/VeOmni}{VeOmni}
& \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold
& Single-/multi-modal pre-training and post-training, SFT-style training, DPO, RL trainer backend; no native Agentic RL environment stack. \\
\href{https://github.com/modelscope/ms-swift}{ms-swift}
& \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold
& Pre-training, SFT, RM, PPO, GRPO, DPO, KTO, ORPO, CPO, SimPO, GKD; Agentic RL via multi-turn GRPO/tool-use training. \\
\href{https://github.com/PKU-Alignment/align-anything}{Align-Anything}
& \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold
& SFT, RM, PPO, DPO, KTO, ORPO, SimPO, rule-based RL; Agent RL on roadmap. \\
\href{https://github.com/OpenRLHF/OpenRLHF-M}{OpenRLHF-M}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& PPO, GRPO, RLOO, Online-RLHF, Rejection Sampling for multimodal models; no native Agentic RL. \\
\href{https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat}{DeepSpeed-Chat}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& SFT, Reward Modeling, PPO-based RLHF; no native Agentic RL. \\
\href{https://github.com/alibaba/ROLL}{ROLL}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& SFT, DPO, distillation, PPO, GRPO, REINFORCE++, GSPO, RAFT++, StarPO, GiGPO; Agentic RL. \\
\href{https://github.com/THUDM/slime}{slime}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& PPO, GRPO, GSPO, REINFORCE++, OPD; Agentic RL via custom generation, tools, sandboxes, and verifier rewards. \\
\href{https://github.com/Gen-Verse/OpenClaw-RL}{OpenClaw-RL}
& \CheckmarkBold & \CheckmarkBold & \XSolidBrush & \XSolidBrush
& Binary RL/GRPO, OPD, Hybrid RL, LoRA training; asynchronous Agentic RL for personalized agents, terminal, GUI, SWE, and tool-call settings. \\
\href{https://github.com/NovaSky-AI/SkyRL}{SkyRL}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& GRPO, DAPO, async RL, Tinker-compatible training; Agentic RL for tool-use, search, SQL, and long-horizon tasks. \\
\href{https://github.com/microsoft/agent-lightning}{Agent Lightning}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& RL, APO, SFT-style optimization hooks; purpose-built Agentic RL for existing agent frameworks. \\
\href{https://github.com/mll-lab-nu/RAGEN}{RAGEN}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& PPO/StarPO-style multi-turn optimization, rollout filtering, environment rewards; Agentic RL. \\
\href{https://github.com/PeterGriffinJin/Search-R1}{Search-R1}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& PPO, GRPO, REINFORCE for reasoning-search interleaved models; Agentic RL for search/tool use. \\
\href{https://github.com/areal-project/AReaL}{AReaL}
& \CheckmarkBold & \XSolidBrush & \XSolidBrush & \XSolidBrush
& PPO/GRPO-style asynchronous RL with agent-framework integration; Agentic RL. \\
\bottomrule[1.1pt]
\end{tabular}
section8/section8.tex
......@@ -12,9 +12,4 @@ While RL has become well established in LLM training and has achieved remarkable
\item \textbf{Predicting the scaling limits of RL.} The final gains from RL depend on many factors, including model capability, training data, reward quality, and optimization settings. In practice, we often need to complete expensive RL runs before knowing the final performance. Developing methods to predict the potential gains and performance limits of RL before full-scale training could significantly improve the efficiency of model development.
\item \textbf{Continual and self-evolving RL.} Most current RL pipelines optimize a model on a fixed training distribution and stop once training is completed. One more ambitious direction is to enable models to continuously learn from their own interactions, feedback, and accumulated experiences after deployment. This requires new mechanisms for experience selection, memory and skill evolution, and stable policy updating without catastrophic forgetting.
\end{itemize}
% future work
% 强化学习作为一种预训练方式
% 高效强化学习方法
% 如何预测RL训练的上届？RL训练往往和Model performance, data, 参数有关系。通常，我们仅仅只能通过完整的训练才能的得知最后的性能。
\ No newline at end of file
\end{itemize}
\ No newline at end of file
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论