@@ -5,6 +5,7 @@ So far, our discussion has mainly focused on various aspects of using and improv
...
@@ -5,6 +5,7 @@ So far, our discussion has mainly focused on various aspects of using and improv
In this section, we delve deeper into the application of RL to enhance the reasoning capabilities of the LLM, a topic that has recently garnered significant attention. We begin by giving a general introduction to test-time scaling that fully unleashes the reasoning potential of LLMs, including best-of-$N$ sampling, step-by-step verification, and Monte Carlo Tree Search. We then discuss two critical issues: how to scale RL effectively, and how to iterate the RL process to enhance the reasoning capabilities of LLMs. Note that the following methods are primarily illustrated through the mathematical reasoning problem, they are applicable to a broad range of decision-making problems.
In this section, we delve deeper into the application of RL to enhance the reasoning capabilities of the LLM, a topic that has recently garnered significant attention. We begin by giving a general introduction to test-time scaling that fully unleashes the reasoning potential of LLMs, including best-of-$N$ sampling, step-by-step verification, and Monte Carlo Tree Search. We then discuss two critical issues: how to scale RL effectively, and how to iterate the RL process to enhance the reasoning capabilities of LLMs. Note that the following methods are primarily illustrated through the mathematical reasoning problem, they are applicable to a broad range of decision-making problems.
\subsection{Test-time Scaling}
\subsection{Test-time Scaling}
\label{sec:tts}
Initially, we review the fundamental objective of RL: to maximize the rewards obtained during output generation. This goal has been extensively achieved through various training-time optimization techniques, such as policy gradient or PPO algorithms. Beyond training, recent research has illuminated the benefits of scaling up test-time computation, a process also known as test-time scaling, which can significantly enhance the maximization of rewards, especially in the reasoning task. In a practical implementation, one simple approach to achieve test-time scaling is the use of prompting techniques. For example, appending the phrase ``Let's think step-by-step.'' to the input can effectively stimulate the LLM to engage in a more detailed reasoning process during the generation. While this approach can improve reasoning accuracy, it often leads to suboptimal performance because the model is not inherently trained to understand the most beneficial reasoning processes. To address this issue, we can perform the test-time scaling with guidance from a reward model, as discussed in the following subsections.
Initially, we review the fundamental objective of RL: to maximize the rewards obtained during output generation. This goal has been extensively achieved through various training-time optimization techniques, such as policy gradient or PPO algorithms. Beyond training, recent research has illuminated the benefits of scaling up test-time computation, a process also known as test-time scaling, which can significantly enhance the maximization of rewards, especially in the reasoning task. In a practical implementation, one simple approach to achieve test-time scaling is the use of prompting techniques. For example, appending the phrase ``Let's think step-by-step.'' to the input can effectively stimulate the LLM to engage in a more detailed reasoning process during the generation. While this approach can improve reasoning accuracy, it often leads to suboptimal performance because the model is not inherently trained to understand the most beneficial reasoning processes. To address this issue, we can perform the test-time scaling with guidance from a reward model, as discussed in the following subsections.
@@ -463,28 +463,33 @@ Here we consider \textbf{SkillRL} as an example to illustrate how to optimize th
...
@@ -463,28 +463,33 @@ Here we consider \textbf{SkillRL} as an example to illustrate how to optimize th
\end{itemize}
\end{itemize}
By repeating the above process, the agent can improve its skill library through accumulated experiences. Specifically, newly collected trajectories provide additional evidence for discovering new skills or refining existing ones, while the updated skill library further guides future RL training. This creates an iterative learning process in which the agent and its skills co-evolve over time. From this perspective, skill optimization provides a practical way to transform low-level interaction experiences into reusable capabilities, and has become an important approach for enabling agents to continuously learn from their own experiences \citep{wang-etal:reinforcement}.
By repeating the above process, the agent can improve its skill library through accumulated experiences. Specifically, newly collected trajectories provide additional evidence for discovering new skills or refining existing ones, while the updated skill library further guides future RL training. This creates an iterative learning process in which the agent and its skills co-evolve over time. From this perspective, skill optimization provides a practical way to transform low-level interaction experiences into reusable capabilities, and has become an important approach for enabling agents to continuously learn from their own experiences \citep{wang-etal:reinforcement,fu-etal:self}.
\subsubsection{Trajectory Refinement}
\subsubsection{Trajectory Refinement}
Learning from early experience.
Different from memory management and skill optimization, which focus on storing and abstracting experiences, trajectory refinement directly improves agent behaviors by revising generated trajectories. This idea is motivated by the self-refinement capability of LLMs. As discussed in Section~\ref{sec:tts}, reasoning-enhanced RL training enables LLMs to evaluate their intermediate reasoning processes and revise incorrect solutions. For example, when solving mathematical problems, an LLM can verify intermediate results after each reasoning step and adjust its solution trajectory based on the verification outcomes. A natural extension is to apply this self-refinement capability to LLM-based agents. During interaction with environments, agents continuously receive external feedback, such as skill execution results. Instead of directly following the initial trajectory, agents can leverage these feedback signals to revise previous decisions. In this way, the agent learns from its own interaction experience by continuously refining its behaviors.
Using the feedback to refine the trajectories.
Feedback is very important. How to obtain feedback from convention?
OpenClaw-RL: Train Any Agent Simply by Talking
Training-free: \\
Reflexion: Language Agents with Verbal Reinforcement Learning \\
PRACT: Optimizing Principled Reasoning and Acting of LLM Agent \\
% \subsection{Self-Evolving Agents}
Training-based: \\
Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (DPO training) \\
Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training (Self-Training) \\
