@@ -239,30 +239,40 @@ In application, this reward decomposition provides denser feedback than final-an
...
@@ -239,30 +239,40 @@ In application, this reward decomposition provides denser feedback than final-an
Another direction is to use RL to improve strategic tool use. For example, ReTool first builds a cold-start model with code-augmented reasoning traces and then applies RL with real-time code execution. During rollout, the model interleaves natural language reasoning with code execution, observes execution results, and revises its subsequent reasoning. This allows the model to discover when and how to invoke a code interpreter based on outcome feedback rather than human-written rules \citep{feng-etal:retool}. Search-R1 and ReSearch follow a similar idea in retrieval-augmented reasoning. They train models to generate search queries during reasoning and use retrieved information to update later steps, rather than relying on fixed retrieval pipelines or hand-crafted search prompts \citep{jin-etal:searchr1,chen-etal:research}.
Another direction is to use RL to improve strategic tool use. For example, ReTool first builds a cold-start model with code-augmented reasoning traces and then applies RL with real-time code execution. During rollout, the model interleaves natural language reasoning with code execution, observes execution results, and revises its subsequent reasoning. This allows the model to discover when and how to invoke a code interpreter based on outcome feedback rather than human-written rules \citep{feng-etal:retool}. Search-R1 and ReSearch follow a similar idea in retrieval-augmented reasoning. They train models to generate search queries during reasoning and use retrieved information to update later steps, rather than relying on fixed retrieval pipelines or hand-crafted search prompts \citep{jin-etal:searchr1,chen-etal:research}.
So far, we have introduced how RL can enhance the core capabilities of LLM-based agents, including planning and tool use. Since an LLM-based agent is inherently driven by an underlying LLM, its RL training can reuse the general LLM-based RL algorithms introduced in Section \ref{sec:example-using-rl-training-llms}. The main differences lie in the agentic setting. Compared with standard response optimization, agentic RL involves longer trajectories, environmental feedback, and intermediate decisions such as planning and tool invocation. Thus, the key challenges are not only the choice of RL algorithms, but also trajectory construction, reward design, and reward sparse.
\subsubsection{Improved Environments}
\subsection{Training-free Methods}
So far, we have introduced how RL can enhance the core capabilities of LLM-based agents, including planning and tool use. Since an LLM-based agent is inherently driven by an underlying LLM, its RL training can reuse the general LLM-based RL algorithms introduced in Section \ref{sec:example-using-rl-training-llms}. The main differences lie in the agentic setting. Compared with standard response optimization, agentic RL involves longer trajectories, environmental feedback, and intermediate decisions such as planning and tool invocation. Thus, the key challenges are not only the choice of RL algorithms, but also trajectory construction, reward design, and environment optimization.
