title={Toolrl: Reward is all tool learning needs},
author={Qian, Cheng and Acikgoz, Emre Can and He, Qi and Wang, Hongru and Chen, Xiusi and Hakkani-Tur, Dilek and Tur, Gokhan and Ji, Heng},
journal={Advances in Neural Information Processing Systems},
volume={38},
pages={105523--105553},
year={2026}
}
@article{schick-etal:toolformer,
title={Toolformer: Language models can teach themselves to use tools},
author={Schick, Timo and Dwivedi-Yu, Jane and Dess{\`\i}, Roberto and Raileanu, Roberta and Lomeli, Maria and Hambro, Eric and Zettlemoyer, Luke and Cancedda, Nicola and Scialom, Thomas},
journal={Advances in neural information processing systems},
volume={36},
pages={68539--68551},
year={2023}
}
@article{yao-etal:react,
title={React: Synergizing reasoning and acting in language models},
author={Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan},
journal={arXiv preprint arXiv:2210.03629},
year={2022}
}
@article{wang-etal:steppo,
@article{wang-etal:steppo,
title={StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning},
title={StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning},
author={Wang, Daoyu and Li, Qingchuan and Cheng, Mingyue and Ouyang, Jie and Yu, Shuo and Liu, Qi and Chen, Enhong},
author={Wang, Daoyu and Li, Qingchuan and Cheng, Mingyue and Ouyang, Jie and Yu, Shuo and Liu, Qi and Chen, Enhong},
@@ -13,11 +13,11 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
...
@@ -13,11 +13,11 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
\subsection{Building Agent Capabilities}
\subsection{Building Agent Capabilities}
One feature of LLM-based agents is their capacity to interpret user instructions, autonomously formulate a step-by-step plan to resolve the given task, and systematically execute those steps. In applications, this planning process inherently involves coordinating interactions with external environments and determining appropriate tool use. The agent then relies on this structured plan as a blueprint to guide its execution trajectory over multiple turns.
One feature of LLM-based agents is their capacity to interpret user instructions, autonomously formulate a step-by-step plan to resolve the given task, and systematically execute those steps. In applications, this planning process inherently involves coordinating interactions with external environments and determining appropriate tool using. The agent then relies on this structured plan as a blueprint to guide its execution trajectory over multiple turns.
To accomplish this process, an LLM-based agent relies on several complementary capabilities. Some of these capabilities are already inherited from LLMs. Recent advances in large-scale pretraining and post-training have significantly improved LLMs in instruction following, reasoning, code generation, and general world knowledge. These capabilities provide a strong foundation for agentic behavior, enabling LLMs to understand complex user requests and generate coherent intermediate reasoning steps without explicit interaction with the real-world environment. The rapid improvement of these intrinsic capabilities has been a key factor driving the emergence of LLM-based agents, making it increasingly feasible for a single model to serve as the reasoning and decision-making core of an autonomous system.
To accomplish this process, an LLM-based agent relies on several complementary capabilities. Some of these capabilities are already inherited from LLMs. Recent advances in large-scale pretraining and post-training have significantly improved LLMs in instruction following, reasoning, code generation, and general world knowledge. These capabilities provide a strong foundation for agentic behavior, enabling LLMs to understand complex user requests and generate coherent intermediate reasoning steps without explicit interaction with the real-world environment. The rapid improvement of these intrinsic capabilities has been a key factor driving the emergence of LLM-based agents, making it increasingly feasible for a single model to serve as the reasoning and decision-making core of an autonomous system.
However, autonomous agents require capabilities that go beyond static reasoning. They must make sequential decisions over long horizons, coordinate multiple tool use, adapt to unexpected environmental feedback, and recover from execution failures. These capabilities cannot be effectively learned from static supervision alone because their quality depends on the outcomes of complete interaction trajectories rather than individual outputs. RL naturally addresses this challenge by optimizing behaviors through trial-and-error interaction, enabling agents to improve decisions based on delayed rewards and environmental feedback \citep{singh-etal:singhagentic}. As a result, RL complements the inherent reasoning abilities of LLMs by strengthening their decision-making and execution capabilities in dynamic environments. In the following sections, we discuss two representative capabilities that have become the primary focus of agentic RL: planning and tool use.
However, autonomous agents require capabilities that go beyond static reasoning. They must make sequential decisions over long horizons, coordinate multiple tool using, adapt to unexpected environmental feedback, and recover from execution failures. These capabilities cannot be effectively learned from static supervision alone because their quality depends on the outcomes of complete interaction trajectories rather than individual outputs. RL naturally addresses this challenge by optimizing behaviors through trial-and-error interaction, enabling agents to improve decisions based on delayed rewards and environmental feedback \citep{singh-etal:singhagentic}. As a result, RL complements the inherent reasoning abilities of LLMs by strengthening their decision-making and execution capabilities in dynamic environments. In the following sections, we discuss two representative capabilities that have become the primary focus of agentic RL: planning and tool using.
\subsubsection{Planning}
\subsubsection{Planning}
...
@@ -46,7 +46,7 @@ $\cdots \cdots$
...
@@ -46,7 +46,7 @@ $\cdots \cdots$
\end{tcolorbox}
\end{tcolorbox}
Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool use, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In practice, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization.
Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool using, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In practice, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization.
\subsubsubsection{Supervised Fine-Tuning}
\subsubsubsection{Supervised Fine-Tuning}
...
@@ -110,7 +110,7 @@ Applying RL to agent planning introduces a key challenge: credit assignment. In
...
@@ -110,7 +110,7 @@ Applying RL to agent planning introduces a key challenge: credit assignment. In
\label{fig:agent-credit-assignment}
\label{fig:agent-credit-assignment}
\end{figure*}
\end{figure*}
Consider the example above. If the agent correctly plans to compute the babysitting payment but calls the database API before obtaining the calculation result, the final task may fail. In this case, the failure should mainly be attributed to the incorrect ordering of tool use rather than to the entire plan. Conversely, if the agent chooses the wrong formula at the beginning, then the later database operation may be executed correctly, but the final result is still wrong. These cases show that different steps in an interaction trajectory may contribute differently to the final outcome.
Consider the example above. If the agent correctly plans to compute the babysitting payment but calls the database API before obtaining the calculation result, the final task may fail. In this case, the failure should mainly be attributed to the incorrect ordering of tool using rather than to the entire plan. Conversely, if the agent chooses the wrong formula at the beginning, then the later database operation may be executed correctly, but the final result is still wrong. These cases show that different steps in an interaction trajectory may contribute differently to the final outcome.
Given a trajectory $\tau=[p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective uses a trajectory-level reward $R(\tau)$ to optimize the whole planning process. However, for long-horizon agent interactions, it is often more useful to decompose this feedback into step-level signals:
Given a trajectory $\tau=[p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective uses a trajectory-level reward $R(\tau)$ to optimize the whole planning process. However, for long-horizon agent interactions, it is often more useful to decompose this feedback into step-level signals:
\begin{eqnarray}
\begin{eqnarray}
...
@@ -134,11 +134,21 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
...
@@ -134,11 +134,21 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
\subsubsection{Tool Use}
\subsubsection{Tool Using}
Tool using is another fundamental capability of LLM-based agents. Tools extend the agent's capability beyond its internal parameters. Prompt-based methods first show that LLMs can use tools through reasoning-action prompting. ReAct is a typical example \citep{yao-etal:react}. However, prompt-based tool use is unstable and depends on the base model. Thus, SFT methods train models on tool-use trajectories \citep{schick-etal:toolformer}. Still, SFT mainly teaches imitation. It cannot directly optimize whether the agent should use a tool, which tool to use, and how to balance tool benefit with tool cost. RL provides a natural framework for efficient tool using. We can define rewards based on task success, tool-use correctness, tool-use completeness, and tool cost \citep{qian-etal:toolrl}.
% 基于提示的方法来去使用Tool 比如react,
% 2. Prompt-based Tool Use
\subsubsubsection{Prompt-based Tool Using}
% 然后就是训练，让模型参数记住如何使用tool 比如toolformer
% 3. SFT for Tool Use
\subsubsubsection{Supervised Fine-Tuning}
% 4. RL for Tool Use
\subsubsubsection{Reinforcement Learning}
% 5. Efficient Tool using
\subsubsubsection{Efficient Tool Using}
...
@@ -146,6 +156,7 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
...
@@ -146,6 +156,7 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
