\textcolor{gray}{Input}& Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes. How much did she earn? Please save the result into the database. \\
\textcolor{gray}{Input}&\textit{Environment}: Calculator, Database API \newline
\textit{Task}: Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes. How much did she earn? Please save the result into the database. \\
\textcolor{gray}{Output}& 1. Extract the relevant numerical values and identify the required computation. \newline
2. Call the \texttt{calculator} tool to evaluate the expression $12\times(50/60)$. \newline
3. Format the computed result into a valid JSON schema. \newline
...
...
@@ -45,25 +46,24 @@ $\cdots \cdots$
\end{tcolorbox}
Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool use, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In this approach, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization. We first discuss how SFT constructs trajectory demonstrations for agent planning, and then introduce how RL improves planning by optimizing agent behaviors with environmental feedback.
\subsubsubsection{Supervised Fine-Tuning}
\begin{figure*}[!t]
\centering
\resizebox{\linewidth}{!}{
\input{section6/Figures/agent-planning-sft}}
\caption{Overview of data construction for improving agent planning.}
\caption{Overview of SFT data construction for improving agent planning.}
\label{fig:agent-planning-sft}
\end{figure*}
% 一段话来去引出强化学习对planning能力的关注，然后使用引出具体下面的内容
\subsubsubsection{Supervised Fine-Tuning}
Before applying RL, SFT is commonly adopted to provide a cold start for LLM-based agents by teaching them basic planning and interaction behaviors \citep{chen-etal:fireact,zhang-etal:agentohana,zeng-etal:agenttuning}. As illustrated in Figure \ref{fig:agent-planning-sft}, constructing SFT data for agent planning typically involves three stages: (1) preparing diverse environments and planning tasks; (2) synthesizing expert-level trajectories, consisting of action-observation sequences, through agent-environment interactions; and (3) selecting high-quality trajectories based on predefined evaluation criteria. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as expert agents and retaining reliable demonstrations according to reward signals or task success criteria. Through this process, LLMs can learn planning behaviors from demonstrations and improve their ability to generate structured execution plans.
\begin{eqnarray}
\mathbf{x}&=& [e,q] \\
\mathbf{y}&=& [p,u_1,o_1,\cdots,u_T,o_T]
\end{eqnarray}
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
