\subsection{Fundamentals of Large Language Models}
\subsection{Fundamentals of Large Language Models}
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and nations necessary for the discussions in future sections.
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and nations necessary for the discussions in future sections.
\caption{Overview of data construction for improving agent planning.}
\label{fig:agent-planning-sft}
\end{figure*}
One commonly used approach to improve the planning capability of LLM-based agents is SFT \citep{chen-etal:fireact,zhang-etal:agentohana}. As illustrated in Figure \ref{fig:agent-planning-sft}, this process typically consists of three stages: 1) preparing environments and planning tasks; 2) synthesizing expert-level trajectories, which consist of sequences of action-observation pairs, on these tasks; and 3) instruction-tuning LLMs using the synthesized trajectory data. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as agent policies and selecting high-quality trajectories based on predefined reward signals or task success criteria. Through this process, LLMs can acquire planning behaviors from demonstrations and improve their ability to generate structured execution plans. Based on this procedure, we can construct SFT data for enhancing the planning capability of LLM-based agents. Specifically, each training instance consists of an input-output pair:
\begin{eqnarray}
\mathbf{x}&=&\{e,q\}\\
\mathbf{y}&=&\{p,u_1,o_1,\cdots,u_T,o_T\}
\end{eqnarray}
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment.
One commonly used approach to improve the planning capability of LLM-based agents is instruction tuning \citep{chen-etal:fireact,zhang-etal:agentohana}. As illustrated in Figure \ref{fig:agent-planning-sft}, this process typically consists of three stages: (i) preparing environments and planning tasks; (ii) synthesizing expert-level trajectories, which consist of sequences of action-observation pairs, on these tasks; and (iii) instruction-tuning LLMs using the synthesized trajectory data. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as agent policies and selecting high-quality trajectories based on predefined reward signals or task success criteria. Through this process, LLMs can acquire planning behaviors from demonstrations and improve their ability to generate structured execution plans.
It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
