@@ -83,14 +83,14 @@ Although SFT can provide a useful cold start for agent planning, it mainly teach
Under the agent planning formulation, the LLM-based agent can be viewed as a high-level policy that generates a plan and then produces a sequence of executable behaviors \citep{li-etal:encouraging}. Given the environment information $e$ and task description $q$, the agent first generates a plan $p$ and then interacts with the environment through a sequence of behaviors and observations:
\begin{eqnarray}
\tau = [p,u_1,o_1,\cdots,u_T,o_T].
\tau = [p,u_1,o_1,\cdots,u_T,o_T]
\end{eqnarray}
The goal of RL is to optimize the agent policy so that the generated trajectory receives higher feedback from the environment. This objective can be written as
where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. This reward can be defined according to task success, progress toward the goal, tool-use correctness, or other environment-specific criteria.
where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. In practice, this reward can be defined according to task success. The objective can then be optimized using standard RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}, such as PPO or GRPO.
Compared with SFT, RL provides two important advantages for agent planning. First, it enables the agent to learn from execution outcomes rather than only from static demonstrations. If a plan leads to failed tool execution or poor environmental feedback, the agent can be penalized and encouraged to explore alternative behaviors. Second, RL supports trajectory-level credit assignment, allowing the model to adjust earlier planning decisions according to later outcomes. This is crucial for agent planning, where early subgoal decomposition or tool selection can substantially affect the final task success.
...
...
@@ -120,7 +120,7 @@ where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the
The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
For illustration, suppose we define rule-based step-level rewards according to predefined verification criteria, such as whether the plan covers the necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. Credit assignment can then decompose the final feedback into step-level rewards:
For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
\begin{eqnarray}
R(\tau)=0.3
\quad\Rightarrow\quad
...
...
@@ -136,6 +136,12 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
