@@ -97,8 +97,9 @@ Compared with SFT, RL provides two important advantages for agent planning. Firs
% Another challenge is training stability. In multi-turn agent RL, long-horizon decision-making and stochastic environment feedback can make trajectory-level optimization unstable, while errors in early steps may accumulate and affect later interactions \citep{wang-etal:ragen,djuhera-etal:tsr}.
\subsubsubsection{Credit Assignment}
Credit assignment is not unique to LLM-based agents. It is a long-standing challenge in traditional RL. When rewards are delayed, the agent must decide which past actions should be reinforced or suppressed. Classical methods, such as temporal-difference learning and eligibility traces, address this issue by propagating reward information backward along the trajectory. In agentic RL, credit assignment becomes more challenging. This is because that each interaction step may involve high-level planning, tool invocation, and environmental feedback, rather than a single low-level action. Therefore, credit assignment often needs to operate at both the trajectory level and the step level.
Applying RL to agent planning introduces a key challenge: credit assignment. In many agent tasks, the environment only provides a sparse reward after the entire trajectory is completed. However, the final success or failure may be caused by an earlier planning decision, an incorrect tool invocation, or an ineffective response to an intermediate observation. Therefore, directly assigning the same trajectory-level reward to all interaction steps can provide noisy supervision and make it difficult for the agent to identify which decisions should be reinforced or suppressed.
This challenge is especially important for agent planning. In many agent tasks, the environment only provides a sparse reward after the entire trajectory is completed. However, the final success or failure may come from an earlier planning decision, an incorrect tool invocation, or an ineffective response to an intermediate observation. If we directly assign the same trajectory-level reward to all interaction steps, the supervision can become noisy. The agent may fail to identify which decisions should be reinforced and which should be suppressed.
\begin{figure*}[!t]
\centering
...
...
@@ -110,13 +111,13 @@ Applying RL to agent planning introduces a key challenge: credit assignment. In
\label{fig:agent-credit-assignment}
\end{figure*}
Consider the example above. If the agent correctly plans to compute the babysitting payment but calls the database API before obtaining the calculation result, the final task may fail. In this case, the failure should mainly be attributed to the incorrect ordering of tool use rather than to the entire plan. Conversely, if the agent chooses the wrong formula at the beginning, then the later database operation may be executed correctly, but the final result is still wrong. These cases show that different steps in an interaction trajectory may contribute differently to the final outcome.
% Consider a simple example. If the agent correctly plans to compute the babysitting payment but calls the database API before obtaining the calculation result, the final task may fail. In this case, the failure should mainly be attributed to the incorrect ordering of tool use rather than to the entire plan. Conversely, if the agent chooses the wrong formula at the beginning, then the later database operation may be executed correctly, but the final result is still wrong. These cases show that different steps in an interaction trajectory may contribute differently to the final outcome.
Given a trajectory $\tau=[p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective uses a trajectory-level reward $R(\tau)$ to optimize the whole planning process. However, for long-horizon agent interactions, it is often more useful to decompose this feedback into step-level signals:
Formally, given a trajectory $\tau=[p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective uses a trajectory-level reward $R(\tau)$ to optimize the whole planning process. However, such a reward only provides coarse feedback on the overall outcome. Therefore, credit assignment aims to decompose the trajectory-level reward into step-level signals:
\begin{eqnarray}
R(\tau) \rightarrow\{r_p,r_1,\cdots,r_T\}
\end{eqnarray}
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
...
...
@@ -188,10 +189,10 @@ This demonstration teaches the model a tool-use pattern. The model learns to rea
% 3. SFT for Tool Use
\subsubsubsection{Supervised Fine-Tuning}
To make tool use more reliable, we can further train LLMs on tool-use trajectories with supervised fine-tuning. In this setting, each training example contains a user task, available tool descriptions, and a demonstrated tool-use process. The model learns when to invoke a tool, which tool to select, how to construct valid arguments, and how to use the returned observation. For example, we can construct a training example as follows:
% \begin{center}
% $\mathbf{x} =$ Task: ``Weng earns \$12 per hour and worked for 50 minutes. Save the result into the database.''; Tools: calculator(expression), update\_record(json)
$\mathbf{x}=$ Task: ``Weng earns \$12 per hour and worked for 50 minutes. Save the result into the database.''; Tools: calculator(expression), update\_record(json) \newline
