\caption{Illustration of credit assignment in agentic RL.}
\caption{Illustration of credit assignment in agentic RL.}
\label{fig:agent-credit-assignment}
\label{fig:agent-credit-assignment}
\end{figure*}
\end{figure*}
...
@@ -116,7 +116,17 @@ Given a trajectory $\tau = [p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective us
...
@@ -116,7 +116,17 @@ Given a trajectory $\tau = [p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective us
\begin{eqnarray}
\begin{eqnarray}
R(\tau) \rightarrow\{r_p,r_1,\cdots,r_T\}
R(\tau) \rightarrow\{r_p,r_1,\cdots,r_T\}
\end{eqnarray}
\end{eqnarray}
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome. The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm}.
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
For illustration, suppose we define rule-based step-level rewards according to predefined verification criteria, such as whether the plan covers the necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. Credit assignment can then decompose the final feedback into step-level rewards:
where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indicate that the agent correctly extracts the numerical values and performs the calculation, $r_3$ indicates that the JSON formatting is partially correct, and $r_4$ indicates that the database update fails. In this way, the agent receives more precise feedback: it should preserve the correct planning.
