\bibitem[Li et~al.(2025{\natexlab{b}})Li, Hu, and Wang]{li-etal:encouraging}
Zhiwei Li, Yong Hu, and Wenqing Wang.
\newblock Encouraging good processes without the need for good answers: Reinforcement learning for llm agent planning.
\newblock In \emph{Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track}, pp.\ 1654--1666, 2025{\natexlab{b}}.
\bibitem[Li et~al.(2024)Li, Xu, Zhang, Lin, Yu, Sun, and Luo]{li-etal:2023remax}
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi{-}Quan Luo.
\newblock Remax: {A} simple, effective, and efficient reinforcement learning method for aligning large language models.
...
...
@@ -483,6 +488,11 @@ Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami.
\newblock Reuse your rewards: Reward model transfer for zero-shot cross-lingual alignment.
title={TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents},
author={Djuhera, Aladin and Kadhe, Swanand Ravindra and Ahmed, Farhan and Zawad, Syed and Ludwig, Heiko and Boche, Holger},
journal={arXiv preprint arXiv:2602.11767},
year={2026}
}
@article{wang-etal:ragen,
title={Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning},
author={Wang, Zihan and Wang, Kangrui and Wang, Qineng and Zhang, Pingyue and Li, Linjie and Yang, Zhengyuan and Jin, Xing and Yu, Kefan and Nguyen, Minh Nhat and Liu, Licheng and others},
journal={arXiv preprint arXiv:2504.20073},
year={2025}
}
@inproceedings{xi-etal:agentprm,
title={Agentprm: Process reward models for llm agents via step-wise promise and progress},
author={Xi, Zhiheng and Liao, Chenyang and Li, Guanyu and Zhang, Zhihao and Chen, Wenxiang and Wang, Binghai and Jin, Senjie and Zhou, Yuhao and Guan, Jian and Wu, Wei and others},
booktitle={Proceedings of the ACM Web Conference 2026},
pages={4184--4195},
year={2026}
}
@inproceedings{li-etal:encouraging,
title={Encouraging good processes without the need for good answers: Reinforcement learning for llm agent planning},
author={Li, Zhiwei and Hu, Yong and Wang, Wenqing},
booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track},
pages={1654--1666},
year={2025}
}
@inproceedings{hu-etal:agentgen,
title={Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation},
Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool use, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In this approach, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization. We first discuss how SFT constructs trajectory demonstrations for agent planning, and then introduce how RL improves planning by optimizing agent behaviors with environmental feedback.
Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool use, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In practice, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization.
\subsubsubsection{Supervised Fine-Tuning}
...
...
@@ -78,14 +78,50 @@ Furthermore, even with effective trajectory selection and representation, scalin
\subsubsubsection{Reinforcement Learning}
Although SFT can provide a useful cold start for agent planning, it mainly teaches the model to imitate offline demonstrations. This limits its ability to improve beyond the quality and coverage of the collected trajectories. In contrast, RL further optimizes planning through direct interaction with the environment, where the agent receives feedback based on the outcome of its generated plans and executed behaviors.
% This makes RL particularly suitable for improving long-horizon planning, because the quality of a plan is often determined by the success of the entire interaction trajectory rather than by the correctness of any single intermediate step.
\subsubsubsection{}
Under the agent planning formulation, the LLM-based agent can be viewed as a high-level policy that generates a plan and then produces a sequence of executable behaviors \citep{li-etal:encouraging}. Given the environment information $e$ and task description $q$, the agent first generates a plan $p$ and then interacts with the environment through a sequence of behaviors and observations:
\begin{eqnarray}
\tau = [p,u_1,o_1,\cdots,u_T,o_T].
\end{eqnarray}
The goal of RL is to optimize the agent policy so that the generated trajectory receives higher feedback from the environment. This objective can be written as
where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. This reward can be defined according to task success, progress toward the goal, tool-use correctness, or other environment-specific criteria.
Compared with SFT, RL provides two important advantages for agent planning. First, it enables the agent to learn from execution outcomes rather than only from static demonstrations. If a plan leads to failed tool execution or poor environmental feedback, the agent can be penalized and encouraged to explore alternative behaviors. Second, RL supports trajectory-level credit assignment, allowing the model to adjust earlier planning decisions according to later outcomes. This is crucial for agent planning, where early subgoal decomposition or tool selection can substantially affect the final task success.
% Another challenge is training stability. In multi-turn agent RL, long-horizon decision-making and stochastic environment feedback can make trajectory-level optimization unstable, while errors in early steps may accumulate and affect later interactions \citep{wang-etal:ragen,djuhera-etal:tsr}.
\subsubsubsection{Credit Assignment}
Applying RL to agent planning introduces a key challenge: credit assignment. In many agent tasks, the environment only provides a sparse reward after the entire trajectory is completed. However, the final success or failure may be caused by an earlier planning decision, an incorrect tool invocation, or an ineffective response to an intermediate observation. Therefore, directly assigning the same trajectory-level reward to all interaction steps can provide noisy supervision and make it difficult for the agent to identify which decisions should be reinforced or suppressed.
\caption{Illustration of credit assignment in agentic RL.}
\label{fig:agent-credit-assignment}
\end{figure*}
Consider the example above. If the agent correctly plans to compute the babysitting payment but calls the database API before obtaining the calculation result, the final task may fail. In this case, the failure should mainly be attributed to the incorrect ordering of tool use rather than to the entire plan. Conversely, if the agent chooses the wrong formula at the beginning, then the later database operation may be executed correctly, but the final result is still wrong. These cases show that different steps in an interaction trajectory may contribute differently to the final outcome.
Given a trajectory $\tau=[p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective uses a trajectory-level reward $R(\tau)$ to optimize the whole planning process. However, for long-horizon agent interactions, it is often more useful to decompose this feedback into step-level signals:
\begin{eqnarray}
R(\tau) \rightarrow\{r_p,r_1,\cdots,r_T\}
\end{eqnarray}
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome. The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm}.
\subsubsubsection{}
% To address this challenge, researchers explore more informative feedback signals for agent planning. For example, tool-use rewards evaluate whether the tool invocation sequence is complete and effective, while process rewards or progress-based rewards assess whether each intermediate decision moves the agent closer to the final goal \citep{li-etal:encouraging,xi-etal:agentprm}. These methods provide denser supervision than sparse final rewards and make it possible to optimize intermediate planning and execution behaviors more directly.
