\bibitem[Song et~al.(2024)Song, Yin, Yue, Huang, Li, and Lin]{song-etal:trial}
Yifan Song, Da~Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill~Yuchen Lin.
\newblock Trial and error: Exploration-based trajectory optimization of llm agents.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 7584--7600, 2024.
\bibitem[Sullivan et~al.(2025)Sullivan, Hartmann, and Koller]{sullivan-etal:randomworld}
\bibitem[Sullivan et~al.(2025)Sullivan, Hartmann, and Koller]{sullivan-etal:randomworld}
Michael Sullivan, Mareike Hartmann, and Alexander Koller.
Michael Sullivan, Mareike Hartmann, and Alexander Koller.
\newblock Procedural environment generation for tool-use agents.
\newblock Procedural environment generation for tool-use agents.
...
@@ -581,6 +596,11 @@ Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou.
...
@@ -581,6 +596,11 @@ Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou.
\newblock Reinforcement learning for self-improving agent with skill library.
\newblock Reinforcement learning for self-improving agent with skill library.
...
@@ -609,10 +629,10 @@ Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi C
...
@@ -609,10 +629,10 @@ Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi C
\newblock In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), \emph{Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023}, 2023{\natexlab{d}}.
\newblock In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), \emph{Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023}, 2023{\natexlab{d}}.
title={Trial and error: Exploration-based trajectory optimization of LLM agents},
author={Song, Yifan and Yin, Da and Yue, Xiang and Huang, Jie and Li, Sujian and Lin, Bill Yuchen},
booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={7584--7600},
year={2024}
}
@inproceedings{fu-etal:agentrefine,
title={Agentrefine: Enhancing agent generalization through refinement tuning},
author={Fu, Dayuan and He, Keqing and Wang, Yejie and Hong, Wentao and Gongque, Zhuoma and Zeng, Weihao and Wang, Wei and Wang, Jingang and Cai, Xunliang and Xu, Weiran},
booktitle={International Conference on Learning Representations},
volume={2025},
pages={65185--65204},
year={2025}
}
@article{yuan-etal:agent,
title={Agent-r: Training language model agents to reflect via iterative self-training},
author={Yuan, Siyu and Chen, Zehui and Xi, Zhiheng and Ye, Junjie and Du, Zhengyin and Chen, Jiecao},
journal={arXiv preprint arXiv:2501.11425},
year={2025}
}
@inproceedings{wang-etal:steca,
title={Steca: Step-level trajectory calibration for llm agent learning},
author={Wang, Hanlin and Wang, Jian and Leong, Chak Tou and Li, Wenjie},
booktitle={Findings of the Association for Computational Linguistics: ACL 2025},
pages={11597--11614},
year={2025}
}
@inproceedings{liu-etal:pract,
title={Pract: Optimizing principled reasoning and acting of llm agent},
author={Liu, Zhiwei and Yao, Weiran and Zhang, Jianguo and Liu, Zuxin and Yang, Liangwei and RN, Rithesh and Lan, Tian and Zhu, Ming and Tan, Juntao and Kokane, Shirley and others},
booktitle={Proceedings of the 28th Conference on Computational Natural Language Learning},
pages={442--446},
year={2024}
}
@article{shinn-etal:reflexion,
@article{shinn-etal:reflexion,
title={Reflexion: Language agents with verbal reinforcement learning},
title={Reflexion: Language agents with verbal reinforcement learning},
@@ -317,6 +317,7 @@ The three approaches mentioned above can be implemented through various techniqu
...
@@ -317,6 +317,7 @@ The three approaches mentioned above can be implemented through various techniqu
\subsubsection{Memory Management}
\subsubsection{Memory Management}
\label{sec:memory-management}
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
\begin{itemize}
\begin{itemize}
...
@@ -369,6 +370,7 @@ Beyond optimizing memory update operations, another important question is how to
...
@@ -369,6 +370,7 @@ Beyond optimizing memory update operations, another important question is how to
\subsubsection{Skill Optimization}
\subsubsection{Skill Optimization}
\label{sec:skill-optimization}
Skill represents a higher-level abstraction of agentic experience. Different from memory, which focuses on storing useful information extracted from previous interactions, skill aims to summarize recurring patterns across multiple experiences and transform them into reusable capabilities. Specifically, a skill can be viewed as a structured instruction package that describes when the skill should be invoked, what workflow it should follow, which tools are required, and how the final outcome should be verified. For example, a reasoning-oriented skill may be represented as follows:
Skill represents a higher-level abstraction of agentic experience. Different from memory, which focuses on storing useful information extracted from previous interactions, skill aims to summarize recurring patterns across multiple experiences and transform them into reusable capabilities. Specifically, a skill can be viewed as a structured instruction package that describes when the skill should be invoked, what workflow it should follow, which tools are required, and how the final outcome should be verified. For example, a reasoning-oriented skill may be represented as follows:
...
@@ -544,28 +546,10 @@ Verify the solution:
...
@@ -544,28 +546,10 @@ Verify the solution:
$x=5$.
$x=5$.
\end{tcolorbox}
\end{tcolorbox}
It is worth noting that we can obtain feedback from either internal evaluation or external evaluators. For example, we can use rule-based verifiers, LLM-based evaluators, or binary reward signals to provide feedback to the agent. Regardless of the feedback source or format, we need to convert it into a language-based representation, enabling the agent to understand errors and refine future behaviors. Here, inspired by the step-level feedback discussed in Section~\ref{sec:step-by-step-verification}, we can further consider whether providing fine-grained feedback at each decision step can help agents generate better-refined trajectories. Compared with outcome-level feedback, step-level feedback provides more precise guidance about where and why the agent makes mistakes \citep{liu-etal:pract,wang-etal:steca}. For example, when an agent fails to solve a mathematical reasoning problem, a binary reward only indicates that the final answer is incorrect, but it does not reveal which reasoning step contains the error or how the agent should revise its solution. However, obtaining step-level feedback is often expensive and challenging, as it requires evaluating intermediate decisions throughout the trajectory. To address this challenge, recent studies explore using methods such as MCTS to automatically construct step-level feedback signals \citep{yuan-etal:agent}.
Training-free: \\
The above approaches mainly rely on prompting to enable trajectory refinement. Such approaches still face several limitations. First, their performance is bounded by the intrinsic refinement ability of the LLM. A pre-trained LLM may not naturally know how to interpret feedback and effectively revise its trajectory without additional optimization. Second, prompt-based refinement mainly improves the current trajectory at inference time, which limits its ability to accumulate and transfer experience across tasks. As discussed in Sections~\ref{sec:memory-management} and~\ref{sec:skill-optimization}, failed trajectories contain valuable experiences about agent weaknesses and can provide useful learning signals for improving future behaviors. However, prompting-based approaches only leverage these experiences within the current interaction context. To address these limitations, recent studies explore training agents to acquire trajectory refinement abilities \citep{fu-etal:agentrefine}, either by updating model parameters or optimizing refinement behaviors based on collected experiences \citep{song-etal:trial}. Interested readers can refer to these works for more information.
Reflexion: Language Agents with Verbal Reinforcement Learning \\
PRACT: Optimizing Principled Reasoning and Acting of LLM Agent \\
Training-based: \\
Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (DPO training) \\
Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training (Self-Training) \\
AgentRefine: Enhancing Agent Generalization through Refinement Tuning (use refinement tuning to improve generalization) \\
Feedback is very important. How to obtain feedback from convention?
OpenClaw-RL: Train Any Agent Simply by Talking
Although trajectory refinement does not explicitly update model parameters like traditional RL, it shares the fundamental principle of RL: improving agent behaviors based on feedback collected from interactions. From the perspective of in-context learning, the feedback obtained during trajectory refinement can be viewed as a temporary reward signal that modifies the agent's subsequent decision-making process. Instead of updating policy parameters, the agent updates its context with reflections, which potentially modifies the policy used for future actions within the current interaction. Therefore, trajectory refinement can be regarded as an in-context form of policy improvement.
