title={Reinforcement learning for self-improving agent with skill library},
author={Wang, Jiongxiao and Yan, Qiaojing and Wang, Yawei and Tian, Yijun and Mishra, Soumya Smruti and Xu, Zhichao and Gandhi, Megha and Xu, Panpan and Cheong, Lin Lee},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={1529--1550},
year={2026}
}
@article{xia-etal:skillrl,
title={Skillrl: Evolving agents via recursive skill-augmented reinforcement learning},
author={Xia, Peng and Chen, Jianwen and Wang, Hanyang and Liu, Jiaqi and Zeng, Kaide and Wang, Yu and Han, Siwei and Zhou, Yiyang and Zhao, Xujiang and Chen, Haifeng and others},
@@ -436,8 +436,34 @@ One limitation of manual skill construction is that the quality and diversity la
The above way of generating skills often suffer from quality issues. First, a general LLM does not inherently know how to design effective skills without additional optimization. Second, the skill generator may optimize for the wrong objective because skill generation and skill utilization involve different contexts. Specifically, an LLM may consider a generated skill effective because it satisfies the instructions provided in the prompt, while another agent may fail to use this skill effectively in real-world tasks. As a result, skill generation should not only focus on producing well-structured skills, but also consider whether these skills can actually improve agent performance during task execution. Recent work resorts to RL to achieve this goal, where the skill generation is optimized based on the actual performance of generated skills.
Here we consider \textbf{SkillRL} as an example to illustrate how to optimize the skill generation though RL \citep{xia-etal:skillrl}.
Illustration of SkillRL \citep{xia-etal:skillrl}. SkillRL maintains a pool of agentic trajectories and initially abstracts reusable skills from these trajectories. The extracted skills are then used to guide RL training. As the agent improves through RL training, it generates higher-quality trajectories, which are further used to refine the skill bank.
}
\label{fig:skillrl}
\end{figure*}
Here we consider \textbf{SkillRL} as an example to illustrate how to optimize the skill generation though RL \citep{xia-etal:skillrl}. The idea is that instead of directly generating skills from stored trajectories, we can first abstract reusable skills from existing trajectories and uses them to guide agentic RL training. During the training process, the agent continuously discovers new skills and updates the skill bank, enabling the joint evolution of the agent and its skills. Figure~\ref{fig:skillrl} shows a schematic illustration of SkillRL. Here we give a brief outline of the key steps involved.
\begin{itemize}
\item Initially, we collect interaction trajectories from environments. These trajectories can include both successful and failed experiences, which provide diverse signals for skill discovery and optimization. Successful trajectories reveal effective behaviors, while failed trajectories help identify potential weaknesses and improvement opportunities.
\item The collected trajectories are then used to extract reusable skills, which initialize the skill bank. Considering that agent capabilities usually include both general problem-solving strategies and task-specific procedures, we can organize skills at different levels of abstraction. Specifically, a hierarchical skill library can be constructed, where a general skill bank stores transferable skills learned across different tasks to improve generalization, while a task-specific skill bank preserves specialized strategies for solving particular tasks.
\item After initializing the skill bank, the agent retrieves relevant skills to guide the RL training process. Specifically, given task $x$, the agent selects suitable skills from the skill bank $\mathcal{S}$:
where $\mathrm{Score}(x,s_i)$ measures the relevance between the current task and each skill, and $\mathcal{S}^{*}$ denotes the selected skill set. The selected skills are then incorporated into the agent's decision-making process and used to guide RL optimization. These skills provide high-level behavioral priors, such as task decomposition and execution workflows, which help the agent explore more effective behaviors and improve the efficiency of RL training.
\item Through interaction with the environment, the agent collects new trajectories. These trajectories are further analyzed to discover new skills or refine existing ones. In this way, the skill bank evolves together with the agent policy: the skill library provides prior knowledge to accelerate RL training, while improved trajectories continuously enrich and refine the skill library.
\end{itemize}
By repeating the above process, the agent can improve its skill library through accumulated experiences. Specifically, newly collected trajectories provide additional evidence for discovering new skills or refining existing ones, while the updated skill library further guides future RL training. This creates an iterative learning process in which the agent and its skills co-evolve over time. From this perspective, skill optimization provides a practical way to transform low-level interaction experiences into reusable capabilities, and has become an important approach for enabling agents to continuously learn from their own experiences \citep{wang-etal:reinforcement}.
