@@ -7,7 +7,7 @@ Using RL to train agents, often referred to as \textit{agentic RL}, is not a new
With the rise of LLMs as core components of autonomous systems, RL has been increasingly used to adapt these models for sequential decision-making in dynamic and open-ended environments. In this setting, agentic RL addresses a key limitation of LLMs: although they encode extensive world knowledge, their outputs are not inherently aligned with long-horizon task requirements. For example, a supervised fine-tuned LLM may correctly generate a script for calling an external API, but it cannot reliably self-correct when the environment returns unexpected errors. This creates new challenges for enabling LLM-based agents to plan autonomously, adapt to environmental feedback, and safely improve their strategies through iterative interaction.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory updates, skill optimization, and trajectory refinement without altering the internal model weights. It is worth noting that while these approaches depart from traditional parametric updates, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory updates, skill optimization, and trajectory refinement without altering the internal model weights. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
\subsection{Building Agent Capabilities}
...
...
@@ -304,31 +304,39 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.
The first approach is memory management though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new but related task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The first approach is \textbf{memory management} though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The second approach is skill optimization. Individual trajectories often contain reusable procedures, such as how to search for information, recover from an invalid tool call, or complete a common sequence of operations. The agent can abstract these procedures into higher-level skills and reuse them in future tasks. Skill optimization therefore converts low-level interaction experience into more general and efficient behaviors.
The second approach is \textbf{skill optimization}. One challenge of using memory to learn from experience is scalability. As agent experience continuously accumulates, the memory size will keep growing. Moreover, a large amount of noisy and redundant information may also be introduced into the memory. As a result, As a result, efficiently storing and retrieving relevant information becomes increasingly difficult. Instead of storing individual experiences in memory, skill aims to identify common patterns across experiences and transform them into abstract behaviors. These skills provide a more compact representation of experience and allow the agent to transfer experience across different tasks. Similar to memory optimization, we can also update and refine the skills of agents to achieve learning from experience.
The third approach is trajectory refinement. Not all collected trajectories are optimal, even when they eventually succeed. Some may contain incorrect actions, redundant steps, or inefficient tool-use patterns. The agent can revise these trajectories by correcting failures, removing unnecessary actions, and constructing better reasoning or execution paths. Both successful and failed trajectories can provide useful signals for this refinement process.
The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that while these approaches depart from traditional parametric updates, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making.
\subsubsection{Memory Management}
However, this approach introduces a key challenge in memory management: deciding when and what information should be updated in memory.
memory-R1
\subsubsection{Skill Optimization}
Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
Reinforcement Learning for Self-Improving Agent with Skill Library
\subsubsection{Trajectory Refinement}
Learning from early experience.
\subsubsection{Trajectory Refinement}
Using the feedback to refine the trajectories.
Feedback is very important. How to obtain feedback from convention?
