@@ -304,7 +304,7 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
...
@@ -304,7 +304,7 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.
The first approach is memory management though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new but related task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}. In this way, a straightforward approach to learn from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences, reuse effective strategies, and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The first approach is memory management though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new but related task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The second approach is skill optimization. Individual trajectories often contain reusable procedures, such as how to search for information, recover from an invalid tool call, or complete a common sequence of operations. The agent can abstract these procedures into higher-level skills and reuse them in future tasks. Skill optimization therefore converts low-level interaction experience into more general and efficient behaviors.
The second approach is skill optimization. Individual trajectories often contain reusable procedures, such as how to search for information, recover from an invalid tool call, or complete a common sequence of operations. The agent can abstract these procedures into higher-level skills and reuse them in future tasks. Skill optimization therefore converts low-level interaction experience into more general and efficient behaviors.
