@@ -302,9 +302,15 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
...
@@ -302,9 +302,15 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
\subsection{Learning from Agentic Experience}
\subsection{Learning from Agentic Experience}
\label{sec:learning_from_agentic_experience}
\label{sec:learning_from_agentic_experience}
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex environments. The space of possible tasks, observations, tool responses, and execution failures is too large to be exhaustively annotated in advance.
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex environments.
% In the era of experience, we expect an agent to improve through repeated interaction with its environment. During this process, the agent performs actions, observes their consequences, and collects both successful and failed trajectories. RL provides a direct way to optimize the agent from such feedback. 本质上，在之前章节中我们所提及的planning and tool use 能力增强 with rl 也是一种agentic 经验学习。 However, learning from experience is not limited to improve update this policy parameters
The first approach is memory management. An agent can store useful information from previous interactions, such as successful solutions and failure patterns. When facing a new but related task, the agent can retrieve relevant memories and use them to guide its decisions. In this way, the agent does not need to solve every problem from scratch and can gradually accumulate knowledge across interactions.
The second approach is skill optimization. Individual trajectories often contain reusable procedures, such as how to search for information, recover from an invalid tool call, or complete a common sequence of operations. The agent can abstract these procedures into higher-level skills and reuse them in future tasks. Skill optimization therefore converts low-level interaction experience into more general and efficient behaviors.
The third approach is trajectory refinement. Not all collected trajectories are optimal, even when they eventually succeed. Some may contain incorrect actions, redundant steps, or inefficient tool-use patterns. The agent can revise these trajectories by correcting failures, removing unnecessary actions, and constructing better reasoning or execution paths. Both successful and failed trajectories can provide useful signals for this refinement process.
