update.

2fc029d1 · wangchenglong · 1e50963d · 2fc029d1
Commit 2fc029d1 authored Sep 02, 2026 by wangchenglong
--- a/section6/section6.tex
+++ b/section6/section6.tex
@@ -3,11 +3,11 @@
 \section{Agentic Reinforcement Learning}
-Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
+Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Atari and Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
 With the emergence of LLMs as core components of autonomous systems, the role of RL has shifted from learning policies from scratch to adapting pretrained models for sequential decision-making in dynamic and open-ended environments. In this context, agentic RL aims to address a key limitation of LLMs: although they contain extensive world knowledge, their outputs are not inherently optimized for long-horizon interactions. For example, an SFT-trained LLM may successfully generate code for calling an external API, but it may fail to recover from unexpected execution errors during interaction.
-In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We first discuss how RL enables agents to acquire fundamental capabilities, including long-horizon planning and tool-integrated reasoning. We then introduce the role of environment design and scaling, which provide the essential foundation for training and evaluating agentic RL systems. Finally, we explore how agents can continuously improve from their interaction experiences through mechanisms, including memory management, skill optimization, and trajectory refinement.
+In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We first discuss how RL enables agents to acquire fundamental capabilities, including long-horizon planning and tool-integrated reasoning. We then introduce the role of environment design and scaling, which provide the essential foundation for training and evaluating agentic RL systems. We also discuss credit assignment, which provides more fine-grained feedback for agent learning. Finally, we explore how agents can continuously improve from their interaction experiences through mechanisms, including memory management, skill optimization, and trajectory refinement.
@@ -69,7 +69,7 @@ Before applying RL, SFT is commonly adopted to provide a cold start for LLM-base
 where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
-Similar to LLM \citep{longpre-etal:flan,zhou-etal:lima}, the scale and quality of trajectory data significantly influence the effectiveness of agent training. High-quality trajectories provide explicit supervision signals for agents to learn how to decompose complex tasks, select appropriate tools, and interact with dynamic environments. Therefore, recent studies have explored constructing trajectory-based instruction tuning datasets to enhance the planning capabilities of LLM-based agents.
+Similar to LLM instruction tuning \citep{longpre-etal:flan,zhou-etal:lima}, the scale and quality of trajectory data significantly influence the effectiveness of agent training. High-quality trajectories provide explicit supervision signals for agents to learn how to decompose complex tasks, select appropriate tools, and interact with dynamic environments. Therefore, recent studies have explored constructing trajectory-based instruction tuning datasets to enhance the planning capabilities of LLM-based agents.
 A straightforward approach is to utilize generated trajectories for instruction tuning. However, trajectories collected from LLM-based agents may contain unsuccessful plans, ineffective tool usage, or execution failures, which can introduce noisy supervision. Therefore, existing methods typically employ filtering strategies based on task success criteria or heuristic rules to select high-quality demonstrations. For example, AgentTuning constructs AgentInstruct by collecting interaction trajectories and filtering unsuccessful samples, while FireAct investigates the impact of diverse trajectory data from multiple tasks and reasoning paradigms for improving agent fine-tuning \citep{zeng-etal:agenttuning,chen-etal:fireact}.
@@ -176,10 +176,10 @@ Although SFT can teach LLMs to imitate tool-use demonstrations, it is still limi
 To address this limitation, researchers have explored RL for enhancing tool use. Instead of only imitating fixed demonstrations, the model can interact with tools during rollout and receive feedback from task outcomes or tool execution results. Given a user task $q$ and a set of available tools $\mathcal{T}$, the model generates a tool-use trajectory:
 \begin{eqnarray}
-\tau = [s_1,u_1,o_1,\cdots,s_T,u_T,o_T,a]
+\tau = [h_1,u_1,o_1,\cdots,h_T,u_T,o_T,y]
 \end{eqnarray}
-where $s_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the model action, $o_t$ denotes the observation returned by the tool or environment, and $a$ denotes the final answer. Here, $u_t$ can be either a natural language reasoning step, a tool invocation with arguments, or a final response. The goal of RL is to optimize the policy so that the generated trajectory receives a higher reward:
+where $h_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the high-level tool-use action, $o_t$ denotes the observation returned by the tool or environment, and $y$ denotes the final answer. Here, $h_t$ can include natural language reasoning, while $u_t$ represents an executable behavior such as selecting a tool and providing arguments. The goal of RL is to optimize the policy so that the generated trajectory receives a higher reward:
 \begin{eqnarray}
 \max_{\theta}\ 
 \mathbb{E}_{\tau \sim \pi_{\theta}(\cdot|q,\mathcal{T})}
@@ -241,6 +241,7 @@ However, constructing high-quality environments manually is expensive and diffic
 \centering
 \resizebox{\linewidth}{!}{
 \input{section6/Figures/environment-design}}
+\vspace{-0.2cm}
 \caption{Overview of environment synthesis and interaction for agentic RL. Agentic RL enables continuous learning through an interaction-learning loop, where agents collect trajectories from synthesized environments and enhance their capabilities.}
 \label{fig:agent-environment-scaling}
 \end{figure}
@@ -261,9 +262,9 @@ One direction focuses on scaling environment diversity. RandomWorld procedurally
 Another direction focuses on executability. EnvScaler first constructs environment skeletons that specify state schemas, available tools, and interaction rules. It then generates multiple task scenarios within each environment \citep{song-etal:envscaler}. Agent World Model follows a similar idea by implementing synthetic environments with executable code and database-backed states \citep{wang-etal:awm}. Compared with purely language-based simulations, these programmatic environments produce more consistent state transitions and generate tool outputs through actual execution.
-Verifiability is equally important because RL requires reliable feedback. Instead of evaluating only the textual final response, an environment can check whether the agent has produced the desired state change. Given the final environment state $s_T$ and the task goal $g$, we can define a state-based reward as
+Verifiability is equally important because RL requires reliable feedback. Instead of evaluating only the textual final response, an environment can check whether the agent has produced the desired state change. Given the final environment state $s^e_T$ and the task goal $g$, we can define a state-based reward as
 \begin{eqnarray}
-R_e(\tau)=V_e(s_T,g)
+R_e(\tau)=V_e(s^e_T,g)
 \end{eqnarray}
 where $V_e(\cdot)$ may be implemented using executable tests, database queries, or rule-based validators. For example, in a database operation task, the verifier can directly check whether the required record has been inserted correctly. This provides more reliable feedback than comparing the generated answer with a reference text. Both EnvScaler and Agent World Model use such mechanisms to connect agent actions with observable state changes.
@@ -281,8 +282,9 @@ This challenge is especially important for agent planning. In many agent tasks, 
    \resizebox{\linewidth}{!}{
        \input{section6/Figures/agent-credit-assignment.tex}
    }
-    \vspace{-1.0cm}
+    \vspace{-1.2cm}
    \caption{Illustration of credit assignment in agentic RL.}
+    \vspace{-0.2cm}
    \label{fig:agent-credit-assignment}
 \end{figure*}
@@ -297,7 +299,7 @@ where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the
 The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct. 
-For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
+For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. These step-level rewards are diagnostic signals rather than a conservative numerical decomposition, so they need not sum to the trajectory-level reward. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
 \begin{eqnarray}
 R(\tau)=0.3
 \quad \Rightarrow \quad
@@ -313,6 +315,7 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
 \subsection{Learning from Agentic Experience}
+\vspace{-0.2cm}
 \label{sec:learning_from_agentic_experience}
 AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.
@@ -341,23 +344,24 @@ Memory management is a direct way for agents to learn from agentic experience. D
 \begin{itemize}
    \item \textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory bank to support the current interaction. The retrieved information can take various forms depending on how the memory bank is constructed. For example, the memory bank may contain successful trajectories from similar tasks, which can provide reusable solutions for the current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles. 
    \item \textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
-    \item \textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in \{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically implement this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
+    \item \textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $\mathrm{op} \in \mathcal{A}^m = \{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically implement this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
 \end{itemize}
-In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''. 
+In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``The user adopted two dogs, Buddy and Scout''. 
 One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
 \begin{eqnarray}
-    (o,m') \sim \pi_{\theta}(\cdot \mid x^\mathrm{mem}, \mathcal{M}_\mathrm{old})
+    (\mathrm{op},m') \sim \pi_{\theta}(\cdot \mid x^\mathrm{mem}, \mathcal{M}_\mathrm{old})
 \end{eqnarray}
-where $o$ denotes the selected memory operation from the operation set $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
+where $\mathrm{op}$ denotes the selected memory operation from the operation set $\mathcal{A}^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
 \begin{figure}[!t]
    \centering
    \input{section6/Figures/reinfroced-memory-manager.tex}
+    \vspace{-2mm}
    \caption{Overview of training a memory manager with RL.}
    \label{fig:reinforced-memory}
 \end{figure}
@@ -367,7 +371,7 @@ During optimization, the reward is defined according to the final task performan
 R_{\text{answer}} = \text{EM}(y_{\text{pred}}, y_{\text{gold}}) 
 \end{eqnarray}
-where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.
+where $\text{EM}(\cdot)$ denotes the Exact Match function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.
 After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately \citep{xu-etal:a-mem}. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. This indicates that memory can serve as an important component of agent learning, beyond simply storing past information.