update.

.vscode/settings.json

 {
     "cSpell.words": [
         "agentic",
+        "Enter",
+        "EnvScaler",
+        "executability",
         "multimodal"
     ]
 }
\ No newline at end of file

section6/section6.tex

@@ -11,7 +11,7 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
 
 
 \subsection{Building Agent Capabilities}
-
+\label{sec:building-agent-capabilities}
 
 LLM-based agents extend LLMs from passive response generators into interactive decision-making systems. Rather than producing a single answer, an agent must repeatedly observe its environment, decide what to do next, and execute an action. This interaction loop allows the agent to solve tasks that require multiple steps, external resources, and adaptation during execution. The underlying LLM already provides several important capabilities, including instruction understanding, reasoning, code generation, and broad world knowledge. These capabilities allow the agent to interpret complex goals and propose plausible intermediate steps. However, they do not automatically guarantee reliable behavior in an interactive environment. An agent must also determine how to decompose a task, choose among alternative actions, coordinate external tools, and recover when an earlier decision leads to an unexpected result.
 
@@ -48,7 +48,7 @@ $\cdots \cdots$
 
 Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool use, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In practice, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization. 
 
-\subsubsubsection{Supervised Fine-Tuning}
+\subsubsubsection{Supervised Planning Data}
 
 
 \begin{figure*}[!t]
@@ -76,7 +76,7 @@ Beyond selecting high-quality trajectories, another challenge lies in effectivel
 Furthermore, even with effective trajectory selection and representation, scaling the generation of diverse and high-quality trajectory data remains challenging. Existing approaches often rely on manually designed environments and planning tasks, which limits the diversity and coverage of collected trajectories. Designing new environments requires substantial human expertise, while constructing tasks with appropriate difficulty levels remains difficult. To address this limitation, AGENTGEN explores automatically generating diverse environments and planning tasks with LLMs, enabling large-scale synthesis of trajectory data with varying task complexity for agent training \citep{hu-etal:agentgen}.
 
 
-\subsubsubsection{Reinforcement Learning}
+\subsubsubsection{Learning to Plan with Reinforcement Learning}
 
 Although SFT can provide a useful cold start for agent planning, it mainly teaches the model to imitate offline demonstrations. This limits its ability to improve beyond the quality and coverage of the collected trajectories. In contrast, RL further optimizes planning through direct interaction with the environment, where the agent receives feedback based on the outcome of its generated plans and executed behaviors. 
 % This makes RL particularly suitable for improving long-horizon planning, because the quality of a plan is often determined by the success of the entire interaction trajectory rather than by the correctness of any single intermediate step.
@@ -187,7 +187,7 @@ Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes.
 This demonstration teaches the model a tool-use pattern. The model learns to reason about the task, invoke an external tool when necessary, read the returned observation\footnote{We refer to the response returned by an external tool as an ``observation''.}, and continue the execution process.
 
 % 3. SFT for Tool Use
-\subsubsubsection{Supervised Fine-Tuning}
+\subsubsubsection{Supervised Tool-use Data}
 To make tool use more reliable, we can further train LLMs on tool-use trajectories with supervised fine-tuning. In this setting, each training example contains a user task, available tool descriptions, and a demonstrated tool-use process. The model learns when to invoke a tool, which tool to select, how to construct valid arguments, and how to use the returned observation. For example, we can construct a training example as follows:
 \begin{center}
     $\mathbf{x} =$ Task: Weng earns \$12 ... Tools: calculator(expression), ... , update\_record(json) \newline
@@ -203,7 +203,7 @@ A representative approach to addressing this data construction problem is Toolfo
 Beyond learning to use a small set of tools, later studies further scale tool-use training to larger and more realistic API spaces. API-Bank builds a benchmark and training set for tool-augmented LLMs, where models need to plan, retrieve, and call APIs in executable environments \citep{li-etal:apibank}. ToolLLM constructs ToolBench with more than 16,000 real-world APIs and uses automatically generated instruction-solution trajectories to train ToolLLaMA \citep{qin-etal:toolllm}. Gorilla focuses on API calling at scale and fine-tunes LLaMA-based models to generate accurate API calls. It also uses a document retriever to reduce API hallucination and adapt to changing API documents \citep{patil-etal:gorilla}.
 
 % 4. RL for Tool Use
-\subsubsubsection{Reinforcement Learning}
+\subsubsubsection{Learning to Use Tools with Reinforcement Learning}
 
 Although SFT can teach LLMs to imitate tool-use demonstrations, it is still limited by the coverage and quality of offline trajectories. The model mainly learns the tool-use patterns that appear in the supervised data. As a result, it may fail to generalize to unfamiliar tools, complex tool combinations, or new interaction patterns. More importantly, SFT does not directly optimize whether a tool call is necessary, whether the selected tool is appropriate, or whether the returned observation improves the final answer. This limitation becomes more serious in multi-step tool-use scenarios, \textit{where the model needs to decide when to call a tool, how to formulate the tool input, how to interpret the tool output, and when to stop using tools}.
 
@@ -280,7 +280,7 @@ This process is illustrated in Figure~\ref{fig:agent-environment-scaling}. It ty
 \item \textbf{Scenario Generation.} Based on the environment specification, we generate diverse initial states and user tasks. The tasks should cover different goals, constraints, and difficulty levels. For example, in a shopping environment, a simple task may ask the agent to purchase a single item, while a more difficult task may require it to compare multiple products.
 \item \textbf{Agent--Environment Interaction.} The agent interacts with the generated environment through multiple rounds of actions and observations. At each step, the agent selects a tool and provides the required arguments. The environment executes the action and returns an observation.
 \item \textbf{Outcome Verification.} After the interaction is completed, a verifier checks whether the task has been successfully solved. The verifier may inspect tool execution results, database states, or other structured records. 
-\item \textbf{Agent Learning.} Finally, the verified trajectories are used to improve the agent. On the one hand, successful trajectories can serve as demonstrations for SFT, while task-level or step-level feedback can support RL training. On the other hand, failed interactions are also valuable because they reveal weaknesses in the current policy and gaps in the task distribution. This can guide the generation of targeted scenarios that exercise the agent's weak capabilities and provide more informative experience for subsequent training \citep{chen-etal:failure-attribution,sun-etal:learning-from-failure}.
+\item \textbf{Agent Learning.} Finally, the verified trajectories are used to improve the agent. On the one hand, successful trajectories can serve as demonstrations for SFT, while task-level or step-level feedback can support RL training. On the other hand, failed interactions are also valuable because they reveal weaknesses in the current policy and gaps in the task distribution. This can guide the generation of targeted scenarios that exercise the agent's weak capabilities and provide more informative experience for subsequent training \citep{chen-etal:failure-attribution,sun-etal:learning-from-failure}. More broadly, this process can be viewed as learning from agentic experience, which we will discuss in Section~\ref{sec:learning_from_agentic_experience}.
 \end{itemize}
 
 
@@ -300,9 +300,9 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
 
 
 \subsection{Learning from Agentic Experience}
+\label{sec:learning_from_agentic_experience}
 
-
-
+% AI 正在从依赖人类数据的"人类数据时代"，转向主要依靠智能体自身交互经验的"经验时代". As mentioned in Section~\ref{sec:building-agent-capabilities}, 构建人类文本、示范和偏好数据，虽然仍然有用，但很难单独支撑贴近人类或者超越人类的智能。这其中主要的原因一方面是，人类的智能无法使用大量的样例数据来进行枚举的。另外一个问题，
 
 
 \subsubsection{Memory Management}