update.

section6/section6.tex

@@ -84,11 +84,13 @@ Although SFT can provide a useful cold start for agent planning, it mainly teach
 Under the agent planning formulation, the LLM-based agent can be viewed as a high-level policy that generates a plan and then produces a sequence of executable behaviors \citep{li-etal:encouraging}. Given the environment information $e$ and task description $q$, the agent first generates a plan $p$ and then interacts with the environment through a sequence of behaviors and observations:
 \begin{eqnarray}
 \tau = [p,u_1,o_1,\cdots,u_T,o_T]
+\label{eq:tau-define}
 \end{eqnarray}
 The goal of RL is to optimize the agent policy so that the generated trajectory receives higher feedback from the environment. This objective can be written as
 \begin{eqnarray}
 \theta^{*} = \arg\max_{\theta} \mathbb{E}_{\tau \sim \pi_{\theta}(\cdot|e,q)}
 \left[ R(\tau) \right]
+\label{eq:reward-optimization-for-agent}
 \end{eqnarray}
 where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. In practice, this reward can be defined according to task success. The objective can then be optimized using standard RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}, such as PPO or GRPO.
 
@@ -310,7 +312,7 @@ The second approach is \textbf{skill optimization}. One challenge of using memor
 
 The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
 
-The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that while these approaches depart from traditional parametric updates, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making.
+The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~\ref{eq:tau-define} and~\ref{eq:reward-optimization-for-agent}, where agents improve their behaviors through sequential interaction and feedback from experience.
 
 
 \subsubsection{Memory Management}