@@ -313,7 +313,7 @@ The second approach is \textbf{skill optimization}. One challenge of using memor
...
@@ -313,7 +313,7 @@ The second approach is \textbf{skill optimization}. One challenge of using memor
The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~\ref{eq:tau-define} and~\ref{eq:reward-optimization-for-agent}, where agents improve their behaviors through sequential interaction and feedback from experience.
The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~\ref{eq:tau-define} and~\ref{eq:reward-optimization-for-agent}, where agents improve their behaviors via feedback from experience.
\subsubsection{Memory Management}
\subsubsection{Memory Management}
...
@@ -424,7 +424,7 @@ Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Us
...
@@ -424,7 +424,7 @@ Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Us
\{More skill descriptions\}\par
\{More skill descriptions\}\par
\textbf{User Query:} Solve the equation $2x+5=13$. \par
\textbf{User Query:} Solve the equation $2x+5=13$. \par
@@ -468,10 +468,82 @@ By repeating the above process, the agent can improve its skill library through
...
@@ -468,10 +468,82 @@ By repeating the above process, the agent can improve its skill library through
\subsubsection{Trajectory Refinement}
\subsubsection{Trajectory Refinement}
Different from memory management and skill optimization, which focus on storing and abstracting experiences, trajectory refinement directly improves agent behaviors by revising generated trajectories. This idea is motivated by the self-refinement capability of LLMs. As discussed in Section~\ref{sec:tts}, reasoning-enhanced RL training enables LLMs to evaluate their intermediate reasoning processes and revise incorrect solutions. For example, when solving mathematical problems, an LLM can verify intermediate results after each reasoning step and adjust its solution trajectory based on the verification outcomes. A natural extension is to apply this self-refinement capability to LLM-based agents. During interaction with environments, agents continuously receive external feedback, such as skill execution results. Instead of directly following the initial trajectory, agents can leverage these feedback signals to revise previous decisions. In this way, the agent learns from its own interaction experience by continuously refining its behaviors.
Different from memory management and skill optimization, which focus on storing and abstracting experiences, trajectory refinement directly improves agent behaviors by revising generated trajectories. This idea is motivated by the self-refinement capability of LLMs. As discussed in Section~\ref{sec:tts}, reasoning-enhanced RL training enables LLMs to evaluate their intermediate reasoning processes and revise incorrect solutions. For example, when solving mathematical problems, an LLM can verify intermediate results after each reasoning step and adjust its solution trajectory based on the verification outcomes.
A natural extension is to apply this self-refinement capability to LLM-based agents. During interaction with environments, agents continuously receive external feedback, such as tool execution results or environment states. Instead of blindly following the initial trajectory, agents can leverage these feedback signals to identify mistakes, revise previous decisions, and generate improved trajectories. In this way, agents can learn from their own interaction experiences by continuously refining their behaviors. One commonly used trajectory refinement approach is Reflexion \citep{shinn-etal:reflexion}. The basic idea is to introduce an explicit reflection process into agent execution. Given a task, the agent first generates an initial trajectory and receives feedback from an evaluator. The agent then analyzes the feedback, summarizes potential errors or improvement strategies, and generates a reflection. Based on this reflection, the agent produces an improved trajectory in subsequent attempts. Here, we take mathematical reasoning as an example.
I need to isolate $x$ by moving the constant term to the other side.
\textbf{Action:}
Subtract 7 from both sides:
\[
3x+7-7=22-7
\]
\textbf{Observation:}
The equation becomes:
\[
3x=15
\]
\textbf{Thought:}
I need to divide both sides by 3 to obtain $x$.
\textbf{Action:}
Calculate:
\[
x=\frac{15}{3}=4
\]
\textbf{Answer:}
$x=4$.
\vspace{0.1cm}
\uline{\textbf{Feedback:} The answer is incorrect. This is because: $\frac{15}{3}=5\neq4$.}
\vspace{0.1cm}
\uline{\textbf{Reflection:} The previous trajectory failed because I made an error during intermediate calculation. For future tasks, I should carefully verify arithmetic operations and check the final answer by substituting it back into the original equation.}
