\node [anchor=north west, text width=7.6cm,align=left] (n4) at ([xshift=\ssep]b4.north east) {Retrieve relevant skills then provide retrieved skills to the agent.
Illustration of SkillRL \citep{xia-etal:skillrl}. SkillRL maintains a pool of agentic trajectories and initially abstracts reusable skills from these trajectories. The extracted skills are then used to guide RL training. As the agent improves through RL training, it generates higher-quality trajectories, which are further used to refine the skill bank.
Illustration of SkillRL \citep{xia-etal:skillrl}. SkillRL maintains a pool of agentic trajectories and initially abstracts reusable skills from these trajectories. The extracted skills are then used to guide RL training. As the agent improves through RL training, it generates higher-quality trajectories, which are further used to refine the skill bank.
}
}
...
@@ -566,4 +565,4 @@ It is worth noting that we can obtain feedback from either internal evaluation o
...
@@ -566,4 +565,4 @@ It is worth noting that we can obtain feedback from either internal evaluation o
The above approaches mainly rely on prompting to enable trajectory refinement. Such approaches still face several limitations. First, their performance is bounded by the intrinsic refinement ability of the LLM. A pre-trained LLM may not naturally know how to interpret feedback and effectively revise its trajectory without additional optimization. Second, prompt-based refinement mainly improves the current trajectory at inference time, which limits its ability to accumulate and transfer experience across tasks. As discussed in Sections~\ref{sec:memory-management} and~\ref{sec:skill-optimization}, failed trajectories contain valuable experiences about agent weaknesses and can provide useful learning signals for improving future behaviors. However, prompting-based approaches only leverage these experiences within the current interaction context. To address these limitations, recent studies explore training agents to acquire trajectory refinement abilities \citep{fu-etal:agentrefine}, either by updating model parameters or optimizing refinement behaviors based on collected experiences \citep{song-etal:trial}. Interested readers can refer to these works for more information.
The above approaches mainly rely on prompting to enable trajectory refinement. Such approaches still face several limitations. First, their performance is bounded by the intrinsic refinement ability of the LLM. A pre-trained LLM may not naturally know how to interpret feedback and effectively revise its trajectory without additional optimization. Second, prompt-based refinement mainly improves the current trajectory at inference time, which limits its ability to accumulate and transfer experience across tasks. As discussed in Sections~\ref{sec:memory-management} and~\ref{sec:skill-optimization}, failed trajectories contain valuable experiences about agent weaknesses and can provide useful learning signals for improving future behaviors. However, prompting-based approaches only leverage these experiences within the current interaction context. To address these limitations, recent studies explore training agents to acquire trajectory refinement abilities \citep{fu-etal:agentrefine}, either by updating model parameters or optimizing refinement behaviors based on collected experiences \citep{song-etal:trial}. Interested readers can refer to these works for more information.
Although trajectory refinement does not explicitly update model parameters like traditional RL, it shares the fundamental principle of RL: improving agent behaviors based on feedback collected from interactions. From the perspective of in-context learning, the feedback obtained during trajectory refinement can be viewed as a temporary reward signal that modifies the agent's subsequent decision-making process. Instead of updating policy parameters, the agent updates its context with reflections, which potentially modifies the policy used for future actions within the current interaction. Therefore, trajectory refinement can be regarded as an in-context form of policy improvement. This perspective is closely related to the emerging research direction of \textit{in-context RL} (namely ICRL), which studies how agents can adapt their behaviors through interaction feedback without explicit parameter updates \citep{monea-etal:llms}.
Although trajectory refinement does not explicitly update model parameters like traditional RL, it shares the fundamental principle of RL: improving agent behaviors based on feedback collected from interactions. From the perspective of in-context learning, the feedback obtained during trajectory refinement can be viewed as a temporary reward signal that modifies the agent's subsequent decision-making process. Instead of updating policy parameters, the agent updates its context with reflections, which potentially modifies the policy used for future actions within the current interaction. Therefore, trajectory refinement can be regarded as an in-context form of policy improvement. This perspective is closely related to the emerging research direction of \textit{in-context RL} (namely ICRL), which studies how agents can adapt their behaviors through interaction feedback without explicit parameter updates \citep{monea-etal:llms}.
\path[use as bounding box] (-11.4cm,-6.4cm) rectangle (11.4cm,1.25cm);
\begin{scope}
\node [phasebox] (phase-1) at (-7.5,0) {\textit{\textbf{Stage 1: Pre-training with \\ Textual Preference Data}}};
\node [phasebox] (phase-2) at ([xshift=7.5cm]phase-1.center) {\textit{\textbf{Stage 2: Fine-tuning with Image \\ Caption-based Preference Data}}};
% \path[use as bounding box] (-11.4cm,-6.4cm) rectangle (11.4cm,1.25cm);
\node [phasebox] (phase-3) at ([xshift=7.5cm]phase-2.center) {\textit{\textbf{Stage 3: Fine-tuning with \\ Visual Preference Data}}};
\scriptsize
\node [phasebox] (phase-1) at (0,0) {\textit{\textbf{Stage 1: Pre-training with \\ Textual Preference Data}}};
\node [box2] (box-1) at ([yshift=-4cm]phase-1.center) {};
\node [phasebox] (phase-2) at ([xshift=5.5cm]phase-1.center) {\textit{\textbf{Stage 2: Fine-tuning with Image \\ Caption-based Preference Data}}};
\node [box1] (box-2) at (box-1)
\node [phasebox] (phase-3) at ([xshift=5.5cm]phase-2.center) {\textit{\textbf{Stage 3: Fine-tuning with \\ Visual Preference Data}}};
{\textit{\underline{Instruction}}: Can you determine the missing number in the sequence: 2, 6, 14, 30, 62, \_\_? \\ ~ \\
\textit{\underline{Chosen Response}}: The missing number in the sequence is 126.\\\textit{\underline{Rejected Response}}: The sequence is 6.};
\node [box2] (box-1) at ([yshift=-3.6cm]phase-1.center) {};
\node [box1] (box-2) at (box-1)
\node [box2] (box-3) at ([yshift=-4cm]phase-2.center) {};
{\textit{\underline{Instruction}}: Can you determine the missing number in the sequence: 2, 6, 14, 30, 62, \_\_? \\ ~ \\
\node [box1] (box-4) at (box-3)
\textit{\underline{Chosen Response}}: The missing number in the sequence is 126.\\\textit{\underline{Rejected Response}}: The sequence is 6.};
{\textit{\underline{Instruction}}: Describe the image in detail.\\\textit{\underline{Image Caption}}: There are four mangosteens,
one of which has been cut open.\\\textit{\underline{Chosen Response}}: There are several \textcolor{temp3}{mangosteens} on a \textcolor{temp3}{wooden table}.\\
\node [box2] (box-3) at ([yshift=-3.6cm]phase-2.center) {};
\textit{\underline{Rejected Response}}: This is a dark fruit, possibly a variety of \textcolor{red}{pitaya}.\\};
\node [box1] (box-4) at (box-3)
{\textit{\underline{Instruction}}: Describe the image in detail.\\\textit{\underline{Image Caption}}: There are four mangosteens,
\node [box2] (box-5) at ([yshift=-4cm]phase-3.center) {};
one of which has been cut open.\\\textit{\underline{Chosen Response}}: There are several \textcolor{temp3}{mangosteens} on a \textcolor{temp3}{wooden table}.\\
\node [box1] (box-6) at (box-5)
\textit{\underline{Rejected Response}}: This is a dark fruit, possibly a variety of \textcolor{red}{pitaya}.\\};
{\includegraphics{section7/Figures/mangosteens.jpg}\\\textit{\underline{Chosen Response}}: There are several \textcolor{temp3}{mangosteens} on a \textcolor{temp3}{wooden table}.\\
\textit{\underline{Rejected Response}}: This is a dark fruit, possibly a variety of \textcolor{red}{pitaya}.\\};
\node [box2] (box-5) at ([yshift=-3.6cm]phase-3.center) {};
\node [text width=4.5cm,anchor=north,align=left] at ([xshift=1.2cm,yshift=-.5cm]box-6.north)
\node [box1] (box-6) at (box-5)
{\textit{\underline{Instruction}}: Describe the image in detail.};
{\includegraphics{section7/Figures/mangosteens.jpg}\\\textit{\underline{Chosen Response}}: There are several \textcolor{temp3}{mangosteens} on a \textcolor{temp3}{wooden table}.\\
\textit{\underline{Rejected Response}}: This is a dark fruit, possibly a variety of \textcolor{red}{pitaya}.\\};
\node[draw,fill=temp4,minimum height=1cm,minimum width=0.75cm,single arrow,anchor=south,shape border rotate=90,single arrow head extend=0.12cm,] (arrow) at ([yshift=0.1cm]box-1.north) {};
\node [text width=2.5cm,anchor=north,align=left] at ([xshift=1.1cm,yshift=-.5cm]box-6.north)
\node[draw,fill=temp4,minimum height=1cm,minimum width=0.75cm,single arrow,anchor=south,shape border rotate=90,single arrow head extend=0.12cm,] (arrow) at ([yshift=0.1cm]box-3.north) {};
{\textit{\underline{Instruction}}: Describe the image in detail.};
\node[draw,fill=temp4,minimum height=1cm,minimum width=0.75cm,single arrow,anchor=south,shape border rotate=90,single arrow head extend=0.12cm,] (arrow) at ([yshift=0.1cm]box-5.north) {};
\node[draw,fill=temp4,minimum height=1cm,minimum width=0.75cm,single arrow,anchor=south,shape border rotate=90,single arrow head extend=0.12cm,] (arrow) at ([yshift=0.1cm]box-1.north) {};
\draw[line width=0.5mm,arrows = {-Stealth[inset=0pt, length=0.2cm, angle'=45]}] (phase-1.east) to ([xshift=0.3cm]phase-1.east)
\node[draw,fill=temp4,minimum height=1cm,minimum width=0.75cm,single arrow,anchor=south,shape border rotate=90,single arrow head extend=0.12cm,] (arrow) at ([yshift=0.1cm]box-3.north) {};
to [out=90,in=180] ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.1cm,0.5cm)$) to [out=0,in=90] ([xshift=-0.5cm]phase-2.west) to (phase-2.west);
\node[draw,fill=temp4,minimum height=1cm,minimum width=0.75cm,single arrow,anchor=south,shape border rotate=90,single arrow head extend=0.12cm,] (arrow) at ([yshift=0.1cm]box-5.north) {};
\draw[line width=0.2mm] ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.1cm,0.8cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.3cm,-0.4cm)$);
\draw[line width=0.2mm] ($1/2*(phase-1.east)+1/2*(phase-2.west)+(0.1cm,0.8cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.1cm,-0.4cm)$);
\draw[line width=0.35mm,arrows = {-Stealth[inset=0pt, length=0.2cm, angle'=45]}] (phase-1.east) to ([xshift=0.18cm]phase-1.east)
\node at ([yshift=-0.9cm]$1/2*(phase-1.east)+1/2*(phase-2.west)$) {Bridging Task Gap};
to [out=90,in=180] ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.1cm,0.35cm)$) to [out=0,in=90] ([xshift=-0.38cm]phase-2.west) to (phase-2.west);
\draw[line width=0.2mm] ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.1cm,0.6cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.3cm,-0.4cm)$);
\draw[line width=0.5mm,arrows = {-Stealth[inset=0pt, length=0.2cm, angle'=45]}] (phase-2.east) to ([xshift=0.3cm]phase-2.east)
\draw[line width=0.2mm] ($1/2*(phase-1.east)+1/2*(phase-2.west)+(0.1cm,0.6cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-1.east)+1/2*(phase-2.west)+(-0.1cm,-0.4cm)$);
to [out=90,in=180] ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.1cm,0.5cm)$) to [out=0,in=90] ([xshift=-0.5cm]phase-3.west) to (phase-3.west);
\node at ([yshift=-0.7cm]$1/2*(phase-1.east)+1/2*(phase-2.west)$) {Bridging Task Gap};
\draw[line width=0.2mm] ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.1cm,0.8cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.3cm,-0.4cm)$);
\draw[line width=0.2mm] ($1/2*(phase-2.east)+1/2*(phase-3.west)+(0.1cm,0.8cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.1cm,-0.4cm)$);
\draw[line width=0.35mm,arrows = {-Stealth[inset=0pt, length=0.2cm, angle'=45]}] (phase-2.east) to ([xshift=0.18cm]phase-2.east)
\node at ([yshift=-0.9cm]$1/2*(phase-2.east)+1/2*(phase-3.west)$) {Bridging Modality Gap };
to [out=90,in=180] ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.1cm,0.35cm)$) to [out=0,in=90] ([xshift=-0.38cm]phase-3.west) to (phase-3.west);
\draw[line width=0.2mm] ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.1cm,0.6cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.3cm,-0.4cm)$);
\end{tikzpicture}
\draw[line width=0.2mm] ($1/2*(phase-2.east)+1/2*(phase-3.west)+(0.1cm,0.6cm)$) .. controls +(-0.5cm,-1.4cm) and +(+0.5cm,1.4cm) .. ($1/2*(phase-2.east)+1/2*(phase-3.west)+(-0.1cm,-0.4cm)$);
\ No newline at end of file
\node at ([yshift=-0.7cm]$1/2*(phase-2.east)+1/2*(phase-3.west)$) {Bridging Modality Gap };
