\subsection{Fundamentals of Large Language Models}
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and nations necessary for the discussions in future sections.
In this subsection, we introduce the fundamentals of supervised fine-tuning for LLMs and focus on presenting the key concepts and notations necessary for the discussions in future sections.
Although pre-trained LLMs possess a vast amount of general knowledge, they are typically limited to tasks related to language modeling. Our ultimate goal is to enable them to perform various specific tasks, such as summarization, question-answer, and machine translation.
One straightforward method to achieve this goal is supervised fine-tuning (SFT), in which the pre-trained LLM is further trained on a dataset comprising task-specific input (i.e., instruction+user input)\footnote{In this paper, the \textit{instruction} represents the description of a specific task, e.g., ``Summarize the following article''; the \textit{user input} represents the extra content to the instruction, e.g., ``Article: In recent years, solar energy has seen a significant increase in the amount of energy available to the public. recent years, solar energy has seen unprecedented growth, becoming the fastest-growing ...''} paired with their expected outputs \citep{ouyang:2022training,wei-etal:2022finetuned}.
Once we collect such preference labels, we can use them, along with the output pair and input, to train the reward model. Of course, these labels are not entirely accurate either. Recent studies have shown that AI-based feedback often exhibits a location bias problem, making it more likely to prefer the output at the front \citep{zheng-etal:2023judging}. We can consider demonstrating a few examples or using advanced prompting techniques, such as Chain-of-Though (CoT), to improve the labeling performance \citep{liu-etal:2023GEval}.
Once we collect such preference labels, we can use them, along with the output pair and input, to train the reward model. Of course, these labels are not entirely accurate either. Recent studies have shown that AI-based feedback often exhibits a location bias problem, making it more likely to prefer the output at the front \citep{zheng-etal:2023judging}. We can consider demonstrating a few examples or using advanced prompting techniques, such as Chain-of-Thought (CoT), to improve the labeling performance \citep{liu-etal:2023GEval}.
For data generation, although it is easy to scale up, it is often necessary to ensure the data is accurate and diverse. Here, the data quality and diversity issues involve not only the labeling of preferences but also the inputs and outputs of the model. Therefore, we often need to use a variety of techniques to obtain large-scale, high-quality data. For example, one can generate diverse model outputs and annotations by using different LLMs, prompts, in-context demonstrations, and so on \citep{cui-etal:2024ultra}. Furthermore, \citet{dubois-etal:2024alpacafarm} report that the variability in pairwise preference data is important for training LLMs from either human or AI feedback.
@@ -77,7 +77,7 @@ In this subsection, we discuss the use of Monte Carlo Tree Search (MCTS), a popu
\begin{itemize}
\item\textbf{Selection}.
This process begins at the root node (i.e., an input), where the algorithm selects promising child nodes (i.e., reasoning steps) based on specific selection strategies, such as the Upper Confidence Bound (UCT). More specifically, at each step $k$, among the sampled candidate reasoning steps, we aim to select the one that maximizes the UCT objective:
This process begins at the root node (i.e., an input), where the algorithm selects promising child nodes (i.e., reasoning steps) based on specific selection strategies, such as the Upper Confidence Bound (namely UCT)\footnote{When MCTS uses the upper confidence bound strategy for node selection, the resulting algorithm is commonly referred to as \textit{Upper Confidence bounds applied to Trees} (UCT).}. More specifically, at each step $k$, among the sampled candidate reasoning steps, we aim to select the one that maximizes the UCT objective:
\begin{eqnarray}
\bar{\mathbf{y}}_{k}^{*}& = &\argmax_{\bar{\mathbf{y}}_{k}}\big(\bar{R}(\bar{\mathbf{y}}_{k}) + c \sqrt{\frac{\ln N_p}{N_{\bar{\mathbf{y}}_{k}}}}\big)
@@ -319,7 +319,7 @@ AI systems are gradually moving from an era dominated by human-generated data to
The first approach is \textbf{memory management} though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The second approach is \textbf{skill optimization}. One challenge of using memory to learn from experience is scalability. As agent experience continuously accumulates, the memory size will keep growing. Moreover, a large amount of noisy and redundant information may also be introduced into the memory. As a result, As a result, efficiently storing and retrieving relevant information becomes increasingly difficult. Instead of storing individual experiences in memory, skill aims to identify common patterns across experiences and transform them into abstract behaviors. These skills provide a more compact representation of experience and allow the agent to transfer experience across different tasks. Similar to memory optimization, we can also update and refine the skills of agents to achieve learning from experience.
The second approach is \textbf{skill optimization}. One challenge of using memory to learn from experience is scalability. As agent experience continuously accumulates, the memory size will keep growing. Moreover, a large amount of noisy and redundant information may also be introduced into the memory. As a result, efficiently storing and retrieving relevant information becomes increasingly difficult. Instead of storing individual experiences in memory, skill aims to identify common patterns across experiences and transform them into abstract behaviors. These skills provide a more compact representation of experience and allow the agent to transfer experience across different tasks. Similar to memory optimization, we can also update and refine the skills of agents to achieve learning from experience.
The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.
...
...
@@ -332,7 +332,7 @@ The three approaches mentioned above can be implemented through various techniqu
Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
\begin{itemize}
\item\textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory bank to support the current interaction. The retrieved information can take various forms depending on how the memory bank is constructed. For example, the memory bank may contain successful trajectories from similar tasks, which can provide reusable solutions for current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles.
\item\textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
\item\textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
\item\textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in\{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically achieve this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
\end{itemize}
...
...
@@ -341,7 +341,7 @@ Memory management is a direct way for agents to learn from agentic experience. D
\centering
\input{section6/Figures/memory-and-retrieval.tex}
\caption{
An overview of a memory system for agentic interaction. We take a travel-planning scenario as an example, where the user asks the agent to plan a trip to Kyoto. The agent first retrieves relevant user preferences from the memory bank, such as vegetarian meals, window seats, and budget hotels, and uses them to support the current conversation. During the interaction, the user provides new information, such as a preference for quiet ryokans near Kyoto Station. This new information is extracted as memory and passed to a memory manager, which updates the memory bank by selecting an operation such as \texttt{ADD}. \colorbox{blue!30}{Blue} highlights denote retrieved memories, while \colorbox{red!10}{read} highlights denote newly extracted memories and their update path back to the memory bank.
An overview of a memory system for agentic interaction. We take a travel-planning scenario as an example, where the user asks the agent to plan a trip to Kyoto. The agent first retrieves relevant user preferences from the memory bank, such as vegetarian meals, window seats, and budget hotels, and uses them to support the current conversation. During the interaction, the user provides new information, such as a preference for quiet ryokans near Kyoto Station. This new information is extracted as memory and passed to a memory manager, which updates the memory bank by selecting an operation such as \texttt{ADD}. \colorbox{blue!30}{Blue} highlights denote retrieved memories, while \colorbox{red!10}{red} highlights denote newly extracted memories and their update path back to the memory bank.
}
\label{fig:memory-and-retrieve}
\end{figure}
...
...
@@ -459,7 +459,7 @@ The above way of generating skills often suffer from quality issues. First, a ge
\end{figure*}
Here we consider \textbf{SkillRL} as an example to illustrate how to optimize the skill generation though RL \citep{xia-etal:skillrl}. The idea is that instead of directly generating skills from stored trajectories, we can first abstract reusable skills from existing trajectories and uses them to guide agentic RL training. During the training process, the agent continuously discovers new skills and updates the skill bank, enabling the joint evolution of the agent and its skills. Figure~\ref{fig:skillrl} shows a schematic illustration of SkillRL. Here we give a brief outline of the key steps involved.
Here we consider \textbf{SkillRL} as an example to illustrate how to optimize the skill generation through RL \citep{xia-etal:skillrl}. The idea is that instead of directly generating skills from stored trajectories, we can first abstract reusable skills from existing trajectories and uses them to guide agentic RL training. During the training process, the agent continuously discovers new skills and updates the skill bank, enabling the joint evolution of the agent and its skills. Figure~\ref{fig:skillrl} shows a schematic illustration of SkillRL. Here we give a brief outline of the key steps involved.
\begin{itemize}
\item Initially, we collect interaction trajectories from environments. These trajectories can include both successful and failed experiences, which provide diverse signals for skill discovery and optimization. Successful trajectories reveal effective behaviors, while failed trajectories help identify potential weaknesses and improvement opportunities.
\item The collected trajectories are then used to extract reusable skills, which initialize the skill bank. Considering that agent capabilities usually include both general problem-solving strategies and task-specific procedures, we can organize skills at different levels of abstraction. Specifically, a hierarchical skill library can be constructed, where a general skill bank stores transferable skills learned across different tasks to improve generalization, while a task-specific skill bank preserves specialized strategies for solving particular tasks.
