where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
where $\mathbf{rat}$ is the labeled CoT rationale for generating a preference between $\mathbf{y}_a$ and $\mathbf{y}_b$. In practice, we can obtain the evaluation results by prompting a standard LLM (as known as LLM-as-a-judge), as demonstrated in Section \ref{sec:automatic-preference-data-generation}. However, this approach typically underperforms compared to LLMs trained with preference data \citep{zhang-etal:2024generative,mahan-etal:2024generative}. One reason is that standard LLMs are not fine-tuned specifically for the reward modeling task, and as a result, they may not fully capture the nuanced decision-making process that aligns better with human preferences. Also, LLMs trained with preference data learn to prioritize outputs based on feedback, enhancing their capability to evaluate outputs according to the desired behaviors and objectives.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
Although incorporating CoT rationales into generative reward models improves preference learning, it also introduces a probability degeneration issue in Eq.~(\ref{eq:apply-generative-rm}). Specifically, preference token probabilities often become saturated near 0 or 1, resulting in limited reward variance. This issue is particularly challenging for RL algorithms such as GRPO, where relative reward differences are essential for policy optimization. To address this limitation, researchers investigate ranking-based reward construction approaches, which leverage the ranking capability of generative reward models to derive more effective rewards for RL \citep{wang-etal:rrc}. Interested readers can refer to this work for further details.
...
@@ -224,7 +223,7 @@ A straightforward approach to evaluate a reward model is to measure its effectiv
...
@@ -224,7 +223,7 @@ A straightforward approach to evaluate a reward model is to measure its effectiv
\item\textbf{Pairwise Ranking Evaluation.}
\item\textbf{Pairwise Ranking Evaluation.}
Although RL-based evaluation directly measures whether a reward model can improve downstream policy optimization, it suffers from two limitations. First, it introduces significant evaluation costs. Since RL training requires substantial computational resources and time, it is difficult to efficiently compare different reward models. Second, the evaluation results can be sensitive to other factors in the RL pipeline, such as the choice of RL algorithms, which may introduce additional variations beyond the quality of the reward model itself. To address these limitations, a more efficient approach is to directly evaluate the preference ranking ability of reward models. Given a prompt $\mathbf{x}$ and two candidate outputs $\mathbf{y}^{+}$ and $\mathbf{y}^{-}$, where $\mathbf{y}^{+}$ is preferred over $\mathbf{y}^{-}$ according to human preference annotations, the reward model predicts their relative preference. Specifically, for discriminative reward models, we compare the predicted scores of the two outputs. If the reward model assigns a higher score to $\mathbf{y}^{+}$, its prediction is consistent with human preference. For generative reward models, the model directly selects $\mathbf{y}^{+}$ as the preferred output based on its generated preference. Otherwise, the prediction is considered incorrect. By constructing a large number of pairwise ranking samples, we can evaluate the reward model using ranking accuracy:
Although RL-based evaluation directly measures whether a reward model can improve downstream policy optimization, it suffers from two limitations. First, it introduces significant evaluation costs. Since RL training requires substantial computational resources and time, it is difficult to efficiently compare different reward models. Second, the evaluation results can be sensitive to other factors in the RL pipeline, such as the choice of RL algorithms, which may introduce additional variations beyond the quality of the reward model itself. To address these limitations, a more efficient approach is to directly evaluate the preference ranking ability of reward models. Given a prompt $\mathbf{x}$ and two candidate outputs $\mathbf{y}^{+}$ and $\mathbf{y}^{-}$, where $\mathbf{y}^{+}$ is preferred over $\mathbf{y}^{-}$ according to human preference annotations, the reward model predicts their relative preference. Specifically, for discriminative reward models, we compare the predicted scores of the two outputs. If the reward model assigns a higher score to $\mathbf{y}^{+}$, its prediction is consistent with human preference. For generative reward models, the model directly selects $\mathbf{y}^{+}$ as the preferred output based on its generated preference. Otherwise, the prediction is considered incorrect. By constructing a large number of pairwise ranking samples, we can evaluate the reward model using ranking accuracy:
where $N$ denotes the number of evaluation samples. Examples of this evaluation approach include RewardBench \citep{lambert-etal:Rewardbench,malik-etal:rewardbench}, RM-Bench \citep{liu-etal:rm-bench}, RMB \citep{zhou-etal:rmb}, and JudgeBench \citep{tan-etal:judgebench}.
where $N$ denotes the number of evaluation samples. Examples of this evaluation approach include RewardBench \citep{lambert-etal:Rewardbench,malik-etal:rewardbench}, RM-Bench \citep{liu-etal:rm-bench}, RMB \citep{zhou-etal:rmb}, and JudgeBench \citep{tan-etal:judgebench}.
\item\textbf{Listwise Ranking Evaluation.}
\item\textbf{Listwise Ranking Evaluation.}
In practical alignment scenarios, the reward model often needs to select the best output from multiple candidates, such as in best-of-$n$ sampling or reranking. Therefore, listwise ranking evaluation measures whether a reward model can produce a consistent ranking over a set of candidate outputs. Specifically, given a prompt $\mathbf{x}$ and a candidate output set:
In practical alignment scenarios, the reward model often needs to select the best output from multiple candidates, such as in best-of-$n$ sampling or reranking. Therefore, listwise ranking evaluation measures whether a reward model can produce a consistent ranking over a set of candidate outputs. Specifically, given a prompt $\mathbf{x}$ and a candidate output set:
The predicted ranking is then compared with human preference rankings to evaluate the consistency between the reward model and human judgments. Examples of this evaluation approach include PPE \citep{frick-etal:ppe}.
The predicted ranking is then compared with human preference rankings to evaluate the consistency between the reward model and human judgments. Examples of this evaluation approach include PPE \citep{frick-etal:ppe}.
@@ -65,6 +65,7 @@ Before applying RL, SFT is commonly adopted to provide a cold start for LLM-base
...
@@ -65,6 +65,7 @@ Before applying RL, SFT is commonly adopted to provide a cold start for LLM-base
\mathbf{x}&=& [e,q] \\
\mathbf{x}&=& [e,q] \\
\mathbf{y}&=& [p,u_1,o_1,\cdots,u_T,o_T]
\mathbf{y}&=& [p,u_1,o_1,\cdots,u_T,o_T]
\end{eqnarray}
\end{eqnarray}
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.
...
@@ -87,12 +88,14 @@ Under the agent planning formulation, the LLM-based agent can be viewed as a hig
...
@@ -87,12 +88,14 @@ Under the agent planning formulation, the LLM-based agent can be viewed as a hig
\tau = [p,u_1,o_1,\cdots,u_T,o_T]
\tau = [p,u_1,o_1,\cdots,u_T,o_T]
\label{eq:tau-define}
\label{eq:tau-define}
\end{eqnarray}
\end{eqnarray}
The goal of RL is to optimize the agent policy so that the generated trajectory receives higher feedback from the environment. This objective can be written as
The goal of RL is to optimize the agent policy so that the generated trajectory receives higher feedback from the environment. This objective can be written as
where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. In practice, this reward can be defined according to task success. The objective can then be optimized using standard RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}, such as PPO or GRPO.
where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. In practice, this reward can be defined according to task success. The objective can then be optimized using standard RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}, such as PPO or GRPO.
Compared with SFT, RL provides two important advantages for agent planning. First, it enables the agent to learn from execution outcomes rather than only from static demonstrations. If a plan leads to failed tool execution or poor environmental feedback, the agent can be penalized and encouraged to explore alternative behaviors. Second, RL supports trajectory-level credit assignment, allowing the model to adjust earlier planning decisions according to later outcomes. This is crucial for agent planning, where early subgoal decomposition or tool selection can substantially affect the final task success.
Compared with SFT, RL provides two important advantages for agent planning. First, it enables the agent to learn from execution outcomes rather than only from static demonstrations. If a plan leads to failed tool execution or poor environmental feedback, the agent can be penalized and encouraged to explore alternative behaviors. Second, RL supports trajectory-level credit assignment, allowing the model to adjust earlier planning decisions according to later outcomes. This is crucial for agent planning, where early subgoal decomposition or tool selection can substantially affect the final task success.
...
@@ -120,6 +123,7 @@ Formally, given a trajectory $\tau = [p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL ob
...
@@ -120,6 +123,7 @@ Formally, given a trajectory $\tau = [p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL ob
\begin{eqnarray}
\begin{eqnarray}
R(\tau) \rightarrow\{r_p,r_1,\cdots,r_T\}
R(\tau) \rightarrow\{r_p,r_1,\cdots,r_T\}
\end{eqnarray}
\end{eqnarray}
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.
The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct.
where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indicate that the agent correctly extracts the numerical values and performs the calculation, $r_3$ indicates that the JSON formatting is partially correct, and $r_4$ indicates that the database update fails. In this way, the agent receives more precise feedback: it should preserve the correct planning.
where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indicate that the agent correctly extracts the numerical values and performs the calculation, $r_3$ indicates that the JSON formatting is partially correct, and $r_4$ indicates that the database update fails. In this way, the agent receives more precise feedback: it should preserve the correct planning.
...
@@ -214,6 +219,7 @@ To address this limitation, researchers have explored RL for tool use. Instead o
...
@@ -214,6 +219,7 @@ To address this limitation, researchers have explored RL for tool use. Instead o
\begin{eqnarray}
\begin{eqnarray}
\tau = [s_1,u_1,o_1,\cdots,s_T,u_T,o_T,a]
\tau = [s_1,u_1,o_1,\cdots,s_T,u_T,o_T,a]
\end{eqnarray}
\end{eqnarray}
where $s_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the model action, $o_t$ denotes the observation returned by the tool or environment, and $a$ denotes the final answer. Here, $u_t$ can be either a natural language reasoning step, a tool invocation with arguments, or a final response. The goal of RL is to optimize the policy so that the generated trajectory receives a higher reward:
where $s_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the model action, $o_t$ denotes the observation returned by the tool or environment, and $a$ denotes the final answer. Here, $u_t$ can be either a natural language reasoning step, a tool invocation with arguments, or a final response. The goal of RL is to optimize the policy so that the generated trajectory receives a higher reward:
\begin{eqnarray}
\begin{eqnarray}
\max_{\theta}\
\max_{\theta}\
...
@@ -222,6 +228,7 @@ where $s_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the mo
...
@@ -222,6 +228,7 @@ where $s_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the mo
R(\tau)
R(\tau)
\right]
\right]
\end{eqnarray}
\end{eqnarray}
In practice, this objective can be optimized with standard RL algorithms such as PPO or GRPO. Recent studies show that such outcome-driven optimization can help LLMs learn more adaptive tool-use behaviors, such as deciding when to search, when to execute code, and how to revise a tool call after receiving an error message \citep{jin-etal:searchr1,chen-etal:research,feng-etal:retool,singh-etal:artist}.
In practice, this objective can be optimized with standard RL algorithms such as PPO or GRPO. Recent studies show that such outcome-driven optimization can help LLMs learn more adaptive tool-use behaviors, such as deciding when to search, when to execute code, and how to revise a tool call after receiving an error message \citep{jin-etal:searchr1,chen-etal:research,feng-etal:retool,singh-etal:artist}.
However, RL-based tool use also introduces several unique challenges. First, the action space is more complex than ordinary text generation. A tool-use action may involve tool selection, argument generation, execution timing, and response integration. A small error in any part can lead to tool failure. Second, the reward signal is often sparse and delayed. The environment may only provide a final success signal after the whole trajectory is completed, making it difficult to identify which tool call causes success or failure. Third, tool use may introduce additional costs. Unnecessary tool calls can increase latency and computation cost, while insufficient tool use may lead to incorrect answers. Therefore, the model needs to learn not only how to use tools, but also how to use them efficiently.
However, RL-based tool use also introduces several unique challenges. First, the action space is more complex than ordinary text generation. A tool-use action may involve tool selection, argument generation, execution timing, and response integration. A small error in any part can lead to tool failure. Second, the reward signal is often sparse and delayed. The environment may only provide a final success signal after the whole trajectory is completed, making it difficult to identify which tool call causes success or failure. Third, tool use may introduce additional costs. Unnecessary tool calls can increase latency and computation cost, while insufficient tool use may lead to incorrect answers. Therefore, the model needs to learn not only how to use tools, but also how to use them efficiently.
...
@@ -235,6 +242,7 @@ R_{\mathrm{ans}}(\tau)
...
@@ -235,6 +242,7 @@ R_{\mathrm{ans}}(\tau)
+ \gamma R_{\mathrm{tool}}(\tau)
+ \gamma R_{\mathrm{tool}}(\tau)
- \lambda C_{\mathrm{tool}}(\tau)
- \lambda C_{\mathrm{tool}}(\tau)
\end{eqnarray}
\end{eqnarray}
where $R_{\mathrm{ans}}$ measures final answer correctness, $R_{\mathrm{fmt}}$ measures whether the model follows the required tool-use format, $R_{\mathrm{exec}}$ measures whether tool calls can be successfully executed, and $R_{\mathrm{tool}}$ measures whether the model selects appropriate tools and passes valid arguments. $C_{\mathrm{tool}}$ denotes the cost of tool use, such as the number of tool calls, latency, or computational expense. The coefficients $\alpha$, $\beta$, $\gamma$, and $\lambda$ control the relative importance of different reward terms.
where $R_{\mathrm{ans}}$ measures final answer correctness, $R_{\mathrm{fmt}}$ measures whether the model follows the required tool-use format, $R_{\mathrm{exec}}$ measures whether tool calls can be successfully executed, and $R_{\mathrm{tool}}$ measures whether the model selects appropriate tools and passes valid arguments. $C_{\mathrm{tool}}$ denotes the cost of tool use, such as the number of tool calls, latency, or computational expense. The coefficients $\alpha$, $\beta$, $\gamma$, and $\lambda$ control the relative importance of different reward terms.
In application, this reward decomposition provides denser feedback than final-answer rewards. For example, if the model selects the correct tool but passes an invalid argument, the execution reward can penalize the invalid call while preserving credit for the correct tool selection. If the model reaches the correct answer but uses many unnecessary tool calls, the cost term can discourage inefficient behavior. ToolRL systematically studies this reward design problem and shows that fine-grained reward decomposition can make RL training more stable and effective for tool selection and tool application \citep{qian-etal:toolrl}.
In application, this reward decomposition provides denser feedback than final-answer rewards. For example, if the model selects the correct tool but passes an invalid argument, the execution reward can penalize the invalid call while preserving credit for the correct tool selection. If the model reaches the correct answer but uses many unnecessary tool calls, the cost term can discourage inefficient behavior. ToolRL systematically studies this reward design problem and shows that fine-grained reward decomposition can make RL training more stable and effective for tool selection and tool application \citep{qian-etal:toolrl}.
...
@@ -250,6 +258,7 @@ The previous subsections focus on optimizing the agent policy for planning and t
...
@@ -250,6 +258,7 @@ The previous subsections focus on optimizing the agent policy for planning and t
\begin{eqnarray}
\begin{eqnarray}
e = (\mathcal{S},\mathcal{U},\mathcal{O},P_e,V_e)
e = (\mathcal{S},\mathcal{U},\mathcal{O},P_e,V_e)
\end{eqnarray}
\end{eqnarray}
where $\mathcal{S}$ is the state space, $\mathcal{U}$ is the set of available actions or tool invocations, and $\mathcal{O}$ is the observation space. The transition function $P_e$ determines how an action changes the environment state, while the verifier $V_e$ determines whether the resulting state satisfies the task objective. This formulation describes how the environment produces observations and feedback, rather than how the agent generates its actions.
where $\mathcal{S}$ is the state space, $\mathcal{U}$ is the set of available actions or tool invocations, and $\mathcal{O}$ is the observation space. The transition function $P_e$ determines how an action changes the environment state, while the verifier $V_e$ determines whether the resulting state satisfies the task objective. This formulation describes how the environment produces observations and feedback, rather than how the agent generates its actions.
The quality of an environment directly affects what an agent can learn. For example, an agent trained in a narrow environment may simply memorize specific APIs, task templates, or interaction patterns. An unreliable environment may return inconsistent observations or incorrect rewards, introducing noise into the learning process. Ideally, an effective environment should provide diverse tasks, executable interactions, reliable state transitions, and verifiable outcomes.
The quality of an environment directly affects what an agent can learn. For example, an agent trained in a narrow environment may simply memorize specific APIs, task templates, or interaction patterns. An unreliable environment may return inconsistent observations or incorrect rewards, introducing noise into the learning process. Ideally, an effective environment should provide diverse tasks, executable interactions, reliable state transitions, and verifiable outcomes.
...
@@ -264,6 +273,7 @@ To encourage generalization, we can train the agent over a distribution of envir
...
@@ -264,6 +273,7 @@ To encourage generalization, we can train the agent over a distribution of envir
R_e(\tau)
R_e(\tau)
\right]
\right]
\end{eqnarray}
\end{eqnarray}
where $p(\mathcal{E})$ denotes the environment distribution, and $p_e(\mathcal{Q})$ denotes the task distribution within environment $e$. The interaction trajectory $\tau$ follows the definition introduced in the previous subsections. This objective shows that agent performance depends not only on policy optimization, but also on the coverage and quality of the environments used for training.
where $p(\mathcal{E})$ denotes the environment distribution, and $p_e(\mathcal{Q})$ denotes the task distribution within environment $e$. The interaction trajectory $\tau$ follows the definition introduced in the previous subsections. This objective shows that agent performance depends not only on policy optimization, but also on the coverage and quality of the environments used for training.
However, constructing high-quality environments manually is expensive and difficult to scale. Real-world systems may be inaccessible and costly. Purely language-based simulations are easier to build, but they may generate inconsistent state transitions \citep{li-etal:word2world}. Thus, recent studies explore procedural and programmatic environment synthesis, where executable programs and structured states are used to provide a scalable and reliable interaction experience.
However, constructing high-quality environments manually is expensive and difficult to scale. Real-world systems may be inaccessible and costly. Purely language-based simulations are easier to build, but they may generate inconsistent state transitions \citep{li-etal:word2world}. Thus, recent studies explore procedural and programmatic environment synthesis, where executable programs and structured states are used to provide a scalable and reliable interaction experience.
...
@@ -297,6 +307,7 @@ Verifiability is equally important because RL requires reliable feedback. Instea
...
@@ -297,6 +307,7 @@ Verifiability is equally important because RL requires reliable feedback. Instea
\begin{eqnarray}
\begin{eqnarray}
R_e(\tau)=V_e(s_T,g)
R_e(\tau)=V_e(s_T,g)
\end{eqnarray}
\end{eqnarray}
where $V_e(\cdot)$ may be implemented using executable tests, database queries, or rule-based validators. For example, in a database operation task, the verifier can directly check whether the required record has been inserted correctly. This provides more reliable feedback than comparing the generated answer with a reference text. Both EnvScaler and Agent World Model use such mechanisms to connect agent actions with observable state changes.
where $V_e(\cdot)$ may be implemented using executable tests, database queries, or rule-based validators. For example, in a database operation task, the verifier can directly check whether the required record has been inserted correctly. This provides more reliable feedback than comparing the generated answer with a reference text. Both EnvScaler and Agent World Model use such mechanisms to connect agent actions with observable state changes.
Scaling environments also introduces substantial heterogeneity. Different environments may vary in task difficulty, available tools, and interaction length. These differences can make RL training unstable and cause the agent to overfit to frequently sampled or easier environments. AutoForge addresses this issue by synthesizing difficult but verifiable tasks and estimating learning signals at the environment level \citep{cai-etal:autoforge}. This design reduces the influence of unstable simulated interactions and improves training across heterogeneous environments.
Scaling environments also introduces substantial heterogeneity. Different environments may vary in task difficulty, available tools, and interaction length. These differences can make RL training unstable and cause the agent to overfit to frequently sampled or easier environments. AutoForge addresses this issue by synthesizing difficult but verifiable tasks and estimating learning signals at the environment level \citep{cai-etal:autoforge}. This design reduces the influence of unstable simulated interactions and improves training across heterogeneous environments.
...
@@ -354,12 +365,14 @@ One promising approach to improve memory management is to optimize memory operat
...
@@ -354,12 +365,14 @@ One promising approach to improve memory management is to optimize memory operat
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.
where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.
After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately \citep{xu-etal:a-mem}. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. This indicates that memory can serve as an important component of agent learning, beyond simply storing past information.
After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately \citep{xu-etal:a-mem}. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. This indicates that memory can serve as an important component of agent learning, beyond simply storing past information.
...
@@ -460,6 +473,7 @@ Here we consider \textbf{SkillRL} as an example to illustrate how to optimize th
...
@@ -460,6 +473,7 @@ Here we consider \textbf{SkillRL} as an example to illustrate how to optimize th
where $\mathrm{Score}(x,s_i)$ measures the relevance between the current task and each skill, and $\mathcal{S}^{*}$ denotes the selected skill set. The selected skills are then incorporated into the agent's decision-making process and used to guide RL optimization. These skills provide high-level behavioral priors, such as task decomposition and execution workflows, which help the agent explore more effective behaviors and improve the efficiency of RL training.
where $\mathrm{Score}(x,s_i)$ measures the relevance between the current task and each skill, and $\mathcal{S}^{*}$ denotes the selected skill set. The selected skills are then incorporated into the agent's decision-making process and used to guide RL optimization. These skills provide high-level behavioral priors, such as task decomposition and execution workflows, which help the agent explore more effective behaviors and improve the efficiency of RL training.
\item Through interaction with the environment, the agent collects new trajectories. These trajectories are further analyzed to discover new skills or refine existing ones. In this way, the skill bank evolves together with the agent policy: the skill library provides prior knowledge to accelerate RL training, while improved trajectories continuously enrich and refine the skill library.
\item Through interaction with the environment, the agent collects new trajectories. These trajectories are further analyzed to discover new skills or refine existing ones. In this way, the skill bank evolves together with the agent policy: the skill library provides prior knowledge to accelerate RL training, while improved trajectories continuously enrich and refine the skill library.
