


\section{Agentic Reinforcement Learning}

Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.

With the rise of LLMs as core components of autonomous systems, RL has been increasingly used to adapt these models for sequential decision-making in dynamic and open-ended environments. In this setting, agentic RL addresses a key limitation of LLMs: although they encode extensive world knowledge, their outputs are not inherently aligned with long-horizon task requirements. For example, a supervised fine-tuned LLM may correctly generate a script for calling an external API, but it cannot reliably self-correct when the environment returns unexpected errors. This creates new challenges for enabling LLM-based agents to plan autonomously, adapt to environmental feedback, and safely improve their strategies through iterative interaction.

In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory management, skill optimization, and trajectory refinement without altering the internal model weights. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.


\subsection{Building Agent Capabilities}
\label{sec:building-agent-capabilities}

LLM-based agents extend LLMs from passive response generators into interactive decision-making systems. Rather than producing a single answer, an agent must repeatedly observe its environment, decide what to do next, and execute an action. This interaction loop allows the agent to solve tasks that require multiple steps, external resources, and adaptation during execution. The underlying LLM already provides several important capabilities, including instruction understanding, reasoning, code generation, and broad world knowledge. These capabilities allow the agent to interpret complex goals and propose plausible intermediate steps. However, they do not automatically guarantee reliable behavior in an interactive environment. An agent must also determine how to decompose a task, choose among alternative actions, coordinate external tools, and recover when an earlier decision leads to an unexpected result.

These agentic capabilities are difficult to learn from static input-output pairs alone. The quality of an individual decision often depends on its effect on later interactions and the final task outcome. A locally plausible action may lead to failure several steps later, while an initially unsuccessful attempt may provide useful information for a better strategy. RL is well suited to this setting because it optimizes complete interaction trajectories through environmental feedback and delayed rewards \citep{singh-etal:singhagentic,feng-etal:retool}. It enables the agent to explore different behaviors, compare their consequences, and gradually improve its decision-making policy.

In this subsection, we focus on two core capabilities in agentic RL: \textit{planning} and \textit{tool use}. Planning allows the agent to organize a complex goal into a sequence of executable steps. Tool use allows it to access external information, perform reliable computation, and take actions beyond the capabilities stored in the model parameters. Together, these capabilities form the basis for effective interaction in given environments.


\subsubsection{Planning}

An agent's autonomy refers to its ability to independently determine the steps required to complete a given task, even when these steps are not explicitly specified in the input. This capability requires the agent to infer the necessary procedure from the task description, decompose the goal into executable subgoals, and decide when and how to invoke external tools. Such autonomy is often reflected in the generation of a structured plan that guides the execution trajectory. For example, given a specific task description, the agent may produce an actionable plan with tool-use decisions as follows:

\begin{tcolorbox}[frame empty]

\begingroup
% \setlength{\leftskip}{-0.3em}
% \setlength{\rightskip}{2em}
\centering
\renewcommand{\arraystretch}{1.5}
\begin{tabularx}{\dimexpr\linewidth-2em\relax}{>{\raggedleft\arraybackslash}p{0.08\linewidth} X}
\textcolor{gray}{Input} & \textit{Environment}: Calculator, Database API \newline 
\textit{Task}: Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes. How much did she earn? Please save the result into the database. \\
\textcolor{gray}{Output} & 
\uline{1. Extract the relevant numerical values and identify the required computation.} \newline
\uline{2. Call the \texttt{calculator} tool to evaluate the expression $12 \times (50 / 60)$.} \newline
\uline{3. Format the computed result into a valid JSON schema.} \newline
\uline{4. Invoke the \texttt{update\_record} API to save the final earnings into the database.}\newline
Next, I will execute the plan step by step. \newline
$\cdots \cdots$
\end{tabularx}

\endgroup

\end{tcolorbox}

Unlike single-turn response generation, agent planning requires a sequence of interdependent decisions, where early choices can affect subsequent tool use, environmental feedback, and final task success. Improving planning capability therefore requires not only teaching the model to generate reasonable intermediate steps, but also enabling it to adjust its behavior based on execution outcomes. To this end, researchers have explored various approaches to achieving this goal. Among them, one promising approach is RL, as it naturally formulates agent planning as sequential decision-making driven by environmental feedback and trajectory-level rewards. In practice, SFT provides a cold start by allowing the model to imitate high-quality planning trajectories, while RL further refines the agent through interaction-based feedback and trajectory-level optimization. 

\subsubsubsection{Supervised Planning Data}


\begin{figure*}[!t]
    \centering
    \resizebox{\linewidth}{!}{
    \input{section6/Figures/agent-planning-sft}}
    \caption{Overview of SFT data construction for improving agent planning.}
    \label{fig:agent-planning-sft}
\end{figure*}

Before applying RL, SFT is commonly adopted to provide a cold start for LLM-based agents by teaching them basic planning and interaction behaviors \citep{chen-etal:fireact,zhang-etal:agentohana,zeng-etal:agenttuning}. As illustrated in Figure \ref{fig:agent-planning-sft}, constructing SFT data for agent planning typically involves three stages: (1) preparing diverse environments and planning tasks; (2) synthesizing expert-level trajectories, consisting of action-observation sequences, through agent-environment interactions; and (3) selecting high-quality trajectories based on predefined evaluation criteria. Specifically, expert trajectories can be generated by leveraging state-of-the-art LLMs as expert agents and retaining reliable demonstrations according to reward signals or task success criteria. Through this process, LLMs can learn planning behaviors from demonstrations and improve their ability to generate structured execution plans.
\begin{eqnarray}
\mathbf{x} &=& [e,q] \\
\mathbf{y} &=& [p,u_1,o_1,\cdots,u_T,o_T]
\end{eqnarray}
where $e$ denotes the environment information, including available tools and external resources, and $q$ represents the task description provided by users. $p$ denotes the generated plan that decomposes the task into intermediate steps. $u_t$ and $o_t$ represent the agent's behavior and the corresponding observation from the environment at the $t$-th interaction step, respectively. Specifically, $u_t$ denotes the executable behavior performed by the agent, such as tool invocation, while $o_t$ denotes the feedback returned by the environment after executing $u_t$. Therefore, $\{u_1,o_1,\cdots,u_T,o_T\}$ forms a sequential interaction trajectory between the agent and the environment. It is worth noting that although $u_t$ and $a_t$ in Section \ref{sec:preliminary} both represent ``action'', they are defined at different levels of abstraction. Specifically, $u_t$ represents a high-level interaction step in agent planning, such as calling a specific tool, whereas $a_t$ represents the low-level token-level action in the standard LLM-based RL formulation, corresponding to the generation of an individual token by the language model.


Similar to LLM \citep{longpre-etal:flan,zhou-etal:lima}, the scale and quality of trajectory data significantly influence the effectiveness of agent training. High-quality trajectories provide explicit supervision signals for agents to learn how to decompose complex tasks, select appropriate tools, and interact with dynamic environments. Therefore, recent studies have explored constructing trajectory-based instruction tuning datasets to enhance the planning capabilities of LLM-based agents.

A straightforward approach is to utilize generated trajectories for instruction tuning. However, trajectories collected from LLM-based agents may contain unsuccessful plans, ineffective tool usage, or execution failures, which can introduce noisy supervision. Therefore, existing methods typically employ filtering strategies based on task success criteria or heuristic rules to select high-quality demonstrations. For example, AgentTuning constructs AgentInstruct by collecting interaction trajectories and filtering unsuccessful samples, while FireAct investigates the impact of diverse trajectory data from multiple tasks and reasoning paradigms for improving agent fine-tuning \citep{zeng-etal:agenttuning,chen-etal:fireact}.

Beyond selecting high-quality trajectories, another challenge lies in effectively representing the complex interaction processes within trajectories. Unlike conventional instruction tuning, agent trajectories contain not only final responses but also intermediate planning processes and multi-step interactions with environments. Therefore, how to organize and annotate different components of trajectories becomes crucial for enabling agents to acquire planning and execution capabilities. To address this challenge, AgentLUMOS introduces a modular training framework that explicitly separates high-level planning and low-level execution \citep{yin-etal:agentlumos}. Specifically, its planning module learns to generate high-level subgoals, while its grounding module learns to translate these subgoals into executable actions through external tools. Such a modular design enables agents to acquire different capabilities from structured trajectory annotations.

Furthermore, even with effective trajectory selection and representation, scaling the generation of diverse and high-quality trajectory data remains challenging. Existing approaches often rely on manually designed environments and planning tasks, which limits the diversity and coverage of collected trajectories. Designing new environments requires substantial human expertise, while constructing tasks with appropriate difficulty levels remains difficult. To address this limitation, AGENTGEN explores automatically generating diverse environments and planning tasks with LLMs, enabling large-scale synthesis of trajectory data with varying task complexity for agent training \citep{hu-etal:agentgen}.


\subsubsubsection{Learning to Plan with Reinforcement Learning}

Although SFT can provide a useful cold start for agent planning, it mainly teaches the model to imitate offline demonstrations. This limits its ability to improve beyond the quality and coverage of the collected trajectories. In contrast, RL further optimizes planning through direct interaction with the environment, where the agent receives feedback based on the outcome of its generated plans and executed behaviors. 
% This makes RL particularly suitable for improving long-horizon planning, because the quality of a plan is often determined by the success of the entire interaction trajectory rather than by the correctness of any single intermediate step.

Under the agent planning formulation, the LLM-based agent can be viewed as a high-level policy that generates a plan and then produces a sequence of executable behaviors \citep{li-etal:encouraging}. Given the environment information $e$ and task description $q$, the agent first generates a plan $p$ and then interacts with the environment through a sequence of behaviors and observations:
\begin{eqnarray}
\tau = [p,u_1,o_1,\cdots,u_T,o_T]
\label{eq:tau-define}
\end{eqnarray}
The goal of RL is to optimize the agent policy so that the generated trajectory receives higher feedback from the environment. This objective can be written as
\begin{eqnarray}
\theta^{*} = \arg\max_{\theta} \mathbb{E}_{\tau \sim \pi_{\theta}(\cdot|e,q)}
\left[ R(\tau) \right]
\label{eq:reward-optimization-for-agent}
\end{eqnarray}
where $\pi_{\theta}$ denotes the agent policy parameterized by the LLM, and $R(\tau)$ denotes the trajectory-level reward that evaluates the overall quality of the planning process. In practice, this reward can be defined according to task success. The objective can then be optimized using standard RL algorithms introduced in Section~\ref{sec:example-using-rl-training-llms}, such as PPO or GRPO.

Compared with SFT, RL provides two important advantages for agent planning. First, it enables the agent to learn from execution outcomes rather than only from static demonstrations. If a plan leads to failed tool execution or poor environmental feedback, the agent can be penalized and encouraged to explore alternative behaviors. Second, RL supports trajectory-level credit assignment, allowing the model to adjust earlier planning decisions according to later outcomes. This is crucial for agent planning, where early subgoal decomposition or tool selection can substantially affect the final task success.

% Another challenge is training stability. In multi-turn agent RL, long-horizon decision-making and stochastic environment feedback can make trajectory-level optimization unstable, while errors in early steps may accumulate and affect later interactions \citep{wang-etal:ragen,djuhera-etal:tsr}.

\subsubsubsection{Credit Assignment}
Credit assignment is not unique to LLM-based agents. It is a long-standing challenge in traditional RL. When rewards are delayed, the agent must decide which past actions should be reinforced or suppressed. Classical methods, such as temporal-difference learning and eligibility traces, address this issue by propagating reward information backward along the trajectory. In agentic RL, credit assignment becomes more challenging. This is because that each interaction step may involve high-level planning, tool invocation, and environmental feedback, rather than a single low-level action. Therefore, credit assignment often needs to operate at both the trajectory level and the step level.

This challenge is especially important for agent planning. In many agent tasks, the environment only provides a sparse reward after the entire trajectory is completed. However, the final success or failure may come from an earlier planning decision, an incorrect tool invocation, or an ineffective response to an intermediate observation. If we directly assign the same trajectory-level reward to all interaction steps, the supervision can become noisy. The agent may fail to identify which decisions should be reinforced and which should be suppressed.

\begin{figure*}[!t]
    \centering
    \resizebox{\linewidth}{!}{
        \input{section6/Figures/agent-credit-assignment.tex}
    }
    \vspace{-1.0cm}
    \caption{Illustration of credit assignment in agentic RL.}
    \label{fig:agent-credit-assignment}
\end{figure*}

% Consider a simple example. If the agent correctly plans to compute the babysitting payment but calls the database API before obtaining the calculation result, the final task may fail. In this case, the failure should mainly be attributed to the incorrect ordering of tool use rather than to the entire plan. Conversely, if the agent chooses the wrong formula at the beginning, then the later database operation may be executed correctly, but the final result is still wrong. These cases show that different steps in an interaction trajectory may contribute differently to the final outcome.

Formally, given a trajectory $\tau = [p,u_1,o_1,\cdots,u_T,o_T]$, a simple RL objective uses a trajectory-level reward $R(\tau)$ to optimize the whole planning process. However, such a reward only provides coarse feedback on the overall outcome. Therefore, credit assignment aims to decompose the trajectory-level reward into step-level signals:
\begin{eqnarray}
R(\tau) \rightarrow \{r_p,r_1,\cdots,r_T\}
\end{eqnarray}
where $r_p$ evaluates the quality of the generated plan, and $r_t$ evaluates the contribution of the $t$-th interaction step $(u_t,o_t)$ to the final task outcome.

The mechanism of credit assignment is illustrated in Figure \ref{fig:agent-credit-assignment}. This decomposition enables the agent to distinguish whether failure comes from an unreasonable plan, an invalid tool call, or a poor adaptation to environmental feedback \citep{li-etal:encouraging,xi-etal:agentprm,wang-etal:steppo}. For example, consider the task of computing the babysitting payment and saving the result into a database. Suppose the agent generates a reasonable plan and correctly calls the calculator, but fails to invoke the database API with the required JSON format. If we only use a final trajectory-level reward, the whole trajectory may receive a low reward, e.g., $R(\tau)=0.3$, even though the early planning and calculation steps are correct. 

For illustration, we can define rule-based step-level rewards using predefined verification criteria, such as whether the plan includes all necessary steps, whether the calculation is correct, whether the JSON schema is valid, and whether the database update succeeds. Then, we can use credit assignment to decompose the final trajectory-level feedback into step-level rewards:
\begin{eqnarray}
R(\tau)=0.3
\quad \Rightarrow \quad
\{r_p=0.4,\ r_1=1.0,\ r_2=1.0,\ r_3=0.7,\ r_4=0.0\}
\end{eqnarray}
where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indicate that the agent correctly extracts the numerical values and performs the calculation, $r_3$ indicates that the JSON formatting is partially correct, and $r_4$ indicates that the database update fails. In this way, the agent receives more precise feedback: it should preserve the correct planning.



% To address this challenge, researchers explore more informative feedback signals for agent planning. For example, tool-use rewards evaluate whether the tool invocation sequence is complete and effective, while process rewards or progress-based rewards assess whether each intermediate decision moves the agent closer to the final goal \citep{li-etal:encouraging,xi-etal:agentprm}. These methods provide denser supervision than sparse final rewards and make it possible to optimize intermediate planning and execution behaviors more directly.



\subsubsection{Tool Use}
Tool use is another fundamental capability of LLM-based agents. Tools extend the agent beyond its internal parameters. They allow the agent to access external knowledge, perform accurate computation, and execute actions in external systems. Early studies show that LLMs can use tools through prompt-based reasoning-action patterns. ReAct is a representative example, where the model alternates between reasoning steps and tool-use actions \citep{yao-etal:react}. However, prompt-based tool use is often unstable. It also depends heavily on the capability of the model. Thus, SFT methods train LLMs on tool-use trajectories, so that the model can learn when and how to invoke tools from demonstrations \citep{komeili-etal:Internet,schick-etal:toolformer}. Still, SFT mainly teaches imitation. It cannot directly optimize whether the agent should use a tool, which tool it should select, or how to balance tool benefit with tool cost. RL provides a natural framework for efficient tool use. We can define rewards based on task success, tool-use correctness, tool-use completeness, and tool cost \citep{qian-etal:toolrl,singh-etal:agentic}.

% 2. Prompt-based Tool Use
\subsubsubsection{Prompt-based Tool Use}
Prompt-based tool is a simple approach to enable LLM-based agents to interact with external tools without updating model parameters. However, the model may not naturally know when to invoke a tool or how to organize tool-use behaviors. Thus, we usually need to provide tool descriptions and a few demonstrations in the prompt. These demonstrations show the model how to alternate between reasoning, tool invocation, and observation. At each step, the model first generates a reasoning trace, then produces a tool-use action. After the tool returns an observation, the model uses this feedback to update its next decision. For example,
\begin{tcolorbox}[frame empty]

\begingroup
\centering
\renewcommand{\arraystretch}{1.5}
\begin{tabularx}{\dimexpr\linewidth-2em\relax}{
>{\raggedleft\arraybackslash}p{0.16\linewidth}
@{\hspace{0.6em}}
>{\raggedright\arraybackslash}X
}

\textcolor{gray}{Tools} & 
\texttt{calculator(expression)}: return the numerical result of an expression. \newline
\texttt{update\_record(json)}: save a JSON record into the database. \\

\textcolor{gray}{Demonstration} &
\textit{Task}: Tom earns \$15 per hour. He worked for 2 hours. How much did he earn? Please save the result into the database. \newline
\textbf{Thought}: I need to compute the total payment. \newline
\colorbox{green!35}{\strut\textbf{Action}: \texttt{calculator}($15 \times 2$)} \newline
\textbf{Observation}: 30 \newline
\textbf{Thought}: The payment is 30 dollars. I need to save it into the database. \newline
\textbf{Action}: \texttt{update\_record}(\{"earning": 30\}) \newline
\textbf{Observation}: success \newline
\textbf{Answer}: Tom earned \$30, and the result has been saved into the database. \\

\textcolor{gray}{Current Task} &
Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes. How much did she earn? Please save the result into the database. \\

\textcolor{gray}{Output} &
\textbf{Thought}: I need to compute the payment based on the hourly wage and working time. \newline
\colorbox{green!35}{\strut\textbf{Action}: \texttt{calculator}($12 \times 50 / 60$)} \newline
\textbf{Observation}: 10 \newline
\textbf{Thought}: The payment is 10 dollars. I need to save it into the database. \newline
\textbf{Action}: \texttt{update\_record}(\{"earning": 10\}) \newline
\textbf{Observation}: success \newline
\textbf{Answer}: Weng earned \$10, and the result has been saved into the database.

\end{tabularx}

\endgroup

\end{tcolorbox}

This demonstration teaches the model a tool-use pattern. The model learns to reason about the task, invoke an external tool when necessary, read the returned observation\footnote{We refer to the response returned by an external tool as an ``observation''.}, and continue the execution process.

% 3. SFT for Tool Use
\subsubsubsection{Supervised Tool-use Data}
To make tool use more reliable, we can further train LLMs on tool-use trajectories with supervised fine-tuning. In this setting, each training example contains a user task, available tool descriptions, and a demonstrated tool-use process. The model learns when to invoke a tool, which tool to select, how to construct valid arguments, and how to use the returned observation. For example, we can construct a training example as follows:
\begin{center}
    $\mathbf{x} =$ Task: Weng earns \$12 ... Tools: calculator(expression), ... , update\_record(json) \newline
    $\mathbf{y} =$ Thought: compute the payment; Action: calculator(12 x 50 / 60); ...; Answer: Weng earned \$10.
\end{center}

With such data, SFT teaches the model to imitate the demonstrated tool-use pattern. The model can learn the format of tool invocation and the role of observations in later generation.

A common way to construct such data is to manually annotate tool-use trajectories. However, this requires annotators to know which tool should be used, how to write valid arguments, and how to judge whether the returned observation is useful. This process is costly and difficult to scale to many tools and tasks.

A representative approach to addressing this data construction problem is Toolformer, which uses a self-supervised method to generate tool-use data \citep{schick-etal:toolformer}. Instead of relying on large-scale human annotations, Toolformer starts from only a few demonstrations for each API. It then lets the language model annotate a large text corpus with possible API calls. After executing these calls, Toolformer keeps only the API calls that help the model better predict future tokens. The filtered data is then used to fine-tune the model. Through this process, the model learns when to call a tool, what arguments to pass, and how to incorporate the tool result into later generation.

Beyond learning to use a small set of tools, later studies further scale tool-use training to larger and more realistic API spaces. API-Bank builds a benchmark and training set for tool-augmented LLMs, where models need to plan, retrieve, and call APIs in executable environments \citep{li-etal:apibank}. ToolLLM constructs ToolBench with more than 16,000 real-world APIs and uses automatically generated instruction-solution trajectories to train ToolLLaMA \citep{qin-etal:toolllm}. Gorilla focuses on API calling at scale and fine-tunes LLaMA-based models to generate accurate API calls. It also uses a document retriever to reduce API hallucination and adapt to changing API documents \citep{patil-etal:gorilla}.

% 4. RL for Tool Use
\subsubsubsection{Learning to Use Tools with Reinforcement Learning}

Although SFT can teach LLMs to imitate tool-use demonstrations, it is still limited by the coverage and quality of offline trajectories. The model mainly learns the tool-use patterns that appear in the supervised data. As a result, it may fail to generalize to unfamiliar tools, complex tool combinations, or new interaction patterns. More importantly, SFT does not directly optimize whether a tool call is necessary, whether the selected tool is appropriate, or whether the returned observation improves the final answer. This limitation becomes more serious in multi-step tool-use scenarios, \textit{where the model needs to decide when to call a tool, how to formulate the tool input, how to interpret the tool output, and when to stop using tools}.

To address this limitation, researchers have explored RL for tool use. Instead of only imitating fixed demonstrations, the model can interact with tools during rollout and receive feedback from task outcomes or tool execution results. Given a user task $q$ and a set of available tools $\mathcal{T}$, the model generates a tool-use trajectory:
\begin{eqnarray}
\tau = [s_1,u_1,o_1,\cdots,s_T,u_T,o_T,a]
\end{eqnarray}
where $s_t$ denotes the reasoning state at the $t$-th step, $u_t$ denotes the model action, $o_t$ denotes the observation returned by the tool or environment, and $a$ denotes the final answer. Here, $u_t$ can be either a natural language reasoning step, a tool invocation with arguments, or a final response. The goal of RL is to optimize the policy so that the generated trajectory receives a higher reward:
\begin{eqnarray}
\max_{\theta}\ 
\mathbb{E}_{\tau \sim \pi_{\theta}(\cdot|q,\mathcal{T})}
\left[
R(\tau)
\right]
\end{eqnarray}
In practice, this objective can be optimized with standard RL algorithms such as PPO or GRPO. Recent studies show that such outcome-driven optimization can help LLMs learn more adaptive tool-use behaviors, such as deciding when to search, when to execute code, and how to revise a tool call after receiving an error message \citep{jin-etal:searchr1,chen-etal:research,feng-etal:retool,singh-etal:artist}.

However, RL-based tool use also introduces several unique challenges. First, the action space is more complex than ordinary text generation. A tool-use action may involve tool selection, argument generation, execution timing, and response integration. A small error in any part can lead to tool failure. Second, the reward signal is often sparse and delayed. The environment may only provide a final success signal after the whole trajectory is completed, making it difficult to identify which tool call causes success or failure. Third, tool use may introduce additional costs. Unnecessary tool calls can increase latency and computation cost, while insufficient tool use may lead to incorrect answers. Therefore, the model needs to learn not only how to use tools, but also how to use them efficiently.

A key direction is to design more informative rewards for tool use. Instead of using only a final-answer reward, we can decompose the reward into multiple components:
\begin{eqnarray}
R(\tau) =
R_{\mathrm{ans}}(\tau)
+ \alpha R_{\mathrm{fmt}}(\tau)
+ \beta R_{\mathrm{exec}}(\tau)
+ \gamma R_{\mathrm{tool}}(\tau)
- \lambda C_{\mathrm{tool}}(\tau)
\end{eqnarray}
where $R_{\mathrm{ans}}$ measures final answer correctness, $R_{\mathrm{fmt}}$ measures whether the model follows the required tool-use format, $R_{\mathrm{exec}}$ measures whether tool calls can be successfully executed, and $R_{\mathrm{tool}}$ measures whether the model selects appropriate tools and passes valid arguments. $C_{\mathrm{tool}}$ denotes the cost of tool use, such as the number of tool calls, latency, or computational expense. The coefficients $\alpha$, $\beta$, $\gamma$, and $\lambda$ control the relative importance of different reward terms.

In application, this reward decomposition provides denser feedback than final-answer rewards. For example, if the model selects the correct tool but passes an invalid argument, the execution reward can penalize the invalid call while preserving credit for the correct tool selection. If the model reaches the correct answer but uses many unnecessary tool calls, the cost term can discourage inefficient behavior. ToolRL systematically studies this reward design problem and shows that fine-grained reward decomposition can make RL training more stable and effective for tool selection and tool application \citep{qian-etal:toolrl}.

Another direction is to use RL to improve strategic tool use. For example, ReTool first builds a cold-start model with code-augmented reasoning traces and then applies RL with real-time code execution. During rollout, the model interleaves natural language reasoning with code execution, observes execution results, and revises its subsequent reasoning. This allows the model to discover when and how to invoke a code interpreter based on outcome feedback rather than human-written rules \citep{feng-etal:retool}. Search-R1 and ReSearch follow a similar idea in retrieval-augmented reasoning. They train models to generate search queries during reasoning and use retrieved information to update later steps, rather than relying on fixed retrieval pipelines or hand-crafted search prompts \citep{jin-etal:searchr1,chen-etal:research}.

So far, we have introduced how RL can enhance the core capabilities of LLM-based agents, including planning and tool use. Since an LLM-based agent is inherently driven by an underlying LLM, its RL training can reuse the general LLM-based RL algorithms introduced in Section \ref{sec:example-using-rl-training-llms}. The main differences lie in the agentic setting. Compared with standard response optimization, agentic RL involves longer trajectories, environmental feedback, and intermediate decisions such as planning and tool invocation. Thus, the key challenges are not only the choice of RL algorithms, but also trajectory construction, reward design, and environment optimization.


\subsection{Environment Design and Scaling}

The previous subsections focus on optimizing the agent policy for planning and tool use. In this subsection, we turn to another important component of agentic RL: the environment that produces interaction experience. An environment is more than a testbed. It determines which actions the agent can perform, what observations it receives, how the underlying state changes after each action, and how task success is evaluated. More formally, we can represent an environment as
\begin{eqnarray}
e = (\mathcal{S},\mathcal{U},\mathcal{O},P_e,V_e)
\end{eqnarray}
where $\mathcal{S}$ is the state space, $\mathcal{U}$ is the set of available actions or tool invocations, and $\mathcal{O}$ is the observation space. The transition function $P_e$ determines how an action changes the environment state, while the verifier $V_e$ determines whether the resulting state satisfies the task objective. This formulation describes how the environment produces observations and feedback, rather than how the agent generates its actions.

The quality of an environment directly affects what an agent can learn. For example, an agent trained in a narrow environment may simply memorize specific APIs, task templates, or interaction patterns. An unreliable environment may return inconsistent observations or incorrect rewards, introducing noise into the learning process. Ideally, an effective environment should provide diverse tasks, executable interactions, reliable state transitions, and verifiable outcomes.

To encourage generalization, we can train the agent over a distribution of environments rather than within a single fixed environment:
\begin{eqnarray}
\theta^{*} = 
\arg\max_{\theta}
\mathbb{E}_{e\sim p(\mathcal{E}), q\sim p_e(\mathcal{Q})}
\mathbb{E}_{\tau\sim\pi_{\theta}(\cdot\mid q,e)}
\left[
R_e(\tau)
\right]
\end{eqnarray}
where $p(\mathcal{E})$ denotes the environment distribution, and $p_e(\mathcal{Q})$ denotes the task distribution within environment $e$. The interaction trajectory $\tau$ follows the definition introduced in the previous subsections. This objective shows that agent performance depends not only on policy optimization, but also on the coverage and quality of the environments used for training.

However, constructing high-quality environments manually is expensive and difficult to scale. Real-world systems may be inaccessible and costly. Purely language-based simulations are easier to build, but they may generate inconsistent state transitions \citep{li-etal:word2world}. Thus, recent studies explore procedural and programmatic environment synthesis, where executable programs and structured states are used to provide a scalable and reliable interaction experience.

\begin{figure}[t!]
\centering
\resizebox{\linewidth}{!}{
\input{section6/Figures/environment-design}}
\caption{Overview of environment synthesis and interaction for agentic RL.}
\label{fig:agent-environment-scaling}
\end{figure}

This process is illustrated in Figure~\ref{fig:agent-environment-scaling}. It typically consists of the following steps:
\begin{itemize}
\item \textbf{Environment Specification.} We first define the state schema\footnote{A state schema specifies the structured variables used to represent the current status of an environment, together with their types and possible values. For example, in a travel-booking environment, it may define fields for available flights, user preferences, and existing reservations.}, available tools and APIs, and transition rules of the environment.
\item \textbf{Programmatic Synthesis.} We then implement the environment as an executable system. This process typically converts the structured specification into program code, database tables, and callable tool interfaces. 
\item \textbf{Scenario Generation.} Based on the environment specification, we generate diverse initial states and user tasks. The tasks should cover different goals, constraints, and difficulty levels. For example, in a shopping environment, a simple task may ask the agent to purchase a single item, while a more difficult task may require it to compare multiple products.
\item \textbf{Agent--Environment Interaction.} The agent interacts with the generated environment through multiple rounds of actions and observations. At each step, the agent selects a tool and provides the required arguments. The environment executes the action and returns an observation.
\item \textbf{Outcome Verification.} After the interaction is completed, a verifier checks whether the task has been successfully solved. The verifier may inspect tool execution results, database states, or other structured records. 
\item \textbf{Agent Learning.} Finally, the verified trajectories are used to improve the agent. On the one hand, successful trajectories can serve as demonstrations for SFT, while task-level or step-level feedback can support RL training. On the other hand, failed interactions are also valuable because they reveal weaknesses in the current policy and gaps in the task distribution. This can guide the generation of targeted scenarios that exercise the agent's weak capabilities and provide more informative experience for subsequent training \citep{chen-etal:failure-attribution,sun-etal:learning-from-failure}. More broadly, this process can be viewed as learning from agentic experience, which we will discuss in Section~\ref{sec:learning_from_agentic_experience}.
\end{itemize}


Despite this progress, several important questions remain. First, how can we scale both the number and diversity of environments? Second, how can we ensure that generated environments are executable and verifiable? Third, how can agents generalize across heterogeneous and previously unseen environments?

One direction focuses on scaling environment diversity. RandomWorld procedurally generates executable tools and compositional tasks for tool-use agents \citep{sullivan-etal:randomworld}. Unlike static datasets that contain only predefined trajectories, these generated environments support online execution and allow agents to explore different tool combinations during training. AgentScaler further constructs heterogeneous simulated environments and separates the learning of general function-calling behaviors from domain-specific adaptation \citep{fang-etal:agentscaler}. At a larger scale, DeepSeek-V3.2 synthesizes diverse environments and complex tasks for agentic post-training, showing that broad interaction experience is important for long-tail generalization \citep{deepseek-ai:deepseekv32}.

Another direction focuses on executability. EnvScaler first constructs environment skeletons that specify state schemas, available tools, and interaction rules. It then generates multiple task scenarios within each environment \citep{song-etal:envscaler}. Agent World Model follows a similar idea by implementing synthetic environments with executable code and database-backed states \citep{wang-etal:awm}. Compared with purely language-based simulations, these programmatic environments produce more consistent state transitions and generate tool outputs through actual execution.

Verifiability is equally important because RL requires reliable feedback. Instead of evaluating only the textual final response, an environment can check whether the agent has produced the desired state change. Given the final environment state $s_T$ and the task goal $g$, we can define a state-based reward as
\begin{eqnarray}
R_e(\tau)=V_e(s_T,g)
\end{eqnarray}
where $V_e(\cdot)$ may be implemented using executable tests, database queries, or rule-based validators. For example, in a database operation task, the verifier can directly check whether the required record has been inserted correctly. This provides more reliable feedback than comparing the generated answer with a reference text. Both EnvScaler and Agent World Model use such mechanisms to connect agent actions with observable state changes.

Scaling environments also introduces substantial heterogeneity. Different environments may vary in task difficulty, available tools, and interaction length. These differences can make RL training unstable and cause the agent to overfit to frequently sampled or easier environments. AutoForge addresses this issue by synthesizing difficult but verifiable tasks and estimating learning signals at the environment level \citep{cai-etal:autoforge}. This design reduces the influence of unstable simulated interactions and improves training across heterogeneous environments.


\subsection{Learning from Agentic Experience}
\label{sec:learning_from_agentic_experience}

AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.

The first approach is \textbf{memory management} though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}, where the agent improves its behavior by incorporating previous experience into the current context. In this way, one example of learning from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent. 

The second approach is \textbf{skill optimization}. One challenge of using memory to learn from experience is scalability. As agent experience continuously accumulates, the memory size will keep growing. Moreover, a large amount of noisy and redundant information may also be introduced into the memory. As a result, As a result, efficiently storing and retrieving relevant information becomes increasingly difficult. Instead of storing individual experiences in memory, skill aims to identify common patterns across experiences and transform them into abstract behaviors. These skills provide a more compact representation of experience and allow the agent to transfer experience across different tasks. Similar to memory optimization, we can also update and refine the skills of agents to achieve learning from experience.

The third approach is to learn from experience via \textbf{trajectory refinement}. From this perspective, test-time scaling in agents can also be viewed as a form of learning from experience. Instead of updating the model parameters, the agent improves its current solution by leveraging immediate feedback during task execution. For example, the agent can use tool execution results or environment feedback to identify mistakes and refine its trajectory. Through repeated attempts and continuous refinement, the agent can gradually improve its decision-making process and achieve better task performance.

The three approaches mentioned above can be implemented through various techniques. Since this paper focuses on RL, we will introduce RL-based methods for these approaches in detail in the following sections. It is worth noting that, although these approaches do not always rely on traditional parametric updates, they can still be formulated under the RL paradigm described in Eqs.~\ref{eq:tau-define} and~\ref{eq:reward-optimization-for-agent}, where agents improve their behaviors through sequential interaction and feedback from experience.


\subsubsection{Memory Management}

Memory management is a direct way for agents to learn from agentic experience. During interaction with an environment, an agent may observe useful information. If this information is discarded after the current task, the agent must solve similar problems from scratch in the future. Therefore, the goal of agent memory is to store useful information from previous interactions and retrieve it when needed. As illustrated in Figure~\ref{fig:memory-and-retrieve}, a typical memory system usually consists of three main stages \citep{chhikara-etal:mem0}:
\begin{itemize}
    \item \textbf{Memory Retrieval.} The agent retrieves relevant memories from the memory bank to support the current interaction. The retrieved information can take various forms depending on how the memory bank is constructed. For example, the memory bank may contain successful trajectories from similar tasks, which can provide reusable solutions for current decision. It may also store user-specific preferences, such as frequently selected hotels or preferred travel styles. 
    \item \textbf{Memory Extraction.} After the agent completes the current task based on retrieved information, useful information can be extracted from the interaction process and stored for future use. For example, For example, if a user says that they are vegetarian, we can extract a memory such as ``the user prefers vegetarian food''. This process can be performed by the agent itself or by other agents.
    \item \textbf{Memory Update.} The agent updates the memory bank by integrating the newly extracted information with existing memories. A straightforward approach is to store all extracted information. However, this strategy is impractical because the amount of stored information continuously grows as the agent operates over time, making memory retrieval increasingly inefficient and introducing a large amount of redundant information. In contrast, a more advanced approach is to selectively update the memory. Specifically, the memory system can formulate memory maintenance as an operation selection problem. Given newly extracted information, a \textit{memory manager} selects one of several operations: $a^m \in \{\texttt{ADD}, \texttt{UPDATE}, \texttt{DELETE}, \texttt{NOOP}\}$, where \texttt{ADD} creates a new memory entry, \texttt{UPDATE} modifies an existing memory with newly observed information, \texttt{DELETE} removes outdated or contradictory memories, and \texttt{NOOP} keeps the memory bank unchanged. We can typically achieve this memory manager by prompting an LLM to select appropriate operations based on the current interaction and existing memories.
\end{itemize}


\begin{figure}[!t]
    \centering
    \resizebox{\linewidth}{!}{
    \input{section6/Figures/memory-and-retrieval.tex}}
    \caption{
    An overview of the memory system.
    }
    \label{fig:memory-and-retrieve}
\end{figure}


In learning from experience, it is easy to observe that the performance of memory-based learning heavily depends on the accuracy of the memory manager. Although prompting an LLM can enable it to select memory operations, such memory systems still rely largely on the LLM's in-context decision-making ability or manually designed rules. As a result, they may struggle with complex memory updates and make incorrect decisions. For example, as shown in \cite{yan-etal:memory-r1}'s work, when a user first says ``I adopted a dog named Buddy'' and later adds ``I adopted another dog named Scout'', a vanilla memory system may incorrectly interpret the new information as a contradiction and perform a \texttt{DELETE}+\texttt{ADD} operation, overwriting the original memory. In contrast, a trained memory manager can recognize that the two statements are complementary and perform an \texttt{UPDATE} operation to consolidate the information into a more complete memory: ``Andrew adopted two dogs, Buddy and Scout''. 


\begin{figure}[!t]
    \centering
    \resizebox{\linewidth}{!}{
    \input{section6/Figures/reinfroced-memory-manager.tex}}
    \caption{Overview of training a memory manager with RL.}
    \label{fig:reinforced-memory}
\end{figure}


One promising approach to improve memory management is to optimize memory operations with RL \citep{yan-etal:memory-r1}. The key idea is to make memory management itself a learnable decision-making process. Given an extracted memory $x^\mathrm{mem}$ and an existing memory bank $\mathcal{M}_\mathrm{old}$, the memory manager acts as a policy that selects a memory operation and generates the updated memory content:
\begin{eqnarray}
    (o,m') \sim \pi_{\theta}(\cdot \mid x^\mathrm{mem}, \mathcal{M}_\mathrm{old})
\end{eqnarray}
where $o$ denotes the selected memory operation from $a^m$ and $m'$ denotes the updated memory content. After applying the selected operation, the updated memory bank is provided to the agent for downstream task solving. By optimizing the memory manager with task-level feedback, the agent can learn when and how to update its memory based on the usefulness of stored experience. Figure~\ref{fig:reinforced-memory} illustrates the training process of a memory manager with RL.

During optimization, the reward is defined according to the final task performance. If the updated memory helps the agent produce a correct answer, the memory manager receives a positive reward; otherwise, it receives a lower reward. A simple reward function can be defined as:
\begin{eqnarray}
R_{\text{answer}} = \text{EM}(y_{\text{pred}}, y_{\text{gold}}) 
\end{eqnarray}
where $\text{EM}(\cdot)$ denotes the matching function, $y_{\text{pred}}$ denotes the predicted answer, and $y_{\text{gold}}$ denotes the ground-truth answer. This design avoids the need to manually annotate individual memory operations. Instead, the memory manager learns which operations are beneficial by directly optimizing their impact on downstream task performance.

After optimizing the memory manager, the agent can dynamically maintain its memory during future interactions. However, learning effective memory operations is only the first step toward experience-driven agent improvement. A key challenge is how to evaluate the long-term utility of accumulated memories. Existing memory optimization methods typically define rewards based on downstream task performance, i.e., whether the retrieved memories help the agent solve the current task. However, the usefulness of a memory is not always reflected immediately \citep{xu-etal:a-mem}. For example, a failure case collected from one interaction may not improve the current response, but it can help the agent avoid similar mistakes in future tasks. This indicates that memory can serve as an important component of agent learning, beyond simply storing past information.

Beyond optimizing memory update operations, another important question is how to construct and organize memories from accumulated experiences. Instead of treating memory construction as a fixed preprocessing step, we can make an agent to learn effective memory construction strategies through RL. In this way, a straightforward approach is to formulate memory construction as a sequential decision-making process, where the agent learns how to select, organize, and transform interaction experiences into useful memories \citep{wang-etal:mem}.

% In this way, instead of treating memory updates as isolated operations for individual tasks, we can view memory management as an ongoing learning process, where agents gradually organize past experiences into reusable knowledge. As a result, we can consider memory as a core component of agent learning rather than a passive storage module \citep{xu-etal:a-mem}. 


\subsubsection{Skill Optimization}

Skill represents a higher-level abstraction of agentic experience. Different from memory, which focuses on storing useful information extracted from previous interactions, skill aims to summarize recurring patterns across multiple experiences and transform them into reusable capabilities. Specifically, a skill can be viewed as a structured instruction package that describes when the skill should be invoked, what workflow it should follow, which tools are required, and how the final outcome should be verified. For example, a reasoning-oriented skill may be represented as follows:

\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\parindent}{0pt}
\setlength{\leftskip}{1.0em}
\setlength{\rightskip}{1.0em}
\textbf{Name: Math-Logic-Reasoning}\par
\textit{Description}: Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Use when a task requires multi-step arithmetic, symbolic reasoning, calculator-backed verification, table analysis, optimization, scheduling or allocation constraints, Pareto trade-offs, proof checking, or translating word problems into equations, code, formulas, or executable checks.\par

\vspace{2mm}
\uline{Math Logic Reasoning}\par
\vspace{0.2em}
Use this skill for reasoning tasks where a wrong intermediate assumption can silently break the final answer. Prefer a compact formal model plus executable verification over unsupported mental arithmetic.\par
\vspace{2mm}
\uline{Workflow}\par
1. Restate the target quantity, decision, or claim.\par
2. Extract givens, units, constraints, and hidden assumptions.\par
3. Convert the problem into equations, inequalities, tables, spreadsheet formulas, graphs, search spaces, constraint satisfaction problems, or short verification scripts.\par
$\cdots$ \par
\vspace{2mm}
\uline{Tool Choices}\par
1. Use Python as a calculator for multi-step arithmetic, combinatorics, simulations, optimization search, or exact rational checks.\par
2. Use Fraction, Decimal, or sympy when exactness matters.\par
3. Use pandas for complex tables, joins, grouped summaries, rankings, or data-cleaning logic.\par
$\cdots$ \par
\vspace{2mm}
\uline{Verification Checklist}\par
1. Recalculate the final numeric answer using a second method when feasible.\par
2. Test boundary cases: zero, negative values, equality limits, duplicate entries, and empty sets.\par
3. Confirm units after every transformation.\par
$\cdots$\par
\vspace{2mm}
\uline{Output Style}\par
Give the answer first when possible. Then include a concise derivation, the checked constraints, and any caveats.

\endgroup

\end{tcolorbox}

In practice, the agent is provided with descriptions of all available skills in the prompt and learns to select appropriate skills based on the current task. These selected skills serve as high-level behavioral templates that guide the agent's planning and execution process. For example,
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\parindent}{0pt}
\setlength{\leftskip}{1.0em}
\setlength{\rightskip}{1.0em}
\textbf{Available Skills:} \par
\uline{Skill 1: Web Search} \par
This skill enables the agent to retrieve and synthesize external information when the task requires up-to-date or domain-specific knowledge $\cdots$ \par
\uline{Skill 2: Math-Logic-Reasoning} \par
Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Use when a task requires multi-step arithmetic, symbolic reasoning $\cdots$ \par
\{More skill descriptions\} \par
\textbf{User Query:} Solve the equation $2x+5=13$. \par
\par
\colorbox{green!35}{\strut\textbf{Selected Skill:} \uline{Math-Logic-Reasoning}} \par
\endgroup
\end{tcolorbox}






\subsubsection{Trajectory Refinement}

Learning from early experience. 

Using the feedback to refine the trajectories.


Feedback is very important. How to obtain feedback from convention?

OpenClaw-RL: Train Any Agent Simply by Talking


% \subsection{Self-Evolving Agents}


% large-scale environment construct










