\bibitem[Sun et~al.(2024)Sun, Liu, Bair, and Kolter]{sun-etal:2023simple}
Mingjie Sun, Zhuang Liu, Anna Bair, and J.~Zico Kolter.
\newblock A simple and effective pruning approach for large language models.
...
...
@@ -541,6 +566,11 @@ Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi C
\newblock In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), \emph{Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023}, 2023{\natexlab{d}}.
@@ -238,16 +238,65 @@ In application, this reward decomposition provides denser feedback than final-an
Another direction is to use RL to improve strategic tool use. For example, ReTool first builds a cold-start model with code-augmented reasoning traces and then applies RL with real-time code execution. During rollout, the model interleaves natural language reasoning with code execution, observes execution results, and revises its subsequent reasoning. This allows the model to discover when and how to invoke a code interpreter based on outcome feedback rather than human-written rules \citep{feng-etal:retool}. Search-R1 and ReSearch follow a similar idea in retrieval-augmented reasoning. They train models to generate search queries during reasoning and use retrieved information to update later steps, rather than relying on fixed retrieval pipelines or hand-crafted search prompts \citep{jin-etal:searchr1,chen-etal:research}.
So far, we have introduced how RL can enhance the core capabilities of LLM-based agents, including planning and tool use. Since an LLM-based agent is inherently driven by an underlying LLM, its RL training can reuse the general LLM-based RL algorithms introduced in Section \ref{sec:example-using-rl-training-llms}. The main differences lie in the agentic setting. Compared with standard response optimization, agentic RL involves longer trajectories, environmental feedback, and intermediate decisions such as planning and tool invocation. Thus, the key challenges are not only the choice of RL algorithms, but also trajectory construction, reward design, and environment optimization.
\subsubsection{Improved Environments}
\subsection{Environment Design and Scaling}
The previous subsections focus on optimizing the agent policy for planning and tool use. In this subsection, we turn to another important component of agentic RL: the environment that produces interaction experience. An environment is more than a testbed. It determines which actions the agent can perform, what observations it receives, how the underlying state changes after each action, and how task success is evaluated. More formally, we can represent an environment as
\begin{eqnarray}
e = (\mathcal{S},\mathcal{U},\mathcal{O},P_e,V_e)
\end{eqnarray}
where $\mathcal{S}$ is the state space, $\mathcal{U}$ is the set of available actions or tool invocations, and $\mathcal{O}$ is the observation space. The transition function $P_e$ determines how an action changes the environment state, while the verifier $V_e$ determines whether the resulting state satisfies the task objective. This formulation describes how the environment produces observations and feedback, rather than how the agent generates its actions.
The quality of an environment directly affects what an agent can learn. For example, an agent trained in a narrow environment may simply memorize specific APIs, task templates, or interaction patterns. An unreliable environment may return inconsistent observations or incorrect rewards, introducing noise into the learning process. Ideally, an effective environment should provide diverse tasks, executable interactions, reliable state transitions, and verifiable outcomes.
To encourage generalization, we can train the agent over a distribution of environments rather than within a single fixed environment:
where $p(\mathcal{E})$ denotes the environment distribution, and $p_e(\mathcal{Q})$ denotes the task distribution within environment $e$. The interaction trajectory $\tau$ follows the definition introduced in the previous subsections. This objective shows that agent performance depends not only on policy optimization, but also on the coverage and quality of the environments used for training.
However, constructing high-quality environments manually is expensive and difficult to scale. Real-world systems may be inaccessible, costly, or unsafe for large-scale exploration. Purely language-based simulations are easier to build, but they may generate inconsistent state transitions, invalid tool outputs, or unreliable evaluations. Thus, recent studies explore procedural and programmatic environment synthesis, where executable programs and structured states are used to provide a scalable and reliable interaction experience.
\begin{figure}[t!]
\centering
\resizebox{\linewidth}{!}{
\input{section6/Figures/environment-design}}
\caption{Overview of environment synthesis and interaction for agentic reinforcement learning.}
\label{fig:agent-environment-scaling}
\end{figure}
This process is illustrated in Figure~\ref{fig:agent-environment-scaling}. It typically consists of the following steps:
\begin{itemize}
\item\textbf{Environment Specification.} We first define the state schema\footnote{A state schema specifies the structured variables used to represent the current status of an environment, together with their types and possible values. For example, in a travel-booking environment, it may define fields for available flights, user preferences, and existing reservations.}, available tools and APIs, and transition rules of the environment.
\item\textbf{Programmatic Synthesis.} We then implement the environment as an executable system. This process typically converts the structured specification into program code, database tables, and callable tool interfaces.
\item\textbf{Scenario Generation.} Based on the environment specification, we generate diverse initial states and user tasks. The tasks should cover different goals, constraints, and difficulty levels. For example, in a shopping environment, a simple task may ask the agent to purchase a single item, while a more difficult task may require it to compare multiple products.
\item\textbf{Agent--Environment Interaction.} The agent interacts with the generated environment through multiple rounds of actions and observations. At each step, the agent selects a tool and provides the required arguments. The environment executes the action and returns an observation.
\item\textbf{Outcome Verification.} After the interaction is completed, a verifier checks whether the task has been successfully solved. The verifier may inspect tool execution results, database states, or other structured records.
\item\textbf{Agent Learning.} Finally, the verified trajectories are used to improve the agent. Successful trajectories can serve as demonstrations for SFT, while task-level or step-level feedback can support RL training. Note that failed interactions are also useful because they reveal weaknesses in the current policy or gaps in the task distribution. For example, repeated failures on invalid API arguments may motivate the generation of additional scenarios involving parameter constraints. In this way, learning results can further guide scenario generation and form an iterative environment--agent improvement loop.
\end{itemize}
So far, we have introduced how RL can enhance the core capabilities of LLM-based agents, including planning and tool use. Since an LLM-based agent is inherently driven by an underlying LLM, its RL training can reuse the general LLM-based RL algorithms introduced in Section \ref{sec:example-using-rl-training-llms}. The main differences lie in the agentic setting. Compared with standard response optimization, agentic RL involves longer trajectories, environmental feedback, and intermediate decisions such as planning and tool invocation. Thus, the key challenges are not only the choice of RL algorithms, but also trajectory construction, reward design, and environment optimization.
Despite this progress, several important questions remain. First, how can we scale both the number and diversity of environments? Second, how can we ensure that generated environments are executable and verifiable? Third, how can agents generalize across heterogeneous and previously unseen environments?
One direction focuses on scaling environment diversity. RandomWorld procedurally generates executable tools and compositional tasks for tool-use agents \citep{sullivan-etal:randomworld}. Unlike static datasets that contain only predefined trajectories, these generated environments support online execution and allow agents to explore different tool combinations during training. AgentScaler further constructs heterogeneous simulated environments and separates the learning of general function-calling behaviors from domain-specific adaptation \citep{fang-etal:agentscaler}. At a larger scale, DeepSeek-V3.2 synthesizes diverse environments and complex tasks for agentic post-training, showing that broad interaction experience is important for long-tail generalization \citep{deepseek-ai:deepseekv32}.
Another direction focuses on executability. EnvScaler first constructs environment skeletons that specify state schemas, available tools, and interaction rules. It then generates multiple task scenarios within each environment \citep{song-etal:envscaler}. Agent World Model follows a similar idea by implementing synthetic environments with executable code and database-backed states \citep{wang-etal:awm}. Compared with purely language-based simulations, these programmatic environments produce more consistent state transitions and generate tool outputs through actual execution.
Verifiability is equally important because RL requires reliable feedback. Instead of evaluating only the textual final response, an environment can check whether the agent has produced the desired state change. Given the final environment state $s_T$ and the task goal $g$, we can define a state-based reward as
\begin{eqnarray}
R_e(\tau)=V_e(s_T,g)
\end{eqnarray}
where $V_e(\cdot)$ may be implemented using executable tests, database queries, or rule-based validators. For example, in a database operation task, the verifier can directly check whether the required record has been inserted correctly. This provides more reliable feedback than comparing the generated answer with a reference text. Both EnvScaler and Agent World Model use such mechanisms to connect agent actions with observable state changes.
Scaling environments also introduces substantial heterogeneity. Different environments may vary in task difficulty, available tools, and interaction length. These differences can make RL training unstable and cause the agent to overfit to frequently sampled or easier environments. AutoForge addresses this issue by synthesizing difficult but verifiable tasks and estimating learning signals at the environment level \citep{cai-etal:autoforge}. This design reduces the influence of unstable simulated interactions and improves training across heterogeneous environments.
\subsection{Learning from Agentic Experience}
...
...
@@ -256,6 +305,7 @@ So far, we have introduced how RL can enhance the core capabilities of LLM-based
