Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
Using RL to train agents, often referred to as \textit{agentic RL}, is not a new concept. In classical machine learning, agentic RL typically involves training a domain-specific decision model from scratch. For example, in tasks such as playing complex games like Go \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} or controlling robotic locomotion \citep{lee-etal:lee2024learning}, the goal is to explore a structured environment and learn an optimal policy starting from random initialization.
With the rise of LLMs as core components of autonomous systems, RL has been increasingly used to adapt these pretrained models for sequential decision-making in dynamic and open-ended environments. In this setting, agentic RL addresses a key limitation of LLMs: although they encode extensive world knowledge, their outputs are not inherently aligned with long-horizon task requirements. For example, a supervised fine-tuned LLM may correctly generate a script for calling an external API, but it cannot reliably self-correct when the environment returns unexpected errors. This creates new challenges for enabling LLM-based agents to plan autonomously, adapt to environmental feedback, and safely improve their strategies through iterative interaction.
With the rise of LLMs as core components of autonomous systems, RL has been increasingly used to adapt these models for sequential decision-making in dynamic and open-ended environments. In this setting, agentic RL addresses a key limitation of LLMs: although they encode extensive world knowledge, their outputs are not inherently aligned with long-horizon task requirements. For example, a supervised fine-tuned LLM may correctly generate a script for calling an external API, but it cannot reliably self-correct when the environment returns unexpected errors. This creates new challenges for enabling LLM-based agents to plan autonomously, adapt to environmental feedback, and safely improve their strategies through iterative interaction.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory updates, skill optimization, and trajectory refinement without altering the internal model weights. It is worth noting that while these approaches depart from traditional parametric updates, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL builds and enhances core agentic capabilities, specifically focusing on long-horizon planning and tool-integrated reasoning. Then we consider training-free methods that apply RL principles to optimize external modules such as memory updates, skill optimization, and trajectory refinement without altering the internal model weights. It is worth noting that while these approaches depart from traditional parametric updates, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making. Finally, we introduce the crucial role of designing better environments, which serve as the foundational testbeds for training agents.
...
@@ -13,18 +13,17 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
...
@@ -13,18 +13,17 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
\subsection{Building Agent Capabilities}
\subsection{Building Agent Capabilities}
One feature of LLM-based agents is their capacity to interpret user instructions, autonomously formulate a step-by-step plan to resolve the given task, and systematically execute those steps. In applications, this planning process inherently involves coordinating interactions with external environments and determining appropriate tool invocations. The agent then relies on this structured plan as a blueprint to guide its execution trajectory over multiple turns.
One feature of LLM-based agents is their capacity to interpret user instructions, autonomously formulate a step-by-step plan to resolve the given task, and systematically execute those steps. In applications, this planning process inherently involves coordinating interactions with external environments and determining appropriate tool use. The agent then relies on this structured plan as a blueprint to guide its execution trajectory over multiple turns.
To accomplish this process, an LLM-based agent relies on several complementary capabilities. Some of these capabilities are already inherited from pretrained LLMs. Recent advances in large-scale pretraining and post-training have significantly improved LLMs in instruction following, reasoning, code generation, and general world knowledge. These capabilities provide a strong foundation for agentic behavior, enabling LLMs to understand complex user requests and generate coherent intermediate reasoning steps without explicit interaction with the environment. The rapid improvement of these intrinsic capabilities has been a key factor driving the emergence of LLM-based agents, making it increasingly feasible for a single model to serve as the reasoning and decision-making core of an autonomous system.
To accomplish this process, an LLM-based agent relies on several complementary capabilities. Some of these capabilities are already inherited from LLMs. Recent advances in large-scale pretraining and post-training have significantly improved LLMs in instruction following, reasoning, code generation, and general world knowledge. These capabilities provide a strong foundation for agentic behavior, enabling LLMs to understand complex user requests and generate coherent intermediate reasoning steps without explicit interaction with the real-world environment. The rapid improvement of these intrinsic capabilities has been a key factor driving the emergence of LLM-based agents, making it increasingly feasible for a single model to serve as the reasoning and decision-making core of an autonomous system.
However, autonomous agents require capabilities that go beyond static reasoning. They must make sequential decisions over long horizons, coordinate multiple tool invocations, adapt to unexpected environmental feedback, and recover from execution failures. These capabilities cannot be effectively learned from static supervision alone because their quality depends on the outcomes of complete interaction trajectories rather than individual outputs. Reinforcement learning naturally addresses this challenge by optimizing behaviors through trial-and-error interaction, enabling agents to improve decisions based on delayed rewards and environmental feedback. As a result, RL complements the inherent reasoning abilities of LLMs by strengthening their decision-making and execution capabilities in dynamic environments.
However, autonomous agents require capabilities that go beyond static reasoning. They must make sequential decisions over long horizons, coordinate multiple tool use, adapt to unexpected environmental feedback, and recover from execution failures. These capabilities cannot be effectively learned from static supervision alone because their quality depends on the outcomes of complete interaction trajectories rather than individual outputs. RL naturally addresses this challenge by optimizing behaviors through trial-and-error interaction, enabling agents to improve decisions based on delayed rewards and environmental feedback \citep{singh-etal:singhagentic}. As a result, RL complements the inherent reasoning abilities of LLMs by strengthening their decision-making and execution capabilities in dynamic environments. In the following sections, we discuss two representative capabilities that have become the primary focus of agentic RL: planning and tool use.
In the following sections, we discuss two representative capabilities that have become the primary focus of agentic RL: planning and tool use.
\subsubsection{Planning}
\subsubsection{Planning}
For example, given a specific task description, the model will output an actionable plan and tool use as follows:
For example, given a specific task description, the model will output an actionable plan and tool use as follows:
