@@ -13,11 +13,11 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
...
@@ -13,11 +13,11 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
\subsection{Building Agent Capabilities}
\subsection{Building Agent Capabilities}
One feature of LLM-based agents is their capacity to interpret user instructions, autonomously formulate a step-by-step plan to resolve the given task, and systematically execute those steps. In applications, this planning process inherently involves coordinating interactions with external environments and determining appropriate tool use. The agent then relies on this structured plan as a blueprint to guide its execution trajectory over multiple turns.
LLM-based agents extend LLMs from passive response generators into interactive decision-making systems. Rather than producing a single answer, an agent must repeatedly observe its environment, decide what to do next, and execute an action. This interaction loop allows the agent to solve tasks that require multiple steps, external resources, and adaptation during execution. The underlying LLM already provides several important capabilities, including instruction understanding, reasoning, code generation, and broad world knowledge. These capabilities allow the agent to interpret complex goals and propose plausible intermediate steps. However, they do not automatically guarantee reliable behavior in an interactive environment. An agent must also determine how to decompose a task, choose among alternative actions, coordinate external tools, and recover when an earlier decision leads to an unexpected result.
To accomplish this process, an LLM-based agent relies on several complementary capabilities. Some of these capabilities are already inherited from LLMs. Recent advances in large-scale pretraining and post-training have significantly improved LLMs in instruction following, reasoning, code generation, and general world knowledge. These capabilities provide a strong foundation for agentic behavior, enabling LLMs to understand complex user requests and generate coherent intermediate reasoning steps without explicit interaction with the real-world environment. The rapid improvement of these intrinsic capabilities has been a key factor driving the emergence of LLM-based agents, making it increasingly feasible for a single model to serve as the reasoning and decision-making core of an autonomous system.
These agentic capabilities are difficult to learn from static input-output pairs alone. The quality of an individual decision often depends on its effect on later interactions and the final task outcome. A locally plausible action may lead to failure several steps later, while an initially unsuccessful attempt may provide useful information for a better strategy. RL is well suited to this setting because it optimizes complete interaction trajectories through environmental feedback and delayed rewards \citep{singh-etal:singhagentic,feng-etal:retool}. It enables the agent to explore different behaviors, compare their consequences, and gradually improve its decision-making policy.
However, autonomous agents require capabilities that go beyond static reasoning. They must make sequential decisions over long horizons, coordinate multiple tool use, adapt to unexpected environmental feedback, and recover from execution failures. These capabilities cannot be effectively learned from static supervision alone because their quality depends on the outcomes of complete interaction trajectories rather than individual outputs. RL naturally addresses this challenge by optimizing behaviors through trial-and-error interaction, enabling agents to improve decisions based on delayed rewards and environmental feedback \citep{singh-etal:singhagentic}. As a result, RL complements the inherent reasoning abilities of LLMs by strengthening their decision-making and execution capabilities in dynamic environments. In the following sections, we discuss two core capabilities that have become the primary focus of agentic RL: planning and tool use.
In this subsection, we focus on two core capabilities in agentic RL: \textit{planning} and \textit{tool use}. Planning allows the agent to organize a complex goal into a sequence of executable steps. Tool use allows it to access external information, perform reliable computation, and take actions beyond the capabilities stored in the model parameters. Together, these capabilities form the basis for effective interaction in given environments.
