@@ -26,7 +26,7 @@ This formulation is equivalent to minimizing the cross-entropy loss. In the foll
\subsection{Fundamentals of Reinforcement Learning}
RL, supported by a well-established theoretical framework, has been widely applied across a broad range of domains. In particular, with the rapid advancement of LLMs, RL has become a standard approach in the post-training stage. Before exploring the interplay between RL and LLMs, we first provide a brief overview of deep reinforcement learning in this subsection. We then introduce the general formulation of RL, along with key terminologies and notations that are essential for understanding RL.
\subsubsection{General Framework of Reinforcement Learning}
\subsubsection{Markov Decision Process}
The general framework of RL is illustrated in Figure~\ref{fig:rl_framework}. An RL system primarily consists of two components: an agent and an environment. At each timestep $t$, the agent observes a state $s_t$ from the environment and selects an action $a_t$ according to a policy $\pi$, which is typically parameterized by a neural network. After executing the action, the environment transitions to a new state $s_{t+1}$ and returns a reward $r_t$ corresponding to the taken action.
\section{Reinforcement Learning for LLM-based Agents}
Using RL to train agents is a well-established practice in traditional machine learning. In this context, the agent has consistently served as the central decision-making entity interacting with an environment. However, the advent of LLMs has fundamentally transformed the motivations, application scenarios, and action spaces of RL-driven agents.
Using RL to train agents is not a new concept, but its focus and application have evolved significantly over time. In classical RL, an agent is typically a domain-specific decision model trained entirely from scratch. For example, in tasks ranging from playing complex games (like GO) \citep{mnih-etal:mnih2013playing,silver-etal:silver2017mastering} to controlling robotic locomotion \citep{lee-etal:lee2024learning}, the primary objective is to comprehensively explore a highly structured environment and find an optimal policy from a randomly initialized state.
In classical RL, an agent is typically a domain-specific decision model trained entirely from scratch. For example, in tasks such as mastering Atari games \citep{mnih-etal:mnih2013playing}, playing Go \citep{silver-etal:silver2017mastering}, or controlling robotic locomotion \citep{lee-etal:lee2024learning}, researchers conventionally construct a relatively small-scale policy network within a highly structured simulated environment characterized by predefined state and action spaces. Through millions of trial-and-error interactions, the agent iteratively learns an optimal policy that maximizes expected rewards. Ultimately, the primary objective in these settings is to comprehensively explore the environment and synthesize this optimal policy from a randomly initialized state.
As LLMs become increasingly central to autonomous systems, the application of RL is more broadly used to empower these pre-trained models to make sequential decisions in dynamic, open-ended environments. In this context, the problem that RL addresses is that while LLMs possess vast world knowledge, their outputs may not naturally align with the demands of long-horizon tasks. For example, a supervised fine-tuned LLM might successfully generate a script to call an external API, but it cannot intrinsically self-correct if the environment returns an unpredictable error. This poses new challenges in ensuring that LLM-based agents can autonomously plan, adapt to environmental feedback, and safely refine their strategies through continuous trial and error.
\subsection{Why use Reinforcement Learning?}
In this section, we focus on the emerging paradigm of agentic RL for LLM-based agents. We begin by discussing how RL enhances core agentic capabilities, such as planning and tool use. Then we consider the integration of RL within advanced agent systems, such as OpenClaw, to highlight the cutting edge of LLM agent research. It is worth noting that while some of these advanced techniques do not update the model's parameters, i.e., leveraging RL purely for dynamic memory management, which departs from traditional parametric RL, they fundamentally adopt the RL modeling paradigm to formulate and optimize sequential decision-making.
\subsection{An Overview of Agentic Reinforcement Learning}
\subsection{Agentic Reinforcement Learning}
\subsubsection{Planning}
\subsection{Advanced Agent Systems}
\subsubsection{Tool Use}
\subsubsection{Memory}
% 当然还有自我反思和自我反馈，这些之前的章节已经有了
\subsection{Reinforcement Learning for Advanced Agent Systems}
