update.

rl-introduction.tex

@@ -5,6 +5,7 @@
 \usepackage{hyperref}
 \usepackage{url}
 \usepackage{booktabs}
+\usepackage{tabularx}
 \usepackage{fontawesome5}
 \usepackage{lineno}
 \usepackage{bbding}

section4/section4.tex

@@ -70,7 +70,7 @@ While learning from AI feedback is highly scalable and generally objective, this
 \begin{figure*}[!t]
     \centering
     \input{section4/Figures/figure-an-overlap-example-dense-reward-model}
-    \caption{
+    \caption[Issue of high variance in Monte Carlo-based advantage estimation]{
     An example of reward shaping. Here, length-based shaping rewards are implemented, e.g., awarding a +1 intermediate reward for every 10 tokens generated. Additionally, a delayed reward from the reward model evaluates the overall quality of the output at the end.
     }
     \label{fig:example-shaping-rewards}
@@ -294,9 +294,6 @@ In addition to optimizing the advantage computation, GRPO modifies the penalty t
 Moreover, there are other methods like GRPO that design advantage estimation without relying on a value model for training LLMs, such as those discussed in \citet{li-etal:2023remax} and \citet{hu:2025reinforce++}.
 
 
-% 实际上，还有一些类似的工作，尝试消除PPO的组件，。。。
-% remax  cpo等等reference or value models的工作
-
 
 \subsection{Efficient Reinforcement Learning Methods}
 Efficiency is a critical consideration for many practical applications of RL, and it becomes even more significant in the context of LLMs due to their vast number of parameters. For example, we might wish to train LLMs using RL, given memory and time constraints. In practice, the efficiency of RL is not a single issue but encompasses a wide range of challenges. While these challenges can be categorized in various ways in the existing literature, we focus on two fundamental aspects: \textit{time} and \textit{space} efficiency, which are commonly considered in efficiency-related problems. For these two efficiency problems, we aim to achieve the desired RL performance with the least amount of time and memory required.

section6/section6.tex

@@ -13,10 +13,44 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
 \subsection{Building Agent Capabilities}
 
 
+One feature of LLM-based agents is their capacity to interpret user instructions, autonomously formulate a step-by-step plan to resolve the given task, and systematically execute those steps. In applications, this planning process inherently involves coordinating interactions with external environments and determining appropriate tool invocations. The agent then relies on this structured plan as a blueprint to guide its execution trajectory over multiple turns. 
+
+To accomplish this process, an LLM-based agent relies on several complementary capabilities. Some of these capabilities are already inherited from pretrained LLMs. Recent advances in large-scale pretraining and post-training have significantly improved LLMs in instruction following, reasoning, code generation, and general world knowledge. These capabilities provide a strong foundation for agentic behavior, enabling LLMs to understand complex user requests and generate coherent intermediate reasoning steps without explicit interaction with the environment. The rapid improvement of these intrinsic capabilities has been a key factor driving the emergence of LLM-based agents, making it increasingly feasible for a single model to serve as the reasoning and decision-making core of an autonomous system.
+
+However, autonomous agents require capabilities that go beyond static reasoning. They must make sequential decisions over long horizons, coordinate multiple tool invocations, adapt to unexpected environmental feedback, and recover from execution failures. These capabilities cannot be effectively learned from static supervision alone because their quality depends on the outcomes of complete interaction trajectories rather than individual outputs. Reinforcement learning naturally addresses this challenge by optimizing behaviors through trial-and-error interaction, enabling agents to improve decisions based on delayed rewards and environmental feedback. As a result, RL complements the inherent reasoning abilities of LLMs by strengthening their decision-making and execution capabilities in dynamic environments.
+
+In the following sections, we discuss two representative capabilities that have become the primary focus of agentic RL: planning and tool use.
+
 
 \subsubsection{Planning}
 
 
+For example, given a specific task description, the model will output an actionable plan and tool use as follows:
+
+\vspace{0.5cm}
+\begin{tcolorbox}[frame empty]
+
+\begingroup
+% \setlength{\leftskip}{0.3em}
+% \setlength{\rightskip}{2em}
+\centering
+\renewcommand{\arraystretch}{1.5}
+\begin{tabularx}{\dimexpr\linewidth-2em\relax}{>{\raggedleft\arraybackslash}p{0.10\linewidth} X}
+\textcolor{gray}{Instruction} & You are an autonomous agent equipped with a calculator and a database API. Create a step-by-step plan to resolve the given task. \\
+\textcolor{gray}{Input} & Weng earns \$12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn? Please save the result using the \texttt{update\_record} API. \\
+\textcolor{gray}{Output} & 1. Extract the key numerical values and the required math operations from the query. \newline
+2. Call the \texttt{calculator} tool to evaluate the expression $12 \times (50 / 60)$. \newline
+3. Format the calculated result into a valid JSON schema. \newline
+4. Invoke the \texttt{update\_record} API to save the final earnings into the database.
+\end{tabularx}
+
+\endgroup
+
+\end{tcolorbox}
+\vspace{0.5cm}
+
+
+
 \subsubsection{Tool Use}
 
 
@@ -29,6 +63,9 @@ In this section, we focus on the emerging paradigm of agentic RL for LLM-based a
 
 \subsubsection{Trajectory Refinement}
 
+
+% 在这里做一个总结，整个大一点的图来去描述整体agent的运作
+
 \subsection{Better Environments}