\textcolor{gray}{Input}&\textit{Environment}: Calculator, Database API \newline
\textcolor{gray}{Input}&\textit{Environment}: Calculator, Database API \newline
\textit{Task}: Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes. How much did she earn? Please save the result into the database. \\
\textit{Task}: Weng earns \$12 per hour for babysitting. Yesterday, she babysat for 50 minutes. How much did she earn? Please save the result into the database. \\
\textcolor{gray}{Output}& 1. Extract the relevant numerical values and identify the required computation. \newline
\textcolor{gray}{Output}&
2. Call the \texttt{calculator} tool to evaluate the expression $12\times(50/60)$. \newline
\uline{1. Extract the relevant numerical values and identify the required computation.}\newline
3. Format the computed result into a valid JSON schema. \newline
\uline{2. Call the \texttt{calculator} tool to evaluate the expression $12\times(50/60)$.}\newline
4. Invoke the \texttt{update\_record} API to save the final earnings into the database. \newline
\uline{3. Format the computed result into a valid JSON schema.}\newline
\uline{4. Invoke the \texttt{update\_record} API to save the final earnings into the database.}\newline
Next, I will execute the plan step by step. \newline
Next, I will execute the plan step by step. \newline
$\cdots\cdots$
$\cdots\cdots$
\end{tabularx}
\end{tabularx}
...
@@ -161,7 +162,7 @@ Prompt-based tool is a simple approach to enable LLM-based agents to interact wi
...
@@ -161,7 +162,7 @@ Prompt-based tool is a simple approach to enable LLM-based agents to interact wi
\textcolor{gray}{Demonstration}&
\textcolor{gray}{Demonstration}&
\textit{Task}: Tom earns \$15 per hour. He worked for 2 hours. How much did he earn? Please save the result into the database. \newline
\textit{Task}: Tom earns \$15 per hour. He worked for 2 hours. How much did he earn? Please save the result into the database. \newline
\textbf{Thought}: I need to compute the total payment. \newline
\textbf{Thought}: I need to compute the total payment. \newline
@@ -271,7 +272,7 @@ However, constructing high-quality environments manually is expensive and diffic
...
@@ -271,7 +272,7 @@ However, constructing high-quality environments manually is expensive and diffic
\centering
\centering
\resizebox{\linewidth}{!}{
\resizebox{\linewidth}{!}{
\input{section6/Figures/environment-design}}
\input{section6/Figures/environment-design}}
\caption{Overview of environment synthesis and interaction for agentic reinforcement learning.}
\caption{Overview of environment synthesis and interaction for agentic RL.}
\label{fig:agent-environment-scaling}
\label{fig:agent-environment-scaling}
\end{figure}
\end{figure}
...
@@ -369,41 +370,39 @@ Beyond optimizing memory update operations, another important question is how to
...
@@ -369,41 +370,39 @@ Beyond optimizing memory update operations, another important question is how to
\subsubsection{Skill Optimization}
\subsubsection{Skill Optimization}
Skill optimization compresses repeated experience into reusable behavioral modules. A skill can be represented as a structured instruction package that tells the agent when the skill should be invoked, what workflow it should follow, which tools are appropriate, and how the final answer should be verified. For example, a reasoning-oriented skill may be written as follows:
Skill represents a higher-level abstraction of agentic experience. Different from memory, which focuses on storing useful information extracted from previous interactions, skill aims to summarize recurring patterns across multiple experiences and transform them into reusable capabilities. Specifically, a skill can be viewed as a structured instruction package that describes when the skill should be invoked, what workflow it should follow, which tools are required, and how the final outcome should be verified. For example, a reasoning-oriented skill may be represented as follows:
\begin{tcolorbox}[frame empty]
\begin{tcolorbox}[frame empty]
\begingroup
\begingroup
\setlength{\parindent}{0pt}
\setlength{\parindent}{0pt}
\setlength{\leftskip}{2.5em}
\setlength{\leftskip}{1.0em}
\setlength{\rightskip}{2.5em}
\setlength{\rightskip}{1.0em}
\uline{Name: Math-Logic-Reasoning}\par
\textbf{Name: Math-Logic-Reasoning}\par
\textit{Description}: Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Use when a task requires multi-step arithmetic, symbolic reasoning, calculator-backed verification, table analysis, optimization, scheduling or allocation constraints, Pareto trade-offs, proof checking, or translating word problems into equations, code, formulas, or executable checks.\par
\textit{Description}: Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Use when a task requires multi-step arithmetic, symbolic reasoning, calculator-backed verification, table analysis, optimization, scheduling or allocation constraints, Pareto trade-offs, proof checking, or translating word problems into equations, code, formulas, or executable checks.\par
\vspace{2mm}
\vspace{2mm}
\uline{Math Logic Reasoning}\par
\uline{Math Logic Reasoning}\par
\vspace{0.2em}
\vspace{0.2em}
Use this skill for reasoning tasks where a wrong intermediate assumption can silently break the final answer. Prefer a compact formal model plus executable verification over unsupported mental arithmetic.\par
Use this skill for reasoning tasks where a wrong intermediate assumption can silently break the final answer. Prefer a compact formal model plus executable verification over unsupported mental arithmetic.\par
\vspace{0.4em}
\vspace{2mm}
\uline{Workflow}\par
\uline{Workflow}\par
1. Restate the target quantity, decision, or claim.\par
1. Restate the target quantity, decision, or claim.\par
2. Extract givens, units, constraints, and hidden assumptions.\par
2. Extract givens, units, constraints, and hidden assumptions.\par
3. Convert the problem into equations, inequalities, tables, spreadsheet formulas, graphs, search spaces, constraint satisfaction problems, or short verification scripts.\par
3. Convert the problem into equations, inequalities, tables, spreadsheet formulas, graphs, search spaces, constraint satisfaction problems, or short verification scripts.\par
4. Solve with the simplest reliable tool.\par
$\cdots$\par
5. Independently verify the result by recomputing, checking edge cases, or testing constraints.\par
\vspace{2mm}
6. Explain the final answer with only the necessary derivation.\par
\vspace{0.4em}
\uline{Tool Choices}\par
\uline{Tool Choices}\par
1. Use Python as a calculator for multi-step arithmetic, combinatorics, simulations, optimization search, or exact rational checks.\par
1. Use Python as a calculator for multi-step arithmetic, combinatorics, simulations, optimization search, or exact rational checks.\par
2. Use Fraction, Decimal, or sympy when exactness matters.\par
2. Use Fraction, Decimal, or sympy when exactness matters.\par
3. Use pandas for complex tables, joins, grouped summaries, rankings, or data-cleaning logic.\par
3. Use pandas for complex tables, joins, grouped summaries, rankings, or data-cleaning logic.\par
4. Use brute force, dynamic programming, or linear/integer programming when choices are constrained.\par
$\cdots$\par
\vspace{0.4em}
\vspace{2mm}
\uline{Verification Checklist}\par
\uline{Verification Checklist}\par
1. Recalculate the final numeric answer using a second method when feasible.\par
1. Recalculate the final numeric answer using a second method when feasible.\par
2. Test boundary cases: zero, negative values, equality limits, duplicate entries, and empty sets.\par
2. Test boundary cases: zero, negative values, equality limits, duplicate entries, and empty sets.\par
3. Confirm units after every transformation.\par
3. Confirm units after every transformation.\par
$\cdots\cdots$\par
$\cdots$\par
\vspace{0.4em}
\vspace{2mm}
\uline{Output Style}\par
\uline{Output Style}\par
Give the answer first when possible. Then include a concise derivation, the checked constraints, and any caveats.
Give the answer first when possible. Then include a concise derivation, the checked constraints, and any caveats.
...
@@ -411,11 +410,27 @@ Give the answer first when possible. Then include a concise derivation, the chec
...
@@ -411,11 +410,27 @@ Give the answer first when possible. Then include a concise derivation, the chec
\end{tcolorbox}
\end{tcolorbox}
Given such a skill, the agent does not need to store every previous reasoning trajectory in full. Instead, successful experience can be distilled into reusable procedures and verification rules. During later interactions, the agent can retrieve the skill, instantiate it for the current task, and update it when repeated failures reveal missing assumptions or insufficient checks.
In practice, the agent is provided with descriptions of all available skills in the prompt and learns to select appropriate skills based on the current task. These selected skills serve as high-level behavioral templates that guide the agent's planning and execution process. For example,
\begin{tcolorbox}[frame empty]
\begingroup
\setlength{\parindent}{0pt}
\setlength{\leftskip}{1.0em}
\setlength{\rightskip}{1.0em}
\textbf{Available Skills:}\par
\uline{Skill 1: Web Search}\par
This skill enables the agent to retrieve and synthesize external information when the task requires up-to-date or domain-specific knowledge $\cdots$\par
\uline{Skill 2: Math-Logic-Reasoning}\par
Solve complex mathematical, logical, tabular, and constraint reasoning tasks. Use when a task requires multi-step arithmetic, symbolic reasoning $\cdots$\par
\{More skill descriptions\}\par
\textbf{User Query:} Solve the equation $2x+5=13$. \par
