title={Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning},
author={Singh, Joykirat and Pandya, Yash and Vajreshwari, Pranav and Magazine, Raghav and Nambi, Akshay},
booktitle={First Workshop on Foundations of Reasoning in Language Models},
year={2025}
}
@inproceedings{melnyk:2024distributional,
title={Distributional Preference Alignment of LLMs via Optimal Transport},
author={Melnyk, Igor and Mroueh, Youssef and Belgodere, Brian and Rigotti, Mattia and Nitsure, Apoorva and Yurochkin, Mikhail and Greenewald, Kristjan and Navratil, Jiri and Ross, Jerret},
@@ -135,11 +135,55 @@ where $r_p$ indicates that the overall plan is reasonable, $r_1$ and $r_2$ indic
\subsubsection{Tool Using}
Tool using is another fundamental capability of LLM-based agents. Tools extend the agent's capability beyond its internal parameters. Prompt-based methods first show that LLMs can use tools through reasoning-action prompting. ReAct is a typical example \citep{yao-etal:react}. However, prompt-based tool use is unstable and depends on the base model. Thus, SFT methods train models on tool-use trajectories \citep{schick-etal:toolformer}. Still, SFT mainly teaches imitation. It cannot directly optimize whether the agent should use a tool, which tool to use, and how to balance tool benefit with tool cost. RL provides a natural framework for efficient tool using. We can define rewards based on task success, tool-use correctness, tool-use completeness, and tool cost \citep{qian-etal:toolrl}.
Tool use is another fundamental capability of LLM-based agents. Tools extend the agent beyond its internal parameters. They allow the agent to access external knowledge, perform accurate computation, and execute actions in external systems. Early studies show that LLMs can use tools through prompt-based reasoning-action patterns. ReAct is a representative example, where the model alternates between reasoning steps and tool-use actions \citep{yao-etal:react}. However, prompt-based tool use is often unstable. It also depends heavily on the capability of the model. Thus, SFT methods train LLMs on tool-use trajectories, so that the model can learn when and how to invoke tools from demonstrations \citep{schick-etal:toolformer}. Still, SFT mainly teaches imitation. It cannot directly optimize whether the agent should use a tool, which tool it should select, or how to balance tool benefit with tool cost. RL provides a natural framework for efficient tool use. We can define rewards based on task success, tool-use correctness, tool-use completeness, and tool cost \citep{qian-etal:toolrl,singh-etal:agentic}.
% 2. Prompt-based Tool Use
\subsubsubsection{Prompt-based Tool Using}
Prompt-based tool is a simple approach to enable LLM-based agents to interact with external tools without updating model parameters. However, the model may not naturally know when to invoke a tool or how to organize tool-use behaviors. Thus, we usually need to provide tool descriptions and a few demonstrations in the prompt. These demonstrations show the model how to alternate between reasoning, tool invocation, and observation. At each step, the model first generates a reasoning trace, then produces a tool-use action. After the tool returns an observation, the model uses this feedback to update its next decision. For example,
\begin{tcolorbox}[frame empty]
\begingroup
\centering
\renewcommand{\arraystretch}{1.5}
\begin{tabularx}{\dimexpr\linewidth-2em\relax}{
>{\raggedleft\arraybackslash}p{0.16\linewidth}
@{\hspace{0.6em}}
>{\raggedright\arraybackslash}X
}
\textcolor{gray}{Tools}&
\texttt{calculator(expression)}: return the numerical result of an expression. \newline
\texttt{update\_record(json)}: save a JSON record into the database. \\
\textcolor{gray}{Demonstration}&
\textit{Task}: Tom earns \$15 per hour. He worked for 2 hours. How much did he earn? Please save the result into the database. \newline
\textbf{Thought}: I need to compute the total payment. \newline
\textbf{Answer}: Weng earned \$10, and the result has been saved into the database.
\end{tabularx}
\endgroup
\end{tcolorbox}
This demonstration teaches the model a tool-use pattern. The model learns to reason about the task, call an external tool when needed, read the returned observation, and continue the execution process. We refer to the response returned by the external tool as an ``observation''. The agent uses this observation to update its reasoning and complete the task.
