where $R_{\mathrm{ans}}$ measures final answer correctness, $R_{\mathrm{fmt}}$evaluates whether the model follows the required tool-use format, $R_{\mathrm{exec}}$ measures whether tool calls can be successfully executed, and $R_{\mathrm{tool}}$ evaluates whether the model selects appropriate tools and passes valid arguments. $C_{\mathrm{tool}}$ denotes the cost of tool use, such as the number of tool calls, latency, or computational expense. The coefficients $\alpha$, $\beta$, $\gamma$, and $\lambda$ control the relative importance of different reward terms.
where $R_{\mathrm{ans}}$ measures final answer correctness, $R_{\mathrm{fmt}}$measures whether the model follows the required tool-use format, $R_{\mathrm{exec}}$ measures whether tool calls can be successfully executed, and $R_{\mathrm{tool}}$ measures whether the model selects appropriate tools and passes valid arguments. $C_{\mathrm{tool}}$ denotes the cost of tool use, such as the number of tool calls, latency, or computational expense. The coefficients $\alpha$, $\beta$, $\gamma$, and $\lambda$ control the relative importance of different reward terms.
In application, this reward decomposition provides denser feedback than final-answer rewards. For example, if the model selects the correct tool but passes an invalid argument, the execution reward can penalize the invalid call while preserving credit for the correct tool selection. If the model reaches the correct answer but uses many unnecessary tool calls, the cost term can discourage inefficient behavior. ToolRL systematically studies this reward design problem and shows that fine-grained reward decomposition can make RL training more stable and effective for tool selection and tool application \citep{qian-etal:toolrl}.
In application, this reward decomposition provides denser feedback than final-answer rewards. For example, if the model selects the correct tool but passes an invalid argument, the execution reward can penalize the invalid call while preserving credit for the correct tool selection. If the model reaches the correct answer but uses many unnecessary tool calls, the cost term can discourage inefficient behavior. ToolRL systematically studies this reward design problem and shows that fine-grained reward decomposition can make RL training more stable and effective for tool selection and tool application \citep{qian-etal:toolrl}.
...
@@ -245,6 +245,7 @@ So far, we have introduced how RL can enhance the core capabilities of LLM-based
...
@@ -245,6 +245,7 @@ So far, we have introduced how RL can enhance the core capabilities of LLM-based
