PaperlayerView exact revision on arXiv

Source-linked AI summary

RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents

Haitian Zhong, Jixiu Zhai, Lei Song, Jiang Bian, Qiang Liu, Tieniu Tan

arXiv:2602.03025v1cs.AIcs.CL

TL;DR

Multi-turn tool calling has sparse rewards and can stall under SFT followed by GRPO when groups lack reward variation, causing uninformative advantages. RC-GRPO trains a reward-conditioned trajectory policy and samples discrete reward tokens within GRPO groups to restore diversity. On BFCLv4, it improves performance across the evaluated open models, reaching 85.00% overall accuracy on Qwen2.5-7B and exceeding the reported closed API baseline.

  • Problem

    Sparse trajectory-level rewards and low within-group reward variation can make GRPO’s normalized advantages uninformative and updates vanish, especially after peaked SFT initialization.

  • Method

    RC-GRPO first fine-tunes an RCTP on mixed-quality trajectories with discrete reward tokens, then conditions GRPO rollouts on diverse sampled tokens within each group.

  • Results

    85.00% overall accuracy on Qwen2.5-7B, above all reported baselines and the best closed API baseline at 61.25%; gains are also reported on LLaMA-3.1-8B.

  • Takeaways & Limitations

    The authors report that RC-GRPO’s gains arise from more informative group-relative advantages and non-vanishing updates rather than simply increasing policy randomness.

  • Takeaways & Limitations

    The paper presents a minimal variance-based explanation rather than a complete theory of multi-turn agent training dynamics.

Abstract

from arXiv · show

Multi-turn tool calling is challenging for Large Language Models (LLMs) because rewards are sparse and exploration is expensive. A common recipe, SFT followed by GRPO, can stall when within-group reward variation is low (e.g., more rollouts in a group receive the all 0 or all 1 reward), making the group-normalized advantage uninformative and yielding vanishing updates. To address this problem, we propose RC-GRPO (Reward-Conditioned Group Relative Policy Optimization), which treats exploration as a controllable steering problem via discrete reward tokens. We first fine-tune a Reward-Conditioned Trajectory Policy (RCTP) on mixed-quality trajectories with reward goal special tokens (e.g., <|high_reward|>, <|low_reward|>) injected into the prompts, enabling the model to learn how to generate distinct quality trajectories on demand. Then during RL, we sample diverse reward tokens within each GRPO group and condition rollouts on the sampled token to improve within-group diversity, improving advantage gains. On the Berkeley Function Calling Leaderboard v4 (BFCLv4) multi-turn benchmark, our method yields consistently improved performance than baselines, and the performance on Qwen-2.5-7B-Instruct even surpasses all closed-source API models.

1. Introduction

Multi-turn tool calling combines sparse trajectory-level rewards with costly exploration, exposing a failure mode in which low within-group reward variation makes GRPO advantages degenerate. RC-GRPO addresses this by conditioning trajectories on discrete reward tokens and sampling diverse tokens during GRPO.

  • Motivation: Sparse trajectory-level rewards make exploration costly in multi-turn tool calling, while GRPO depends on within-group reward variability for its learning signal.When group rewards have near-zero or zero standard deviation, normalized advantages become degenerate and policy updates vanish.
  • Motivation: SFT can create a peaked golden-path policy that reduces rollout diversity and leaves group-relative advantages with little informative contrast.This failure mode is especially severe when partial observability encourages repeated short-horizon behavior.
  • Approach: RC-GRPO makes within-group diversity controllable by first training an RCTP on mixed-quality trajectories with reward-goal tokens.The reward-conditioned policy learns to generate systematically different-quality trajectories under different tokens.
  • Approach: During GRPO, RC-GRPO samples diverse reward tokens within each group and conditions rollouts on them, injecting variance for non-degenerate group-normalized updates.This treats diversity as a controlled variable rather than a byproduct of sampling temperature.
  • Evaluation: Experiments evaluate LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct on the BFCLv4 multi-turn tool-calling split against SFT+GRPO and closed API baselines.The study also analyzes entropy, entropy–reward correlation, advantage spread, KL, gradient statistics, and variance-based theory.

2. Related Work

Prior work spans tool-use agents, function-calling benchmarks, critic-free policy optimization, multi-turn agent reinforcement learning, and return-conditioned sequence modeling. RC-GRPO is positioned at the intersection of these strands, targeting variance collapse in group-based optimization for multi-turn tool calling.

  • Tool-use benchmarks: BFCLv4 evaluates stateful multi-turn tool use, whereas several other benchmarks emphasize single-turn function selection or parameter correctness.The multi-turn subset requires maintaining state across sequential tool calls to solve complex queries.
  • Tool-use agents: Toolformer, ReAct, and ToolLLM established self-supervised tool learning, interleaved reasoning and acting, and scaling to real-world API collections.Related closed-loop agents address challenges including error recovery and long-horizon reasoning.
  • Policy optimization: PPO uses a separate critic, while GRPO estimates advantages relative to grouped outputs and reduces memory by approximately 50%.DPO also avoids explicit reward modeling but requires pairwise preferences.
  • Agent reinforcement learning: Recent multi-turn agent RL methods address locally overfit reasoning, tool-feedback instability, sparse credit assignment, and model-based planning.Examples include RAGEN, SimpleTIR, implicit step rewards, and Agent Early Experience.
  • Return-conditioned learning: Return-conditioned methods learn policies conditioned on target returns and can generate optimal trajectories when prompted with high return values.Decision Transformers and Upside-Down RL frame reinforcement learning as sequence modeling rather than value-function or policy-gradient estimation.

3. Method

RC-GRPO formalizes multi-turn tool calling as a POMDP and uses a two-stage pipeline: RCTP fine-tuning on reward-labeled trajectories followed by reward-conditioned GRPO. Discrete reward-token sampling creates structured within-group variation for trajectory-level advantages while preserving the GRPO optimization framework.

  • Problem formulation: Multi-turn tool calling is modeled as a POMDP in which the agent interleaves JSON tool calls with environment observations until termination.The interaction history accumulates prior queries, actions, and observations into a trajectory.
  • Reward conditioning: The policy is parameterized as πθ(at|ht, r), where discrete high- and low-reward tokens indicate expected trajectory quality.High-reward conditioning elicits optimal behavior, while low-reward conditioning generates plausible failures.
  • Stage 1: RCTP fine-tuning: Stage 1 fine-tunes an RCTP on mixed-quality trajectories paired with reward tokens, combining successful expert data with exploration-generated failures.The reward token is prepended to the history before each assistant turn, and training maximizes conditional log-likelihood.
  • Stage 2: RC-GRPO: Stage 2 samples a reward token for each trajectory in a GRPO group and conditions rollout generation on that token to inject structured variance.The sampling prior matches the successful-trajectory proportion in the RCTP dataset while guaranteeing variance injection.
  • Reward function: The binary trajectory reward equals one only when the desired state and procedural constraints are satisfied; otherwise it equals zero.State consistency compares terminal state with golden-action replay, while action coverage checks required tool calls and parameters.
  • Optimization: Group-relative advantages normalize trajectory rewards across tokens, and the PPO-style clipped objective adds KL regularization during optimization.The pipeline initializes Stage 2 from the Stage 1 reference policy and updates the trainable policy over sampled groups.

4. Experiments

Experiments on BFCLv4 evaluate RC-GRPO against training and API-model baselines, ablate reward conditioning and initialization, and analyze whether its gains arise from structured diversity rather than higher entropy. The results support a variance-based explanation in which reward-conditioned sampling preserves informative group-relative advantages.

  • Main Results: 85.00% overall accuracy on Qwen2.5-7B, exceeding every reported training baseline and the best closed API baseline at 61.25%.The full pipeline combines RCTP-FT initialization with RC-GRPO.
  • Main Results: 48.75% overall accuracy on LLaMA-3.1-8B, outperforming the Base Model and RCTP-FT + GRPO while matching the reported SFT-based baselines.The reported comparisons are Base Model 0.00%, SFT + GRPO 35.00%, SFT + RC-GRPO 35.00%, and RCTP-FT + GRPO 46.25%.
  • Reward Conditioning: +11.25% on Qwen2.5-7B and +2.50% on LLaMA-3.1-8B when RC-GRPO replaces standard GRPO from the same RCTP-FT initialization.The gains are reported as 73.75%→85.00% and 46.25%→48.75%, respectively.
  • RCTP Initialization: +38.75% on Qwen2.5-7B and +13.75% on LLaMA-3.1-8B when RCTP-FT replaces SFT initialization under RC-GRPO.Adding RC from an SFT initialization yields negligible gains, whereas changing the initialization produces large improvements.
  • Training Dynamics: RC-GRPO improves while entropy decreases from 0.079 to 0.037 and entropy–reward correlation is negative (ρ = −0.15), unlike standard GRPO’s ρ = +0.49.These results indicate that the reported improvement is not explained by simply increasing policy randomness.
  • Theoretical Analysis: When group rollouts receive identical rewards, normalized advantages become zero; near-identical rewards likewise make the learning signal negligible.The variance-based analysis connects peaked policies, collapsed within-group reward variance, and vanishing or noisy GRPO updates.

5. Conclusion

RC-GRPO is a two-stage pipeline that mitigates variance collapse in group-normalized policy optimization by controlling within-group diversity. It consistently improves BFCLv4 performance across Qwen2.5-7B and LLaMA-3.1-8B, with especially large gains from RCTP initialization.

  • 5. Conclusion: RC-GRPO combines RCTP finetuning with reward-conditioned GRPO to make within-group diversity a controlled variable.The pipeline addresses variance collapse in group-normalized policy optimization for multi-turn tool calling.
  • 5. Conclusion: RC-GRPO delivers consistent BFCLv4 gains for both Qwen2.5-7B and LLaMA-3.1-8B.The conclusion reports particularly large improvements when RC-GRPO is combined with RCTP initialization.
  • 5. Conclusion: The analysis attributes the gains to more informative group-relative advantages and non-vanishing updates rather than simply increased policy randomness.The paper connects these effects to its variance-based explanation of standard GRPO degeneration under peaked SFT policies.

Impact Statement

The work aims to improve reinforcement-learning stability for multi-turn tool-calling agents and enable more reliable tool use. Such systems may support routine digital-task automation and capable assistants, while retaining dual-use implications without safeguards.

  • Impact Statement: The paper targets more stable reinforcement-learning optimization for multi-turn tool-calling agents.Its stated goal is to improve stability in this setting.
  • Impact Statement: More reliable tool use could support routine digital-task automation and more capable assistants.The impact statement presents these as beneficial applications.
  • Impact Statement: Tool-using LLM capabilities may have dual-use implications without operational safeguards such as access controls.The statement preserves the paper’s deployment caveat.

A. Theoretical Proofs

The appendix supplies formal proofs for the theoretical results presented in the main text.

  • A. Theoretical Proofs: The appendix provides formal proofs for the theoretical results presented in the main text.It serves as the proof section for the paper’s theoretical claims.

A.1. Proof of Proposition 4.2

The proof uses probability bounds to show that a peaked SFT policy can make all group rollouts follow the optimal trajectory, collapsing reward variance and the GRPO update signal. It defines the relevant probability-space and KL assumptions before deriving the group-collapse result.

  • A.1. Proof of Proposition 4.2: The proof models actions on a discrete probability space and treats πref and π∗ as probability measures.π∗ is later specified as a Dirac measure concentrated on the optimal action.
  • A.1. Proof of Proposition 4.2: For a Dirac optimal policy, DKL(π∗∥π) ≤ ϵ implies π(a∗) ≥ e^-ϵ ≥ 1 − ϵ.This KL-probability bound supplies the per-step probability guarantee used in the proposition.
  • A.1. Proof of Proposition 4.2: Applying the per-step bound over T steps gives P(τ = τ∗) ≥ e^-Tϵsft ≥ 1 − Tϵsft.The proof applies the bound along a fixed optimal trajectory.
  • A.1. Proof of Proposition 4.2: For G independent rollouts, P(τ1 = ··· = τG = τ∗) ≥ e^-GTϵsft ≥ 1 − GTϵsft.The group-level probability follows by raising the single-rollout probability to the group size.
  • A.1. Proof of Proposition 4.2: When all group trajectories coincide, rewards are identical, σg = 0, and every numerator R(τj) − µg is zero.The proof therefore establishes the conditions for advantage collapse.

A.2. Proof of Proposition 4.3

The proof uses the law of total variance to separate within-group and between-group contributions, establishing a lower bound on total variance from between-group variance. It then applies this decomposition to reward-conditioned GRPO groups sampled from high- and low-reward conditions.

  • The law of total variance decomposes total variance into within-group and between-group variance.The within-group term is non-negative, so total variance is at least the between-group component.
  • Lemma A.4 introduces i.i.d. group samples with finite reward variance for analyzing the group second central moment.
  • The proof samples a high- or low-reward condition, generates a trajectory from the corresponding reference policy, and defines its bounded reward as X.The condition probability satisfies p = Psample(r = high) ∈ (0, 1), with conditional means µh and µl.
  • The proof invokes the law of total variance after defining the reward-conditioned sampling process.
  • A GRPO group is formed from G i.i.d. condition–trajectory pairs, with each reward defined as Xj := R(τj).

B. The Details of the Dataset Formulations

The appendix defines BFCLv4’s multi-turn benchmark, constructs aligned SFT, reward-conditioned fine-tuning, and RL datasets, and specifies the reward notation and components used for evaluation. The pipeline preserves ID-based splits, consistent formats, and both successful and failed trajectories for reward conditioning.

  • B.1. Berkeley Function Calling Leaderboard (BFCLv4): BFCLv4 evaluates tool-calling capabilities across APIs and multi-turn scenarios requiring context maintenance, state changes, and sequential tool execution.The multi-turn subset contains 200 human-curated trajectories spanning GorillaFileSystem, TwitterAPI, MathAPI, and TicketAPI domains.
  • B.1.1. DATA CURATION PIPELINE: The data pipeline aligns SFT, RCTP-FT, and RL datasets while using a shared ID-based 90%/10% train/test split.The split prevents question overlap between training and evaluation and keeps shared IDs together.
  • B.1.1. DATA CURATION PIPELINE: SFT uses 800 expert trajectories, while RCTP-FT combines 800 expert and 800 failure trajectories collected from exploration rollouts.Failure trajectories include incorrect tool selection, parameter errors, and premature termination.
  • B.1.1. DATA CURATION PIPELINE: RCTP-FT appends high- or low-reward tokens to the first user message, whereas SFT uses expert trajectories without reward tokens.The RCTP-FT dataset therefore maintains an exact 50-50 success–failure balance.
  • B.1.1. DATA CURATION PIPELINE: RL samples use minimal task pointers, with 720 training and 80 test examples under the same ID-based split.The environment adapter constructs the full prompt and interaction context at runtime.
  • B.1.1. DATA CURATION PIPELINE: All phases enforce OpenAI chat formatting, merged multi-step task prompts, and an explicit tool calls field in assistant messages.The format examples contrast separate multi-turn observations with a combined RL task prompt.
  • B.1.2. DATA EXAMPLES: The trajectory examples show reward-conditioned high-reward expert actions versus low-reward failure actions for the same task.The low-reward example copies a file instead of moving it.
  • D. Formal Reward Function Definition: The reward function multiplies state consistency and action coverage, requiring matching final state and required tool actions with correct parameters.Action coverage permits reordering of independent commutative actions, while state consistency compares the resulting environment state with the ground truth.

E. Experimental Settings Details

The experimental-settings appendix records the hardware, configuration tables, and model-specific hyperparameter references used for RC-GRPO, RCTP-FT, and baseline methods.

  • Experiments use 8 NVIDIA H200 GPUs.
  • The appendix specifies hyperparameter configurations for RC-GRPO, RCTP-FT, and the main baselines.
  • Tables 10 and 11 summarize the configurations for RC-GRPO and RCTP-FT.
  • Table 12 summarizes the baseline hyperparameter configurations.
Loading 2602.03025v1…