PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, Jun Zhao

arXiv:2606.26027v1cs.CLcs.LG

TL;DR

Multi-step tool-use RL can become unstable, with control-token amplification disrupting valid tool invocation despite preserved underlying capability. This paper studies supervisory signals and training schemes for stabilizing RL, finding that interleaved supervision is more stable and effective than synchronous training, while generalization remains limited by the study’s relatively small training-data scale.

  • Problem

    The paper asks how supervisory signals should be integrated into RL and which forms best stabilize long-horizon, structured tool-use training.

  • Method

    The study systematically evaluates diverse supervisory signals, including SFT, off-policy, erroneous-trajectory, hint-based, and process-reflection supervision, under synchronous and interleaved RL.

  • Results

    Interleaved training is more stable and effective than synchronous training, while RL failures involve control-token amplification that disrupts tool-use structure without degrading underlying reasoning ability.

  • Takeaways & Limitations

    Agentic RL collapse is primarily a structural execution problem, and carefully designed supervision can regulate token-level patterns for more stable tool-use learning.

  • Takeaways & Limitations

    The training data is relatively limited, and the impact of data scale on the results is not explored.

Abstract

from arXiv · show

Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.

1 Introduction

The paper identifies multi-turn agentic RL failure as structural collapse rather than capability loss, caused by special-token dynamics that disrupt tool-invocation sequences. It systematically studies supervisory signals and finds that interleaved supervision improves stability and generalization, whereas synchronous methods suffer distribution mismatch.

  • Failure mode: RL can abruptly collapse tool-use performance to near zero while valid tool-invocation structures break down.Erroneous response frequencies can also rise sharply at certain training stages.
  • Supervisory signals: The study evaluates six supervisory configurations across two model families using synchronous and interleaved training paradigms.The configurations include SFT then RL, off-policy supervision, erroneous trajectory supervision, Hint-based guidance, and Process Reflection Supervision.
  • Failure mechanism: Structural collapse preserves underlying task competence but degenerates generation into malformed control-token sequences.RL disproportionately amplifies special control tokens, redistributing policy mass and disrupting structured execution.
  • Training findings: Interleaved training consistently improves stability and generalization, while synchronous methods are prone to distribution mismatch.Supervised fine-tuning on high-quality tool-use trajectories also provides strong initialization, improves early performance, and prevents collapse.

2 Related Works

Recent work equips LLMs with tool-use capabilities for interacting with external APIs and environments, supporting task execution, information retrieval, and planning. RL advances these capabilities through trajectory exploration and reward-based updates, while expert trajectories and targeted SFT address poor sampling quality.

  • Tool-use extensions let LLMs interact with external APIs and environments beyond text generation, supporting task execution, information retrieval, and planning.
  • RL improves models through trajectory exploration and reward-based updates but can stagnate when sampling quality is poor.
  • Recent methods incorporate expert or ground-truth trajectories into RL, including LUFFY’s trajectory replacement and ReLIFT’s alternating RL and targeted SFT.

3 Preliminary

The paper models multi-turn tool use as trajectories alternating actions and feedbacks. Actions include tool invocations and natural-language responses, while feedback depends on the current action and preceding interaction history.

  • Trajectory representation: A tool-use trajectory is represented as τ = (a1, r1, a2, r2, . . . , aT, rT), alternating actions and feedbacks.Each action at belongs to action space A, and each feedback rt belongs to feedback space R.
  • Action space: The action space includes both feasible tool invocations and natural-language responses.Actions are drawn from A at each step.
  • Feedback modeling: Feedback is generated from the current action and preceding trajectory, encompassing environment outputs and user responses.The feedback function is rt = R(at | a1, r1, . . . , at−1, rt−1) ∈ R.

4 Method

The method analyzes why pure multi-step tool-use RL becomes unstable, showing that collapse reflects malformed control-token dynamics rather than loss of underlying capability. It then studies supervisory signals and training paradigms designed to stabilize exploration and preserve valid tool-use structure.

  • Experimental Setup: Experiments use BFCL-V3 across Base, Miss Func, Miss Param, and Long Context, training on 300 sampled questions from the first three settings.The benchmark requires multi-turn tool invocation across interactive environments.
  • Experimental Setup: SFT decomposes multi-turn interactions into isolated single-turn instances, whereas RL trains on complete trajectories requiring all tool invocations for positive feedback.The analysis uses Qwen2.5-1.5B-Instruct and Qwen3-1.7B as base models.
  • Training Instability: Direct RL can cause catastrophic collapse through reward drops and KL spikes, while SFT before RL generally produces smoother and more stable training.Qwen2.5-1.5B-Instruct shows the clearest collapse; Qwen3-1.7B remains milder but can still degrade or occasionally collapse.
  • Structural Collapse: Collapse shifts probability mass toward malformed combinations of control tokens such as <tool_call> and <|im_end|>, corrupting tool-invocation structure without necessarily destroying underlying capability.Outputs progress from healthy tool calls and responses through Text Pollution to Collapsed states dominated by degenerate termination patterns.
  • Supervisory Signals: The framework compares synchronous and interleaved supervision, including SFT-then-RL, off-policy supervision, hint-based guidance, erroneous trajectories, and process reflection supervision.Process reflection converts intermediate trajectory information into explicit textual guidance and is jointly trained with erroneous trajectories to improve adherence to correct tool-use formats.

5 Experiments

Experiments on Qwen2.5-1.5B-Instruct and Qwen3-1.7B show that targeted structural supervision is generally necessary for stable multi-step tool-use RL, while performance and stability depend on supervisory signal, learning rate, and invocation format. OOD results indicate that apparent RL collapse can be format-specific, whereas process-oriented supervision improves robustness to distribution shifts.

  • Both Qwen models have negligible baseline performance, underscoring the difficulty of multi-turn tool use without targeted training.
  • For Qwen2.5-1.5B-Instruct, SFT raises the performance floor before GRPO refinement, whereas GRPO alone provides no meaningful gains and can catastrophically collapse.Process Reflection Supervision achieves the highest average score at 25.75, followed by Erroneous Trajectory Supervision.
  • For Qwen3-1.7B, SFT initially outperforms direct RL but eventually collapses, which the authors attribute to “thinking” mode disrupting structured trajectory learning during sampling.
  • 1×10^-5 improves SFT and consistently improves all metrics when combined with ETS, while 1 × 10^-6 yields decreased gains.The results suggest that stabilizing multi-turn training may require larger learning rates than commonly used in RL.
  • SFT methods that excel in-distribution sharply degrade under Format and Content OOD, while unstable RL methods behave more stably there, indicating format-specific rather than global collapse.Process Reflection Supervision mitigates interleaved-training declines and acts as a process-oriented regularizer that improves resilience to distribution shifts.

6 Conclusion

The paper identifies instability and limited gains as key failure modes of reinforcement learning for multi-turn tool use. It shows that supervisory-signal integration, especially interleaved training, provides a more stable and effective framework while examining learning rate, SFT data distribution, and generalization.

  • Failure modes: Naive RL can over-amplify control tokens, disrupting structured tool use and causing collapse or plateaus.This is identified as a central mechanism behind training instability and limited gains.
  • Failure modes: The study identifies training instability and limited gains as two key failure modes in multi-turn tool-use reinforcement learning.These failure modes motivate the systematic examination of supervisory-signal integration.
  • Supervisory signals: Interleaved training is more stable and effective than synchronous training for integrating supervisory signals.The conclusion frames interleaving as part of a principled framework for stable training.
  • Training analysis: The work also analyzes learning rate, SFT data distribution, and generalization to support a principled framework for stable training.These analyses broaden the study beyond supervisory-signal integration alone.

Limitations

The study analyzes multi-turn tool-use failure modes and mitigation signals, but its conclusions are limited by relatively small training data and an unexplored data-scale effect.

  • Data limitations: The training data is relatively limited because open-source verifiable tool-invoking environments are scarce, and the impact of data scale remains unexplored.This constrains assessment of how training-data scale affects the reported results.

Ethics Statement

The authors report no ethical concerns and disclose using AI assistance solely to polish the manuscript’s language.

  • Ethics Statement: The work reports no ethical concerns and used AI assistance for vocabulary correction and spell checking.The assistance was limited to language polishing of the manuscript.

A Analysis

The analysis classifies model outputs by tool-call and termination structure, distinguishing valid executions and responses from structurally polluted or fully collapsed outputs. Collapsed outputs represent severe RL failure because they contain no meaningful action or response.

  • Output structure: Healthy tool calls contain correctly matched <tool_call> tags, valid internal actions, schema-conforming JSON, and no stray control tokens.This is the expected valid tool-invocation structure.
  • Output structure: Healthy responses omit tool-call tags, end with <|im_end|>, and provide coherent natural-language content without extraneous tags.They represent valid non-tool responses.
  • Output structure: Text-polluted outputs misuse tool-call tags through missing closures, embedded <|im_end|>, <think> insertion, or extra tags appended to normal text.These structures indicate corruption that can violate the protocol or disrupt parsing.
  • Output structure: Collapsed outputs reduce the interaction to <|im_end|> without semantic content, reflecting severe RL failure that short-circuits meaningful action or response.The collapse may occur directly or when all tool invocations become only the end-of-turn token.

B Training Details · C The Analysis of Qwen3 Training

The paper specifies Qwen3’s tool-use training format, supervisory interventions, reward design, and training schedules. It attributes Qwen3’s post-SFT-then-RL collapse primarily to inconsistencies between required thinking-mode formatting and SFT outputs.

  • B Training Details: Qwen3 RL prepends <think>\n\n</think>\n\n before each action to disable thinking mode, then removes this prefix during gradient updates.Only generated content is used for optimization.
  • B Training Details: Off-policy supervision samples 7 model trajectories and 1 ground-truth trajectory, while hint-based guidance generates 6 normal and 2 hint-conditioned trajectories.These are the experimental setups for the supervisory interventions.
  • B Training Details: Interleaved training uses 50, 100, and 150 RL rounds for Qwen2.5-1.5B-Instruct, versus 50, 50, and 100 for Qwen3-1.7B.The passage reports these round counts for the respective models.
  • B Training Details: The SFT and RL parameter configurations are summarized in Tables 4 and 5.Table 4 covers SFT configuration, while Table 5 covers RL configuration.
  • B Training Details: RL rewards 1 only when both state-based and response-based evaluations equal 1, and otherwise assigns 0.State-based evaluation checks the final backend state; response-based evaluation checks the execution path against the minimal viable function-call path.
  • B Training Details: State-based evaluation targets state-modifying tasks, whereas response-based evaluation enables read-only task evaluation such as stock or weather queries.The two evaluation types compare different aspects of tool-use execution.
  • C The Analysis of Qwen3 Training: The Qwen3 collapse after SFT-then-RL is primarily attributed to prompt-format inconsistencies involving its required thinking-mode reasoning segment.When reasoning is disabled, Qwen3 requires an explicit <think>\n\n</think> placeholder, but SFT data contains direct tool-call outputs without that wrapper.

D Detailed Evaluation · E Training Dynamic · F Prompt and Example

The paper reports checkpoint-dependent evaluation and training-dynamics analyses for Qwen2.5 and Qwen3, then specifies an RPS prompt that diagnoses tool-use errors and generates validated augmentation scenarios. The prompt emphasizes correct raw tool-call formatting, precondition checks, and actionable generalization across error categories.

  • D Detailed Evaluation: Evaluation uses the final checkpoint, although some models achieve stronger intermediate performance that is not retained at convergence.Best training-process performance is reported for Qwen2.5 in Figures 9–11 and Qwen3 in Figures 12–14.
  • E Training Dynamic: Training dynamics are presented for Qwen2.5 and Qwen3, including reward comparisons with and without the thinking token.The analyses appear in Figure 5, Figure 15, and Figure 8.
  • F Prompt and Example: The RPS prompt requires comprehensive natural-language analysis of erroneous tool interactions while enforcing raw <tool_call> ... </tool_call> formatting for correct calls.It asks the model to use the ground-truth tool-use sequence as reference and generalize the analysis to similar scenarios.
  • F Prompt and Example: Its first requirement is to identify the root cause, support it with interaction-log evidence, and provide one immediate actionable fix.The prompt labels this requirement “Core Error Analysis (must be first).”
  • F Prompt and Example: The prompt requests 3-5 similar scenarios covering missing parameters, wrong tool selection, output mismatch, timeout or failure, and ambiguous or multi-intent requests.Each scenario includes the request context, likely mistake, correct tool-call sequence, transferable reasoning, differences, and validation strategy.
  • F Prompt and Example: A separate scenario must be similar but require a different solution, with the similarity, differing tool requirements, correct calls, and rationale explained.This tests whether generated cases distinguish shared reasoning features from tool-specific handling.
  • F Prompt and Example: The examples emphasize validating authentication preconditions before protected actions and avoiding guessed credentials when required information is missing.The Twitter example identifies unauthenticated posting and retweeting, plus guessed credentials, as the root failure.
  • F Prompt and Example: Successful transactional flows authenticate with supplied credentials, perform the requested action, and preserve explicit identifiers when API argument shapes allow alternatives.The Twitter example accepts either empty retweet arguments or tweet_id, while recommending explicit tweet_id to reduce ambiguity.
Loading 2606.26027v1…