PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGEN-2: Reasoning Collapse in Agentic RL

Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li

arXiv:2604.06268v1cs.LG

TL;DR

Multi-turn agent RL can suffer template collapse, in which reasoning remains apparently diverse but becomes input-agnostic and escapes entropy-based diagnosis. RAGEN-2 introduces MI proxies for online diagnosis and an SNR-based filtering method that selects high-signal prompts. Across tasks, the method improves input dependence and task performance, while its assumptions and scope remain bounded by several stated limitations.

  • Problem

    Entropy tracks within-input reasoning diversity but cannot reveal when reasoning becomes input-agnostic across different inputs.

  • Method

    The paper introduces retrieval-based mutual information proxies for diagnosis, explains collapse through reward-variance-driven SNR, and filters high-signal prompts before updates.

  • Results

    SNR-Aware Filtering improves input dependence and task performance across tasks, model scales, and modalities, while MI tracks task performance better than conditional entropy.

  • Takeaways & Limitations

    Mutual information provides a more reliable reasoning-quality diagnostic than entropy within the reported experiments, and reward-variance filtering offers a practical intervention.

  • Takeaways & Limitations

    The method assumes reward variance reliably proxies signal, an assumption that degrades in sparse or noisy reward environments; experiments are single-agent.

Abstract

from arXiv · show

RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.

1. Introduction

The paper identifies template collapse as a silent failure in multi-turn agent RL: reasoning can appear diverse while becoming input-agnostic. It diagnoses this gap with mutual information and proposes SNR-Aware Filtering to improve input dependence and task performance.

  • Entropy can remain high while failing to reveal whether reasoning responds to different inputs, so it cannot detect template collapse.
  • The paper proposes a mutual information proxy that scores each reasoning chain against all batch inputs to measure input dependence without external models.
  • SNR-Aware Filtering uses reward variance as a lightweight SNR proxy to select high-signal prompts each iteration without additional supervision.
  • Across planning, mathematical reasoning, web navigation, code execution, and tool use, SNR-Aware Filtering consistently improves input dependence and task performance.
  • Template collapse occurs when reasoning appears diverse within inputs but becomes input-agnostic across inputs, making agents unreliable and silently harming reasoning abilities.

2. Template Collapse in Multi-turn Agent RL

Template collapse separates reasoning diversity from input dependence: entropy can remain high while mutual information falls, yielding diverse but input-agnostic boilerplate. The paper develops retrieval-based MI proxies for diagnosis and explains collapse through reward-variance-driven gradient signal-to-noise.

  • 2.1. Setup and Preliminaries: Closed-loop multi-turn agent RL repeatedly rolls out trajectories, where each turn produces reasoning tokens, executable actions, and rewards.
  • 2.1. Setup and Preliminaries: The context X includes the system prompt, prior observations, actions, and reasoning tokens, while Z denotes the current turn’s reasoning sequence excluding action tokens and boundary markers.
  • 2.1. Setup and Preliminaries: PPO/GRPO regularization terms such as KL divergence and entropy bonus act uniformly across inputs regardless of content.
  • 2.2. Rethinking Reasoning Collapse from an Information-Theoretic Lens: Reasoning quality decomposes into within-input diversity H(Z|X) and input dependence I(X;Z), but entropy metrics primarily proxy only the former.
  • 2.2. Rethinking Reasoning Collapse from an Information-Theoretic Lens: Template collapse is the high-H(Z|X), low-I(X;Z) regime, where reasoning remains diverse yet becomes input-agnostic boilerplate.
  • 2.3. Mutual Information Proxy Family: The MI proxy estimates input dependence by in-batch cross-scoring each reasoning trace against prompts using matched and marginal log-likelihoods.
  • 2.3. Mutual Information Proxy Family: Under template collapse, retrieval accuracy approaches chance level 1/P, equal to 1.56% when P=64.
  • 2.3. Mutual Information Proxy Family: +0.39 for Trajectory MI-ZScore is the reported Spearman correlation with final task performance, versus −0.11 to −0.14 for entropy metrics.

3. The Mechanism of Template Collapse: A Signal-to-Noise Ratio (SNR) View

The paper explains template collapse as a low-SNR failure mode: weak task-discriminative gradients allow input-agnostic regularization to erase cross-input reasoning differences. It proposes filtering prompts by reward variance to concentrate updates on higher-signal examples.

  • Template collapse is input-agnostic reasoning with low I(X; Z) despite high H(Z| X), revealing the blind spot of entropy-only monitoring.
  • 3.1. Observing Signal-Noise Imbalance in RL Gradients: Reward variance provides a signal proxy because task-gradient norms increase with within-input reward variance, while regularizer-gradient norms remain flat across buckets.The gradient patterns hold across both evaluated algorithms.
  • 3.1. Observing Signal-Noise Imbalance in RL Gradients: Low-RV prompts weaken task-discriminative updates, allowing uniform KL and entropy regularization to produce fluent, diverse, but input-agnostic reasoning.
  • 3.2. Formalizing the SNR Mechanism via Gradient Decomposition: When reward variance approaches zero, advantages and task gradients weaken while regularization remains active, driving I(X; Z) toward zero without requiring H(Z| X) to decline.
  • 3.2. Formalizing the SNR Mechanism via Gradient Decomposition: The gradient decomposition separates task signal, task noise, and regularization noise, with regularization acting uniformly across reasoning chains.The SNR compares task signal against task noise plus regularization magnitude.
  • 3.3. SNR-Aware Filtering: Prioritizing High-Signal Updates: Filtering concentrates the gradient budget on prompts whose advantage estimates are more likely to align with task-relevant directions.
  • 3.3. SNR-Aware Filtering: Prioritizing High-Signal Updates: SNR-Aware Filtering estimates each prompt’s reward variance and retains only the top fraction before computing parameter updates.This removes low-variance updates likely to be dominated by input-agnostic regularization.
  • 3.3. SNR-Aware Filtering: Prioritizing High-Signal Updates: The method estimates prompt-level reward variance from multiple same-prompt trajectories, then ranks prompts by variance for adaptive high-signal selection.Top-p filtering ranks prompts by variance rather than token probability and adjusts the kept count to the variance distribution.

4. Experiments

Experiments across diverse environments and training configurations show that template collapse consistently appears as declining input dependence despite sustained entropy. SNR-Aware Filtering, especially Top-p selection, preserves reasoning diversity and improves task performance across broad settings.

  • 4. Experiments: The evaluation spans seven environments covering irreversible planning, sparse-reward stochastic navigation, symbolic mathematics, web tasks, and code execution.
  • 4.2. Template Collapse as a Consistent Failure Mode: Template collapse reliably emerges across training configurations, with I(X; Z) declining while H(Z| X) remains high.
  • 4.3. SNR-Aware Filtering Consistently Improves Performance: Top-p filtering best preserves both task performance and reasoning diversity, whereas no filtering shows the weakest performance.
  • 4.2. Template Collapse as a Consistent Failure Mode: Mutual information declines significantly before task performance degrades, while conditional entropy remains elevated throughout training.This divergence identifies template collapse earlier than task success alone.
  • 4.2. Template Collapse as a Consistent Failure Mode: Reasoning length declines monotonically across all eight evaluated environments, providing a behavioral signature of template collapse.
  • 4.3. SNR-Aware Filtering Consistently Improves Performance: Top-p filtering consistently achieves higher success rates than Top-k filtering and no filtering across four representative environments.Its adaptive selection can reject entire batches when most prompts carry weak signal.
  • 4.3. SNR-Aware Filtering Consistently Improves Performance: Across algorithms, model scales, types, and modalities, SNR-Aware Filtering improves peak task success rate in most settings.The experimental matrix includes PPO, DAPO, GRPO, Dr. GRPO, Qwen2.5 scales, Llama3.2-3B, and text- or image-conditioned inputs.
  • 4.3. SNR-Aware Filtering Consistently Improves Performance: With filtering at ρ=0.9, fewer groups enter gradient computation, reducing per-step time by 26–41%.Reward-variance computation itself adds <0.1% of iteration time.

5. Analysis

MI-family metrics better separate high- and low-performance runs than entropy, while SNR-Aware Filtering improves both input dependence and task success. The analyses attribute template collapse to low reward variance weakening task gradients and show clear boundary conditions for filtering.

  • MI versus entropy: MI separates high- and low-performance runs across intervention families, whereas entropy can conflate them.Entropy- and KL-based interventions mainly shift conditional entropy; SNR-Aware Filtering improves both MI and task success.
  • MI versus entropy: +0.39 Spearman correlation is achieved by Trajectory MI-ZScore, compared with −0.11 to −0.14 for entropy metrics.The correlations are computed across runs varying entropy regularization, KL constraints, and Top-p filtering.
  • Filtering effects: SNR-Aware Filtering consistently improves peak task success across four tasks, algorithms, model scales, and input modalities.Table 4 reports gains as +Δ values next to each peak.
  • SNR mechanism: Low reward variance weakens task gradients while input-agnostic regularization remains, erasing input dependence.The proposed SNR account is tested through reward-variance buckets and controlled environmental noise.
  • Boundary conditions: At 80%–100% environmental stochasticity, filtering’s advantage closes because transition noise weakens reward variance as a signal proxy.Across increasing stochasticity, task return and MI decrease while conditional entropy rises.
  • SNR mechanism: Filtering is most useful when cross-prompt reward-variance heterogeneity separates signal-rich from noise-only prompts.When Std(RV)/Mean(RV) is near zero, filtering discards data uniformly; FrozenLake shows Δ=−5.0% at ratio 0.33.
  • Training dynamics: As training converges, zero-variance prompts increase and the effective kept ratio falls, making filtering more selective.Late-stage reward distributions show compressed prompt-level variance and less task-discriminative gradient information.
  • Diagnostic scope: Format validity remains largely decoupled from MI, so structurally correct responses can still exhibit low input dependence.Reward variance is also largely orthogonal to entropy and response length, targeting a distinct update-quality axis.

6. Related Work

Related work studies collapse, reasoning diversity, and training stabilization in closed-loop LLM-agent RL. Existing diversity and stabilization methods generally do not directly test or prevent whether reasoning remains input-dependent.

  • Collapse phenomena: Prior agent-RL work reports reasoning collapse as templated rationales with weaker input correspondence and policy degeneracy as concentrated behavioral patterns.These phenomena resemble collapse observed in self-training even when average metrics remain stable.
  • Reasoning evaluation: Most diversity metrics measure within-input variability rather than whether differences are systematically driven by input goals.Common approaches include lexical overlap, embedding dispersion, uncertainty analysis, behavioral tests, and retrieval-style matching.
  • Training stabilization: Prior stabilization methods include KL control, entropy regularization, clipping, reward shaping, curricula, replay mixtures, and stepwise rewards.These methods target optimization stability, credit assignment, or reward quality across multi-step agents.
  • Training stabilization: When rollouts receive similar rewards regardless of reasoning quality, gradient updates carry little information about which reasoning path matters.This motivates analyzing template collapse through a signal-to-noise perspective.
  • Training stabilization: Figure 13 compares training trajectories at steps 10 and 400 under three interventions, with color intensity encoding intervention strength.Arrows indicate later steps.

7. Conclusions and Limitations

The paper defines template collapse as input-agnostic reasoning that can remain fluent and diverse-looking while conditional entropy stays stable. It diagnoses this failure with MI, explains it through low reward variance, and addresses it with SNR-Aware Filtering, subject to explicit scope and signal-quality limitations.

  • Conclusions: Template collapse is fluent, input-agnostic boilerplate that can persist while conditional entropy remains stable.The paper presents it as a silent failure that can undermine reasoning reliability.
  • Conclusions: The paper introduces an MI proxy, an SNR mechanism, and SNR-Aware Filtering to diagnose, explain, and address template collapse.Filtering prioritizes prompts with reward variance before each parameter update.
  • Conclusions: SNR-Aware Filtering improves performance on average across tasks, model scales, and modalities and can integrate with existing training pipelines.The conclusion frames filtering as a practical response to input-agnostic reasoning drift.
  • Limitations: The SNR decomposition assumes task signal and regularization noise separate cleanly, although gradient accumulation may couple them.This assumption limits how directly the mechanism should be interpreted.
  • Limitations: All experiments are single-agent, leaving propagation of template collapse in multi-agent RL open.The paper also notes risks from gaming reward variance and from reduced exploration under aggressive filtering.
  • Limitations: The method requires reward variance to remain a reliable signal proxy, which degrades in sparse or noisy reward environments.The kept mass also requires per-task tuning.

B.1. Environments and Tasks

The evaluation spans synthetic environments covering planning, stochastic control, mathematical reasoning, web search and shopping, and code generation. Experiments use multiple policy-gradient methods and standardized rollout, filtering, stopping, and evaluation settings.

  • Environments and tasks: The seven-environment testbed covers Sokoban, FrozenLake, MetaMathQA, Countdown, SearchQA, WebShop, and DeepCoder.These tasks span irreversible planning, stochastic control, mathematics, search, web navigation, and program synthesis.
  • Interactive environments: Sokoban tests irreversible multi-step planning, while FrozenLake tests long-horizon navigation with sparse rewards and stochastic transitions.Sokoban puzzles are procedurally generated; FrozenLake uses a 2% random-rate variant.
  • Reasoning environments: MetaMathQA and Countdown evaluate multi-step mathematical and compositional arithmetic reasoning with correctness-based rewards.Countdown requires constructing an expression that reaches a target using source numbers at most once.
  • Coding environment: DeepCoder evaluates Python program synthesis against public and hidden test cases.Its benchmark draws from PrimeIntellect, TACO, and LiveCodeBench v5.
  • Web environments: SearchQA requires iterative web search and answer synthesis, while WebShop evaluates goal-directed navigation and attribute matching.Both environments require sequential multi-turn interaction.
  • Experimental setup: Main experiments use Qwen2.5-3B with PPO, DAPO, GRPO, and Dr.GRPO for up to 400 rollout-update iterations.Each iteration collects 128 trajectories organized as 8 prompt groups with 16 samples per prompt.
  • Experimental setup: Filtering compares top_p=0.9 against an unfiltered setting, with early stopping triggered by reward-variance collapse or sustained low validation success.Evaluation uses 512 fixed validation prompts per environment and reports success rate as the primary metric.

C. Filtering Ablation Results

Filtering performance depends strongly on both sampling and filtering choices. Reward-variance filtering best preserves MI while improving task performance, whereas low-variance or entropy-based strategies are less reliable.

  • Experimental Setup: The ablation evaluates filtering on Qwen2.5-3B in the Sokoban environment using metrics including task performance, MI proxy, entropy, and collapse.Table 9 reports absolute values and changes relative to block-specific baselines.
  • Sampling Settings: Reducing top_p or min_p generally improves task performance and lowers entropy, but affects MI retention heterogeneously.Top_k more often preserves or improves MI, while task-performance gains are less consistent.
  • Filtering Metrics: SNR-Aware Filtering achieves strong task performance while preserving MI better than entropy-, entropy-variance-, and length-based filters.The alternatives either suppress MI or fail to prevent collapse.
  • Keep Strategy: High-variance trajectory groups produce substantially higher task performance and MI proxy than low-variance groups, which also markedly increase entropy.The asymmetry supports high-variance rollouts as more informative training signal.
  • Summary: Reward-variance filtering consistently improves task performance while maintaining information content, whereas entropy-based heuristics are more prone to collapse.More aggressive filtering is not universally beneficial; the filtering metric remains critical.

D.1. MI Proxy Metrics During Training

The appendix tracks reasoning with multiple MI proxy metrics over training. Without filtering, these proxies degrade early, while filtering preserves information retention and adaptively selects higher-signal prompts using reward variance.

  • MI Proxy Metrics: All six MI proxy metrics confirm that MI degrades early without filtering, signaling reasoning collapse.The metrics include MI estimates, retrieval measures, and conditional entropy.
  • MI Proxy Metrics: Filtering preserves information retention across MI proxies throughout training, with top-p SNR-aware filtering performing best in Figure 14.The figure compares six proxy metrics over training steps.
  • Reward Variance: Low reward variance makes rollouts weakly distinguishable by reward, whereas high variance may reflect trajectory-dependent signal or evaluation noise.Zero-variance groups can optionally be excluded before selection.
  • Proxy Interpretation: The proxies are scorer-defined quantities measured in log-likelihood units and are not exact Shannon entropies or mutual information unless the scorer matches the evaluated distribution.They also depend on the sampled prompt set and fixed scorer.
  • Filtering Method: SNR-Aware Filtering ranks prompts by descending within-prompt reward variance and retains high-variance groups adaptively.Top-p selects the smallest prefix reaching a cumulative variance-mass threshold and can reject degenerate batches.

H.3. Task-gradient magnitude is RV-controlled

The theory links within-prompt reward variance to task-gradient strength and estimator signal-to-noise ratio. Low-variance prompts therefore receive weaker task signals and are more vulnerable to noise or reward-agnostic regularization.

  • Task-Gradient Bound: The task-gradient norm for a prompt is at most proportional to the square root of its within-prompt reward variance.Thus, small RV(x) implies a provably weak task gradient.
  • SNR Bound: When reward variance is low relative to reward noise, the Monte Carlo task-gradient estimator is dominated by noise.The SNR bound is expressed through RV(x) and the noise scale σ(x).
  • Template Mixing: A prompt-independent mixture component contracts mutual information by at least a factor of (1 − α), formalizing erosion of input dependence.This models partial drift toward a shared template.
  • Filtered Estimation: Filtering high-noise, low-RV groups lowers the filtered estimator’s mean-squared error by removing groups with larger noise variances.The filtered estimator remains unbiased for the filtered mean, but can be biased relative to the original objective.
  • Regularizer Influence: Low RV weakens task gradients while regularizer gradients remain relatively flat, increasing the regularizer’s influence on the total update.Under slowly varying regularizer and score norms, smaller RV implies a larger dominance ratio.
  • KL Constraint: Strong KL constraints preserve input dependence when the policy stays close to a reference policy, but do not necessarily increase it.The continuity result assumes finite input and reasoning supports.

M. Decomposing Changes in Input Dependence

Changes in input dependence equal changes in marginal reasoning diversity minus changes in within-input diversity. Therefore, increasing within-input entropy more than marginal entropy reduces mutual information.

  • Decomposition: The change in mutual information decomposes as ΔI = Δmarg − Δin.Δin and Δmarg denote changes in conditional and marginal entropy, respectively.
  • Interpretation: Input dependence decreases whenever within-input entropy increases more than marginal entropy.Formally, Iθ(X; Z) < I0(X; Z) whenever Δin > Δmarg.
  • Entropy Bonus: An entropy bonus directly increases per-prompt dispersion but does not explicitly encourage comparable cross-prompt separation.This makes a decrease in mutual information plausible when Δin exceeds Δmarg.
  • Mechanism: Low reward variance can weaken task updates, allowing reward-agnostic regularizers to exert greater relative influence.This connects the decomposition to the low-RV mechanism discussed in Appendix K.

N. GRPO Normalization Amplifies Noise at Low RV

GRPO-style normalization creates a variance floor that grows as reward variance decreases, making low-RV prompts especially vulnerable to noisy gradient updates.

  • The per-prompt gradient estimator experiences RV(x)^−1 noise amplification in its mean-squared error.
  • GRPO normalization uses an advantage defined as reward minus a prompt-dependent expected-reward baseline, based on K independent rollouts.The estimator uses score terms s_k = ∇θ log πθ(z_k|x).
  • RV(x) causes the gradient estimator’s variance floor to scale as RV(x)^−1, so small-RV prompts receive disproportionately noisy GRPO updates.This scaling is the central consequence of the GRPO variance-floor proposition.
  • If σ(x) = 0, the stated lower bound becomes zero and is therefore vacuous.
  • Smaller RV(x) yields a larger variance floor for the normalized estimator when other factors remain fixed.
Loading 2604.06268v1…