PaperlayerView exact revision on arXiv

Source-linked AI summary

On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement

Wenlong Deng, Yushu Li, Boying Gong, Yi Ren, Christos Thrampoulidis, Xiaoxiao Li

arXiv:2512.04220v2cs.CL

TL;DR

Tool-integrated GRPO can collapse, but the mechanism behind this instability has been insufficiently understood. This paper identifies LLD as a driver of collapse and introduces selective likelihood-preserving regularization. LLDS stabilizes training and improves results across QA benchmarks.

  • Problem

    Although GRPO collapse has been observed in tool-integrated reinforcement learning, its underlying mechanisms lack understanding.

    Source: Models · pp. 1–2

  • Method

    LLDS activates regularization only when an action’s likelihood decreases and penalizes only the tokens responsible for that decrease.

    Source: §5.5 · p. 6

  • Results

    LLDS delivers consistent gains across seven QA benchmarks; on Qwen2.5-3B, LLDS-MA achieves an average score of 0.440, a +45.2% improvement over GRPO.

    Source: §6.1 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    Selective likelihood regularization stabilizes training and suppresses gradient explosion in GRPO-based tool-integrated reinforcement learning.

    Source: §7 · p. 9

Abstract

from arXiv · show

Tool-integrated (TI) reinforcement learning (RL) enables large language models (LLMs) to perform multi-step reasoning by interacting with external tools such as search engines and retrievers. Group Relative Policy Optimization (GRPO), exemplified by the recent Search-R1, offers fast convergence and a value-free formulation that makes it appealing for this setting, yet consistently suffers from training collapse. We identify Lazy Likelihood Displacement (LLD), a systematic reduction or stagnation in the likelihood of both correct and incorrect responses, as the core mechanism driving this failure. LLD emerges early and triggers a self-reinforcing LLD Death Spiral, where declining likelihood leads to low-confidence responses, inflating gradients, and ultimately causing collapse. We empirically characterize this process across models on a Search-R1-style, search-integrated question answering task, revealing a consistent three-phase trajectory: early stagnation, steady decay, and accelerated collapse. To address this, we propose a likelihood-preserving regularization LLDS that activates only when a response action's likelihood decreases, and regularizes only the tokens responsible. This fine-grained structure mitigates LLD with minimal interference. Our method stabilizes training, prevents gradient explosion, and yields substantial performance improvements across seven benchmarks, including relative improvements of +45.2% on Qwen2.5-3B and +37.1% on Qwen2.5-7B over vanilla GRPO training. Our results establish LLD as a previously overlooked bottleneck in GRPO-based TIRL and provide a practical path toward stable, scalable training of tool-integrated RL.

Models

The paper identifies Lazy Likelihood Displacement (LLD) as an overlooked source of collapse in tool-integrated reinforcement learning and proposes selective regularization to address it.

Models

  • LLD is identified as a previously overlooked source of collapse in GRPO-based tool-integrated reinforcement learning.It involves sustained likelihood decay, gradient amplification, entropy inflation, and eventual training collapse.Source: Models · p. 2
  • The proposed LLDS regularizer activates only when likelihood decreases and penalizes only the tokens responsible for that decrease.The selective design is intended to mitigate LLD while minimally interfering with GRPO optimization.Source: Models · p. 2
  • LLDS stabilizes training and delivers consistent gains across seven open-domain and multi-hop QA benchmarks.The paper presents these gains as evidence for a more robust and reliable approach to tool-integrated reinforcement learning.Source: Models · p. 2

2. Related Work

Prior work develops tool-integrated reasoning systems and observes GRPO collapse in multi-turn settings, but existing explanations do not account for its structural origin.

Tool-Integrated Reasoning and Agentic LLMs

  • Tool use enables large language models (LLMs) to perform adaptive, multi-step reasoning, a capability discussed as tool-integrated reasoning in agentic LLMs.Source: §2 · p. 2

Training Collapse in Tool-Integrated GRPO

  • GRPO often collapses in multi-turn tool settings, while PPO is comparatively stable; prior explanations do not account for PPO’s robustness or the instability’s structural origin.The cited explanations attribute collapse to training–inference mismatch or low-likelihood incorrect responses (Xue et al., 2025).Source: §2 · p. 2
  • Existing mitigation methods add supervision or computational cost, whereas the paper argues outcome rewards suffice once GRPO is stabilized.Turn-level shaping may use ground-truth documents or an external LLM judge, while TreeGRPO uses tree-based rollouts.Source: §2 · p. 2

3. Preliminary

The preliminary section describes tool-integrated trajectories and the GRPO update used to train policies from grouped responses and rewards.

3. Preliminary

  • A tool-integrated model generates actions from accumulated context, which may include feedback returned after a previous tool call.Tool feedback is out-of-distribution for the language model; feedback tokens are masked during training but still affect subsequent predictions.Source: §3 · pp. 2–3
  • GRPO samples a group of responses and normalizes their rewards into advantages, with each token in a trajectory sharing its response’s normalized advantage.The method is presented as value-free and uses group-relative reward normalization.Source: §3 · p. 3
  • A trajectory alternates model actions and tool feedback, and its number of tool calls determines the number of multi-turn interactions.Each action comprises tokens, while the response receives a reward.Source: §3 · p. 3

4. Lazy Likelihood Displacement in Tool-integrated GRPO

In tool-integrated GRPO, likelihood displacement becomes a trajectory-level instability associated with low-likelihood incorrect responses and similarity between correct and incorrect trajectories.

4. Lazy Likelihood Displacement in Tool-integrated GRPO

  • Tool-integrated GRPO exhibits a distinctive trajectory-level instability in which likelihood displacement interacts with external observations and shared decision prefixes, amplifying gradients and destabilizing training.The paper contrasts this setting with single-turn GRPO analyses that lack external observations.Source: §4 · p. 3
  • The training trajectory has early stagnation, steady likelihood decay despite rising rewards, and a later acceleration phase with a sharp correct-response likelihood drop and gradient surge.The cited example places steady decay at steps 60–120 and acceleration after step 120.Source: Figure 2 · p. 3
  • Tool-LLD is defined through action-level log-likelihood change after optimization, with LLD occurring when that change is at or below a small or non-positive threshold.The definition computes likelihood over actions while masking tool feedback.Source: §4 · p. 3
  • LLD can reduce the likelihood of correct actions when incorrect responses have low likelihood and high embedding similarity to correct responses, allowing their negative gradients to dominate.Low likelihood produces large prediction-error weights, while similar representations let negative gradients interfere with positive updates.Source: §4 · p. 4
  • Correct-response likelihood reductions are frequent in tool-integrated GRPO, indicating an acute LLD regime with progressive output-likelihood decay.The paper characterizes this compounding decay as distinctive to Search-R1-style tool-integrated RL.Source: §4 · p. 4

5. LLD in Tool-Integrated GRPO Collapse

The section traces how response-likelihood stagnation develops into accelerating decay and collapse, examines why tool-integrated GRPO amplifies it, and introduces selective regularization to suppress it.

5.1. Likelihood Dynamic

  • After early stagnation and steady decline, response likelihood drops sharply around step 120 as gradient magnitude rises rapidly toward gradient explosion and eventual collapse.Source: §5.1 · p. 4

5.2. LLD Death Spiral

  • The authors formalize accelerated likelihood decay as an LLD death spiral, a self-reinforcing pattern examined beyond the acceleration phase.Source: §5.2 · p. 4

5.3. Empirical Evidence for LLD Death Spiral

  • In an NQ controlled study, >50% of samples show likelihood drops during iterations 120–140; entropy accelerates while response length and valid-search frequency stay nearly constant.Source: §5.3 · pp. 4–5, Figure 4 · p. 5

5.4. Why Tool-Integrated (TI) GRPO Amplifies LLD

5.5. LLD Suppression regularization

  • LLDS compares token-level likelihood changes and regularizes only decrease-responsible tokens when the action’s total likelihood decreases.Source: §5.5 · p. 6
  • LLDS-MA softens regularization on final-answer tokens; the preserving set Ypre includes non-negative-advantage responses, and LLDS-MA is used when collapse becomes single-turn behavior.Source: §5.5 · p. 6

6. Experiments and Analysis

Across Qwen2.5-3B and 7B variants, LLDS improves QA results over GRPO and stabilizes training. Ablations show that regularization strength and answer masking affect these outcomes.

6.1. Experimental Results

  • +45.2% improvement gives Qwen2.5-3B-Base an average score of 0.440 with LLDS-MA over GRPO; LLDS also improves 3B-Instruct by 31.3%.The 3B-Base model tends to search only once; LLDS without MA improves its score from 0.312 to 0.360.Source: Table 1 · p. 7, §6.1 · p. 7

6.2. Ablation Study and Analysis

  • LLDS stabilizes training across four Base and Instruct variants, while vanilla GRPO collapses within the first 300 steps.Figure 7 compares reward curves for Qwen-2.5-3B and 7B models.Source: Figure 7 · p. 7, §6.2 · p. 8

6.1. Experimental Results

  • LLDS outperforms Tree-GRPO by 4% on 3B and 10% on 7B, while using outcome rewards rather than dense-reward supervision.The paper reports that dense-reward alternatives may require turn-level ground truth, external LLM judges, or more expensive tree-based rollouts.Source: §6.1 · p. 7
  • 37.1% relative gain raises Qwen2.5-7B-Base’s average score from 0.350 to 0.480; Qwen2.5-7B-Instruct gains 22.0%, from 0.396 to 0.483.The reported comparisons are across the two 7B variants; the results tables cover General QA and Multi-Hop QA.Source: §6.1 · p. 7, Table 2 · p. 8

6.2. Ablation Study and Analysis

  • With λ = 0.2, training remains stable, whereas λ = 0 collapses around step 200 and λ = 0.01 delays collapse only until around step 220.For 3B-Base, masking answer tokens raises the NQ+Hotpot average from 0.360 with LLDS to 0.430.Source: §6.2 · p. 8, Figure 8 · p. 8

7. Conclusion

The paper identifies likelihood displacement as a driver of GRPO failure in search-based tool-integrated RL and introduces selective regularization to address it. Across seven QA benchmarks, LLDS stabilizes training and yields consistent gains.

7. Conclusion

  • LLD is identified as a primary failure mechanism, emerging early and developing into a self-reinforcing cycle of likelihood decay, entropy inflation, and gradient explosion.The paper also describes tool integration as coupling likelihood updates across trajectories, producing instability distinctive to tool-using LLMs.Source: §7 · p. 9
  • LLDS activates only when likelihood decreases and regularizes only the tokens responsible, stabilizing training with consistent gains across seven QA benchmarks.Source: §7 · p. 9
  • The paper proposes likelihood dynamics as a signal for early failure detection and identifies extending LLDS to other tools and agentic settings as future work.Source: §7 · p. 9

A. Theorem and Proof

The theorem characterizes how negative-gradient effects can make correct-response action likelihood changes smaller or negative. It identifies low-likelihood incorrect responses and representational similarity as factors that amplify this effect.

Theorem A.1

  • The action-level likelihood change for a correct response becomes lazier or potentially negative as the impact of negative gradients increases.The theorem’s stated factors include low-likelihood incorrect responses and similarity between incorrect and correct responses.Source: A. Theorem and Proof · p. 11
  • Low-likelihood incorrect responses receive disproportionately large gradient weights, magnifying their negative influence on similar correct trajectories.Source: A. Theorem and Proof · p. 11
  • When incorrect and correct responses have similar representations, their overlap amplifies negative contributions and can make the model unconfident.Source: A. Theorem and Proof · p. 11

A.1. Proof of Theorem A.1

The proof treats response actions as the likelihood-bearing units while tool feedback remains in context, then aggregates action-level changes to characterize trajectory-level behavior. The appendix also defines the LLD Death Spiral and situates the problem within tool-integrated RL.

A.1. Proof of Theorem A.1

  • With feedback tokens masked from the loss but retained in context, the tool-integrated GRPO objective has the standard GRPO form over action tokens.The proof obtains the trajectory-level result by summing likelihood changes across actions.Source: §A.1 · pp. 12–13

A.2. Formal Definition of the LLD Death Spiral

  • The LLD Death Spiral requires persistent correct-trajectory likelihood decay, confidence erosion under response similarity, and self-amplifying decay over training.The accompanying analysis reports that low-likelihood incorrect trajectories can exert outsized negative influence on similar correct trajectories.Source: §A.2 · p. 13

B.1.1. TOOL-INTEGRATED REASONING AND AGENTIC LLMS.

  • Tool-integrated RL extends tool use from static instruction-following toward feedback-driven policies for multi-step reasoning across domains.Source: §B.1.1 · p. 14

B.1.2. TRAINING COLLAPSE IN TOOL-INTEGRATED GRPO.

  • Prior work documents GRPO collapse in multi-turn tool-integrated settings, but principled explanations remain limited; this paper identifies LLD as the core mechanism.Related approaches use turn-level dense rewards, which can require ground-truth documents, external judges, or costly tree-based rollouts.Source: §B.1.2 · p. 14

B.2.2. BASELINE METHODS

The baseline suite spans direct answering, retrieval, search-aware reasoning, and alternative RL methods. LLDS variants differ in which likelihood decreases trigger penalties, with action-level gating performing best in the reported comparison.

B.2.2. BASELINE METHODS

  • The baselines include single-pass Direct Inference, fixed-context RAG, search-aware Search-R1, and RL approaches with or without search.Search-o1 is inference-only; TreeGRPO uses branching rollouts but adds complexity, while StepSearch requires curated step-level annotations and CriticSearch an auxiliary judge.Source: §B.2.2 · p. 15

B.4.1. PERFORMANCE COMPARISON AMONG LLDS VARIANTS

  • 0.353 to 0.360 on Qwen2.5-3B-Base and 0.419 to 0.441 on Qwen2.5-3B-Instruct are the overall-average gains from response-level to action-level LLDS.Action-level gating achieves the strongest reported performance across general and multi-hop QA; all variants outperform vanilla GRPO.Source: Table 3 · p. 16, §B.4.1 · p. 16

B.3. LLDS Variants

  • Action-level gating activates only when an individual response action's likelihood decreases, avoiding response-level mismatches that can over- or under-penalize actions.The paper adopts action-level gating as its default LLDS formulation.Source: §B.3 · p. 16

B.4.1. PERFORMANCE COMPARISON AMONG LLDS VARIANTS

  • Token-level LLDS performs worst among the compared variants because penalizing likelihood-reducing tokens even in globally improving responses can overconstrain policy optimization.The comparison includes token-, response-, and action-level variants; action-level gating performs best under the reported NQ+Hotpot settings.Source: §B.3 · p. 15, §B.4.1 · p. 16

B.4.2. RESULTS OF TRAINING WITH NQ DATASET ONLY.

The experiments compare training on NQ alone with broader multi-hop settings and test LLDS across models and training variants. LLDS improves reported QA performance, while answer masking is associated with more valid searches per query and qualitative examples show multi-step tool use.

B.4.3. ADDITIONAL RESULTS ON LLAMA

  • 0.427 overall average is achieved by Llama3.2-3B-Base with LLDS-MA, alongside 0.343 MH-Avg and 0.538 Gen-Avg.LLDS raises the overall average from 0.212 under vanilla GRPO to 0.374; LLDS-MA further promotes multi-turn tool use and multi-hop performance.Source: Table 4 · p. 17, §B.4.3 · p. 17

B.4.2. RESULTS OF TRAINING WITH NQ DATASET ONLY

  • 0.332 to 0.360 is the overall-average increase for Qwen2.5-3B-Base with LLDS when training adds HotpotQA to NQ.The passage reports that the GRPO baseline rises from 0.303 to 0.312 and response-level LLDS from 0.323 to 0.353 under NQ+Hotpot versus NQ-only training.Source: §B.4.2 · pp. 16–17

B.4.4. ENCOURAGING MULTI-STEP REASONING VIA ANSWER MASKING

  • GRPO+LLDS-MA increases valid searches per query beyond 2.0, while vanilla GRPO collapses at 1.0 and GRPO+LLDS remains single-turn.These Qwen2.5-3B-Base behaviors are reported in Figure 9b; the method reduces regularization on answer tokens.Source: §B.4.4 · p. 17, Figure 9 · p. 17

B.5. Behavioral Characteristics

  • Qualitative Qwen2.5-3B-Ins and Qwen2.5-7B-Ins traces show stepwise planning, self-verification, consecutive searches when needed, and concise correct answers.The reported examples contrast with standard GRPO models described as often undergoing premature collapse.Source: §B.5 · p. 18

C. Case Studies of Likelihood Displacement

Paired rollouts illustrate how similar trajectories can receive sharply different group-relative updates and how long, low-likelihood errors can accumulate negative gradients. The discussion connects these examples to tool-integrated GRPO vulnerabilities and recommends monitoring likelihood dynamics.

C. Case Studies of Likelihood Displacement

  • For the NRL question, nearly identical reasoning and search trajectories differ at the final answer: the incorrect rollout truncates “North Queensland Cowboys” to “North Queensland.”The authors say the incorrect rollout pushes negative gradients onto tokens nearly identical in embedding space to the correct rollout's tokens.Source: C. Case Studies of Likelihood… · p. 18, Figure 12 · p. 22, Figure 13 · p. 23
  • The incorrect green-eggs-and-ham rollout begins with a long, low-likelihood reasoning segment, violates tool protocol, and accumulates many negative-gradient summands despite eventually searching.Its correct counterpart is shorter and higher-likelihood, performs one search, and answers “Sam-I-Am”; the authors connect the incorrect trajectory to the LLD Death Spiral.Source: C. Case Studies of Likelihood… · p. 18, Figure 14 · p. 24, Figure 15 · p. 25

D. More Discussion and Guideline

  • Tool feedback tokens are out-of-distribution, and GRPO's single scalar reward can give conflicting signals to shared prefixes across correct and incorrect multi-stage trajectories.The discussion identifies these structural conditions as amplifying likelihood drift and reward–token misalignment in tool-integrated settings.Source: D. More Discussion and Guideline · p. 18
  • Reward can rise while correct-response likelihood declines, so the authors recommend tracking action- and trajectory-level log-likelihood, entropy, and likelihood-ratio spikes.They describe likelihood or entropy curves as early indicators of approaching instability.Source: D. More Discussion and Guideline · p. 19
  • Turn-level hidden rewards are proposed as future work to provide finer credit assignment across tool calls, query formulation, and reasoning steps.The authors present this as a possible complement to likelihood-preserving regularization.Source: D. More Discussion and Guideline · p. 19
Loading 2512.04220v2…