PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao

arXiv:2608.27420v1cs.CL

TL;DR

RLVR can collapse policy entropy and narrow reasoning coverage, motivating methods that address neglected cross-model, non-parametric exploration. The paper uses weak-model partial prefixes to steer target-model continuation, and reports consistent improvements over vanilla RLVR with larger gains at higher pass@k.

  • Problem

    RLVR often reduces policy entropy and exploration coverage, while cross-model non-parametric perturbation remains neglected.

  • Method

    The framework injects partial reasoning trajectories from smaller auxiliary models as prefixes that steer the target model toward under-explored reasoning regions.

  • Results

    Across multiple mathematical reasoning benchmarks, the method consistently outperforms vanilla GRPO, with increasingly pronounced pass@k improvements as k increases.

  • Takeaways & Limitations

    Cross-model prefix guidance provides a simple input-trajectory mechanism for mitigating entropy collapse and improving reasoning coverage.

  • Takeaways & Limitations

    The improvement is sensitive to pre-configured hyperparameters, and the framework has so far been evaluated only on mathematical reasoning tasks.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

1 Introduction

RLVR strengthens reasoning by optimizing verifier-rewarded responses but can sharply reduce entropy and narrow exploration. The paper introduces cross-model prefix guidance, showing broader reasoning coverage and stronger gains at larger pass@k.

  • Motivation: RLVR increases high-reward response probabilities through verifier feedback but existing evaluations often emphasize pass@1.The supplied background describes sequence-level optimization from external verification and notes the emphasis on pass@1 evaluation.
  • Entropy Collapse: Early RLVR training sharply decreases policy entropy, limiting exploration and reducing opportunities to discover alternative correct reasoning paths.The model becomes over-confident in established trajectories, while other work finds RLVR mainly strengthens already-established reasoning paths.
  • Prefix-Guided RLVR: Cross-model generational diversity significantly affects RLVR exploration dynamics, motivating non-parametric perturbation alongside internal training changes.Prior approaches modify entropy regularization, policy gradients, or reward and advantage design within the training objective.
  • Prefix-Guided RLVR: The framework steers target models toward under-explored reasoning regions using partial trajectories generated by auxiliary models.Its cross-model prefixes exploit distributional differences caused by distinct pretraining data, architectures, and optimization dynamics.
  • Results: Across multiple reasoning benchmarks, prefix guidance consistently improves over vanilla GRPO, with increasingly pronounced gains as pass@k grows.The reported pattern indicates improved reasoning coverage and alleviated diversity collapse relative to vanilla RLVR.
  • Results: Analyses suggest cross-model prefix guidance mitigates entropy collapse even when auxiliary prefixes provide little or occasionally misleading guidance.The paper presents this as a simple mechanism for enhancing larger-model RLVR training.

2 Preliminaries on GRPO and Entropy

GRPO optimizes verifier-scored response groups with normalized advantages, PPO-style clipping, and a reference-model KL penalty. Its increasing confidence in preferred trajectories creates an entropy-collapse trade-off that limits exploration.

  • Group Relative Policy Optimization (GRPO): GRPO compares multiple sampled responses for one prompt to estimate advantages without training a separate value model.Each response receives a verifier or reward-model score, and the resulting group-normalized advantage is assigned across its tokens.
  • GRPO Objective: The objective uses a PPO-style clipped surrogate loss with a KL penalty against a frozen reference policy.The token-level importance ratio compares current and rollout policies; epsilon clips updates and beta controls KL regularization strength.
  • Entropy Collapse: GRPO increases confidence in high-reward reasoning trajectories, often reducing generation diversity and concentrating probability mass on erroneous paths for some queries.This reflects an exploitation–exploration trade-off under repeated sampling.
  • Entropy: Policy entropy measures uncertainty over the next token after preceding tokens, approaching zero when one token’s probability approaches one.Lower entropy indicates reduced uncertainty and weaker exploration.
  • Step-Level Entropy: Reasoning trajectories are segmented into newline- or period-delimited steps, with each step represented as a token sequence.Step-level entropy is defined as the average token-level entropy within each reasoning step.
  • Step-Level Entropy: The entropy analysis uses preceding reasoning steps and preceding within-step tokens as conditioning context for the target policy.The parameter theta denotes the target policy being trained, not the auxiliary model generating prefixes.

3 Motivation: Cross-Model Prefixes Increase Policy Uncertainty

Prefixes generated by other models make the target model less certain during early continuation, creating an opportunity to broaden exploration. This relative uncertainty persists after GRPO training even as entropy decreases overall.

  • Cross-Model Uncertainty: Cross-model prefixes make the target model less adaptable to autonomous continuation and increase uncertainty when selecting subsequent tokens.The observed shift away from the target model’s familiar generation distribution exposes under-explored regions.
  • Before and After GRPO: Before GRPO, Qwen2.5-7B has substantially higher early step-level entropy after Gemma-2-2B prefixes than after its own prefixes.The comparison uses prefixes generated by Gemma-2-2B versus Qwen2.5-7B itself.
  • Before and After GRPO: Entropy gradually converges across prefix types as generation proceeds, indicating adaptation to unfamiliar prefixes after several reasoning steps.The early uncertainty provides an opportunity to broaden the exploration space.
  • Before and After GRPO: After GRPO training, entropy decreases overall but cross-model prefixes still produce higher uncertainty during early reasoning steps.The relative prefix effect remains similar despite entropy collapse.
  • Implication: Injecting auxiliary-model trajectories perturbs the target away from over-confident paths and encourages alternative trajectories as non-parametric steering.The proposed mechanism is intended to mitigate diversity collapse during RLVR training.

4 Methodology

Prefix-completion RLVR conditions the target model on truncated reasoning trajectories from auxiliary models, then trains it to complete the solution. Prefixes are selected to preserve unfamiliar, high-entropy reasoning states while mixing prefix-conditioned and question-only training.

  • 4.1 Prefix-Completion RLVR: Prefix-completion RLVR trains the target model to complete a solution after an auxiliary model supplies a partial reasoning trajectory.The auxiliary trajectory is truncated into a prefix, and the target model generates the remaining suffix.
  • 4.1 Prefix-Completion RLVR: The approach exposes the target model to unfamiliar reasoning states, encouraging exploration beyond its dominant generation distribution.Successful continuations can receive positive reward, increasing the likelihood of alternative trajectories.
  • 4.2 Entropy-Based Prefix Truncation: Prefix trajectories are truncated before the largest adjacent step-level entropy drop, retaining the high-entropy region for exploration.Entropy is computed with the target base model after auxiliary solutions are segmented into reasoning steps.
  • 4.2 Entropy-Based Prefix Truncation: The truncation strategy avoids cutting through reasoning steps and computes the truncation point once because the entropy trend remains largely stable during training.The retained prefix preserves the region judged to provide the strongest exploration signal.
  • 4.3 Mixed Training: Mixed training uses prefix-completion RLVR with probability p and question-only RLVR with probability 1 − p.The main experiments set p = 0.2 to balance prefix-driven exploration with solving from the original question.

5 Experiments

The experiments train Qwen2.5-7B and Qwen2.5-Math-7B with prefixes from LLaMA-3.2-1B and Gemma-2-2B, evaluating across six mathematical reasoning benchmarks. Prefix guidance consistently improves performance, with larger gains at higher pass@k and robust cross-model generalization.

  • 5.1 Training Setup: The experiments train Qwen2.5-7B and Qwen2.5-Math-7B using LLaMA-3.2-1B and Gemma-2-2B as auxiliary prefix generators.The small models are selected to provide diverse reasoning styles while limiting computational cost.
  • 5.1 Training Setup: Evaluation covers AIME 2024, AIME 2025, AMC 2023, MATH 500, Minerva, and Olympiad Bench using average pass@k across benchmarks.Training uses the 7500-problem MATH training set and GRPO.
  • 5.2 Main Results: Prefix-guided training consistently improves larger target models across all six reasoning benchmarks compared with standard RLVR.The main results are summarized in Table 1.
  • 5.2 Main Results: Gains become more pronounced as k increases, indicating broader reasoning coverage than standard RLVR training.This pattern is reported as alleviating the narrowing of reasoning coverage caused by standard RLVR.
  • 5.2 Main Results: Gemma-2-2B prefixes generalize robustly to Qwen2.5-7B and Qwen2.5-Math-7B target models.LLaMA-3.2-1B prefixes are particularly effective at preserving or improving pass@128 while maintaining comparable pass@1 accuracy.

6 Analysis

Prefix-guided RLVR maintains exploration by perturbing the target model with heterogeneous partial trajectories, while prefix quality and injection probability shape the resulting trade-offs. The analyses indicate that trajectory discrepancy, rather than faithful guidance, is the central utility of auxiliary prefixes.

  • Dynamic Exploration via Distributional Perturbation: Higher prefix injection probabilities maintain higher policy entropy, especially early in training, delaying premature policy collapse.Entropy later converges toward a level similar to vanilla GRPO, but the early high-entropy phase supports broader exploration.
  • Exploration Cost and Informative Reward Signals: External prefixes diversify sampled trajectories and produce more heterogeneous reward patterns, mitigating GRPO’s sparse relative-advantage signals.The effect is particularly relevant during the early stages of training, when same-group samples often receive identical rewards.
  • Prefix Quality Assessment: Small-model prefixes often provide little or misleading guidance, showing that auxiliary models act as perturbation sources rather than expert teachers.The prefix assessment found that most prefixes from small models offered no substantial guidance or incorrect directions.
  • Prefix Quality Assessment: Low-quality prefixes do not prevent recovery: the target model can still derive correct solutions within a limited number of samples.The benefit therefore comes primarily from exposing the target model to unfamiliar reasoning states rather than from prefix correctness.
  • Prefix Quality Assessment: Semantic guidance accuracy and perturbation utility are decoupled: homologous Qwen prefixes are more accurate but yield weaker gains than non-homologous alternatives.Gemma-2-2B produces higher-fidelity prefixes than LLaMA-3.2-1B, while Qwen2.5-1.5B and Qwen2.5-7B provide higher semantic quality without comparable improvements.
  • Ablation Study: Entropy-based truncation outperforms random step-level truncation, while excessive injection at p = 1.0 can degrade performance through training–evaluation inconsistency.Replacing the smaller generator with a stronger homologous model also produces no noticeable gain over vanilla GRPO.

7 Conclusions

The work uses weaker-model partial solutions to perturb RLVR trajectories and preserve exploration. Across mathematical reasoning benchmarks, this broadens search and improves pass@k, despite weaker-model outputs often being incorrect or misleading.

  • Conclusions: The method provides a trajectory-level approach to using small models for exploration rather than requiring them to supply correct guidance.Its stated contribution is preserving generative diversity during RLVR training.
  • Conclusions: Weaker-model partial solutions raise target policy entropy and broaden the search space, enhancing pass@k across mathematical reasoning benchmarks.The framework operates at the input-trajectory level and aims to preserve generative diversity during RLVR training.

Limitations

The reported benefits are sensitive to pre-configured hyperparameters and have so far been evaluated only on mathematical reasoning tasks.

  • Limitations: The improvement is delicate with pre-configured hyperparameters and remains under-explored beyond mathematical reasoning, including logic and code generation.These conditions bound the current evidence for the framework’s generality.

A.1 Reasoning with High-level Thought Guidance

Prior guidance methods mainly simplify problems or select exemplars for frozen-model inference, leaving abstract problem-solving guidance in on-policy RL under-explored.

  • A.1 Reasoning with High-level Thought Guidance: Task decomposition and exemplar selection reduce complexity but rely heavily on manual engineering and primarily enhance frozen models during inference.The passage contrasts these approaches with the under-explored problem of integrating abstract guidance into on-policy RL training.

A.2 Exploration Mechanism and Learning Dynamics

The paper distinguishes prompt templates for 7B target models, smaller prefix-generation models, and prefix-quality assessment. These templates support the framework’s use of smaller-model outputs as external reasoning prefixes.

  • Prompt Templates: 7B target models use the original chat template with a SimpleRL-style prompt.The specified 7B models are Qwen2.5-7B and Qwen2.5-Math-7B.
  • Prompt Templates: Smaller models use a separate prompt for high-quality prefix generation.The smaller models include Gemma-2-2B, LLaMA-3.2-1B, and Qwen2.5-1.5B.
  • Prompt Templates: Prefix quality is assessed with a dedicated prompt template.The assessment prompt presents the evaluator as an expert mathematical evaluator analyzing an incomplete reasoning prefix.
Loading 2608.27420v1…