PaperlayerView exact revision on arXiv

Source-linked AI summary

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

Yiran Xu, Yiming Ren, Zicheng Lin, Chufan Shi, Yukang Chen, Dingdong Wang, Tianhe Wu, Junjie Wang, Yujiu Yang, Yu Qiao, Ruihang Chu

arXiv:2605.30789v3cs.LGcs.AI

TL;DR

GRPO needs diverse rollouts, yet token-level randomness can introduce incoherent step-wise noise. S2L-PO instead uses smaller models as policy-level explorers and anneals toward the large learner’s sampling, improving mathematical reasoning while reducing rollout compute.

  • Problem

    GRPO’s learning signals depend on diverse rollouts, but token-level randomness can destabilize long reasoning trajectories and produce incoherent samples.

  • Method

    S2L-PO uses a frozen smaller model to generate exploratory rollouts offline, then progressively transitions toward the larger learner’s on-policy sampling.

  • Results

    Across two model families and four mathematical reasoning benchmarks, S2L-PO consistently improves final performance and sample efficiency over standard GRPO.

  • Takeaways & Limitations

    Smaller models provide structured policy-level diversity that supports more coherent exploration, faster convergence, and reduced rollout compute for larger-model training.

  • Takeaways & Limitations

    Using small-model rollouts throughout training is insufficient: performance can plateau and regress as distribution shift widens between the static explorer and evolving learner.

Abstract

from arXiv · show

We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.

1. Introduction

GRPO depends on diverse rollouts, but token-level randomness can destabilize reasoning; this work identifies smaller models as policy-level explorers and proposes S2L-PO to guide larger learners.

  • GRPO’s advantage estimates depend on diverse candidate rollouts, because homogeneous groups can collapse learning signals.
  • Higher-temperature sampling adds step-wise randomness that can compound across long reasoning chains and produce incoherent trajectories.
  • At large sample counts, smaller models can match or exceed larger models in pass@k despite lower pass@1 performance.A 4B model surpasses an 8B model once k ≥32 and can outperform a 14B model around k ≈200.
  • Policy-level diversity arises from parameter-level compression, shifting the whole policy while preserving temporal correlation and internal consistency.This structured perturbation focuses exploration on reasoning strategies rather than independent local token flips.
  • S2L-PO uses a frozen smaller model for early rollout exploration, then progressively mixes in the larger learner’s sampling until standard on-policy GRPO is recovered.The schedule balances exploration with exploitation while modifying only rollout generation.
  • Across two model families and four mathematical benchmarks, S2L-PO improves final performance and sample efficiency over standard GRPO.Using a 1.7B explorer to guide an 8B model yields an average gain of about 9%.

2. Preliminary

GRPO estimates policy updates from within-group reward comparisons, making rollout diversity central to gradient quality. The paper frames model compression within a family as a structured policy perturbation rather than token-level noise.

  • GRPO samples multiple outputs per query, standardizes their rewards within each group, and updates the policy without an explicit critic.
  • Its PPO-style objective uses importance sampling, clipping, and KL regularization toward a reference policy.
  • Homogeneous candidates can make relative advantages vanish, while temperature-based token noise may produce locally random but globally incoherent trajectories.
  • Within a model family, compression and distillation are treated as inducing a policy-level perturbation in parameter space.Qwen3 models at or below 14B are described as students distilled from larger teachers.
  • The resulting deviation is modeled as a coherent policy shift arising from reduced capacity rather than token-level random perturbation.

3. Method

This section contrasts token-level randomness with policy-level perturbations and explains how their temporal structure affects GRPO exploration and gradient estimation. It then presents S2L-PO, which uses smaller models to provide structured exploratory rollouts for larger-policy training.

  • Token-Level vs. Policy-Level Perturbations: Token-level perturbations add step-wise randomness beyond GRPO’s baseline temperature, typically through softened token sampling.The injected noise sources are independent across steps, although realized tokens remain state-dependent.
  • Token-Level vs. Policy-Level Perturbations: Increasing token-level randomness can make long-range decisions less consistent and weaken cross-step reinforcement in trajectory-level updates.Prefix sharing decays exponentially with sequence position, and long-horizon score contributions can become noisier and less stable.
  • Token-Level vs. Policy-Level Perturbations: Policy-level perturbations model parameter-level compression as a time-invariant shift shared across all rollout steps.The same perturbation produces context-dependent logit changes under one consistently shifted policy.
  • Token-Level vs. Policy-Level Perturbations: Shared parameter perturbations couple trajectory deviations across time, encouraging coherent alternative strategies and stronger cross-step gradient alignment.The analysis links this structure to more coherent trajectory-level GRPO signals and constructive gradient interference under sufficient Hessian alignment.
  • S2L-PO: Small-to-Large Policy Optimization: S2L-PO uses smaller compressed models for exploration while training larger models for exploitation with GRPO.The framework is designed around the contrast between token-level noise and policy-level diversity.

4. Experiment

Experiments evaluate S2L-PO across model families, mathematical benchmarks, diversity metrics, and controlled transition studies. Results show that small-model exploration improves performance and efficiency, with progressive annealing preserving gains beyond the small model’s capacity limits.

  • Method: S2L-PO combines smaller-model rollout actors with progressive annealing into fully on-policy GRPO training.The algorithm gradually changes rollout composition before switching to pure on-policy GRPO.
  • Main Results: S2L-PO reaches higher performance ceilings with faster convergence and reduced rollout computation than standard GRPO.Offline small-model rollouts can be reused, while the learner progressively transitions to its own sampling.
  • Main Results: About 9% average gain is achieved when a 1.7B explorer guides an 8B target model.The reported improvement is relative to the GRPO baseline in the evaluated setting.
  • Diversity Analysis: The 1.7B model achieves 21% higher Unique Answer Ratio than the 14B model, with values of 0.576 versus 0.476.The comparison uses AIME24 with K = 64 rollouts and supports strategy-level diversity across model scales.
  • Diversity Analysis: Removing diversity from small-model rollouts returns performance to the GRPO baseline, supporting policy-level diversity as the relevant factor.The controlled experiment filters small-model rollouts to match the large model’s diversity metrics.
  • Ablation Study: Small-only rollouts produce rapid initial gains but later plateau and regress, failing to match progressive annealing’s peak performance.The paper attributes this to widening distribution shift between the static small-model explorer and evolving large-model learner.
  • Ablation Study: Progressive annealing outperforms abrupt switching, while shortening the transition reduces training stability and the performance ceiling.Gradual handover avoids the instability associated with a sudden training-distribution shock and gives the learner time to absorb exploration signals.

5. Related Work

GRPO depends on diverse candidate trajectories, but common token-level exploration can destabilize training. S2L-PO instead uses policy-level diversity from smaller models and progressively transitions to on-policy learning.

  • The framework compares progressive transition with abrupt two-phase switching and evaluates annealing schedules that reach zero small-model rollouts after 8 versus 5 steps.
  • GRPO relies on diverse candidate trajectories because homogeneous groups weaken advantage estimation and gradient signals.
  • Token-level methods such as higher temperature, top-p sampling, and entropy regularization may produce local, step-wise randomness rather than structured trajectory diversity.
  • S2L-PO reuses a smaller same-family model to introduce policy-level diversity at near-zero cost.
  • Progressive annealing smoothly shifts from small-model exploration to on-policy learning, avoiding the performance plateau of purely offline approaches.

6. Conclusion

S2L-PO enhances GRPO by using smaller models as structured explorers for larger learners. Its policy-level diversity and annealing strategy improve mathematical reasoning while reducing rollout compute and accelerating convergence.

  • S2L-PO uses smaller models as structured explorers for larger learners in GRPO.
  • Parameter-level compression gives smaller models policy-level diversity that supports more coherent exploration and improved learning signals than token-level randomness alone.
  • Annealing balances exploration and exploitation during training.
  • S2L-PO achieves gains on mathematical reasoning tasks while reducing rollout compute and accelerating convergence.

Impact Statement

The study uses publicly available open-source datasets for research and does not generate or collect new multimodal content.

  • The work relies exclusively on publicly available open-source datasets widely used and validated in prior academic research.
  • No new text, images, audio, or video content is generated or collected.
  • The datasets and trained models are used strictly for research, without commercial deployment or application.

A. Reproducibility Statement

The authors describe an open-source release and provide methodological, experimental, training, and hyperparameter details intended to support reproduction of the reported results.

  • The complete project codebase will be released as open-source software.
  • The methodology and algorithmic design are described in Section 3, with experimental setup specified in Section 4.1.
  • Training protocols, implementation details, and key hyperparameter configurations are provided in the appendix.
  • The authors state that these materials are sufficient to reproduce all reported experimental results.

B. Use of Large Language Models

The manuscript reports limited use of language models for language editing and specifies the core GRPO training and rollout-scheduling procedures.

  • B. Use of Large Language Models: The authors used a large language model only for grammar, clarity, and readability edits.They state that research ideas, methods, experiments, analyses, and conclusions were produced and verified by the authors.
  • B. Use of Large Language Models: Training used fixed GRPO hyperparameters, including batch size 1024 and response length 4096 tokens.The actor used a learning rate of 1 × 10−6, PPO-style updates, mini-batches of 16, and per-GPU micro-batches of 2.
  • B. Use of Large Language Models: The progressive schedule mixes offline-source and current-policy trajectories while linearly reducing the offline contribution to zero.This provides low-cost early exploration and ends with optimization under the learner’s own on-policy distribution.

E. Formal Proofs for Theoretical Analysis

The appendix formalizes how token-level and policy-level perturbations affect trajectory prefix matching and cross-step covariance. Its analysis supports stronger, temporally constructive gradient signals under policy-level perturbations when alignment is sufficient.

  • E. Formal Proofs for Theoretical Analysis: The formal analysis distinguishes token-level from policy-level perturbations using cross-step covariance bounds for GRPO gradients.The appendix frames the theoretical comparison around how perturbations influence gradient signals across decoding steps.
  • E.1. Notation and Setup: A trajectory o is compared with a deterministic reference trace o⋆ through the prefix-match indicator M_t.The setup also defines per-step score functions u_i,t and scalar projections z_i,t bounded by B.
  • E.1. Notation and Setup: Under token-level perturbation with per-step deviation probability at least p, prefix matching decays no slower than (1 − p)^t.The result follows by recursively multiplying conditional prefix-match probabilities.
  • E.1. Notation and Setup: The policy-level analysis perturbs score projections through the Hessian term v⊤H_tδθ, assuming zero-mean parameter perturbations with covariance Σ_δ.The lower-bound argument uses the perturbation covariance γ and standard-temperature prefix divergence.
  • E.1. Notation and Setup: The covariance expansion cancels zero-mean cross terms and yields Cov(ẑ_t, ẑ_s) = Cov(z_t, z_s) + γ + O(∥δθ∥^3).The lower bound becomes positive when Hessian alignment is strong and standard-temperature prefix divergence is moderate.
  • E.1. Notation and Setup: For same-family distilled models, structured capacity reduction is argued to align Hessian directions across decoding steps, unlike increasingly vacuous token-level bounds at high temperature.The appendix treats δθ as a fixed vector in practice and contrasts it with random token-level perturbations.
  • E.2. Summary: Token-Level vs. Policy-Level Signal Growth: The summary contrasts a token-level upper bound that becomes vacuous with a policy-level lower bound that can remain positive and reinforce gradients across time.A shared, time-invariant policy-level signal makes per-step gradient contributions reinforce constructively rather than cancel like a random walk.

F. Limitations

The empirical evaluation is limited in model-family and benchmark coverage, especially beyond mathematical reasoning with verifiable rewards.

  • F. Limitations: Computational constraints prevented exhaustive evaluation across prominent model families and benchmark categories.The authors have not validated S2L-PO on tasks using non-verifiable or open-ended rewards.
  • F. Limitations: The method’s capability boundary across broader model scales, task domains, and modalities remains unexplored.This scope boundary is identified as future work rather than an established failure mode.
Loading 2605.30789v3…