PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Exploration from Optimization in RLVR

Saif Punjwani, Micah Goldblum

arXiv:2610.10536v1cs.LGcs.AIcs.CL

TL;DR

RLVR’s discovery of new reasoning strategies depends on exploration, but applying novelty pressure directly to a model can degrade capabilities outside the narrow reward signal. ExpDis separates exploration from optimization by distilling filtered explorer trajectories into a student trained without novelty. Across mathematical reasoning benchmarks, it outperforms DAPO at matched compute, with further gains from parallel explorers and repeated rounds.

  • Problem

    RLVR can reinforce only sampled solutions, so discovering new reasoning strategies depends on exploration, while its narrow verifiable reward may not detect damage to other capabilities.

    Source: §1 · p. 1

  • Method

    ExpDis trains separate novelty-rewarded explorers, filters their trajectories, and distills them into a student trained with correctness-only RLVR.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    ExpDis outperforms DAPO at the same compute on mathematical reasoning benchmarks, and parallel explorers and multiple rounds yield further gains.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    Decoupling lets exploration be scaled aggressively while filtered trajectories pass discoveries to a student without transferring exploration pressure.

    Source: §1 · p. 2, §6 · p. 9

    The observed capability drops are modest at the tested scale; larger-scale degradation from directly applied exploration bonuses is a speculation.

    Source: §5 · p. 8

Abstract

from arXiv · show

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.

1 Introduction

RLVR can discover new reasoning strategies only through what a model samples, but novelty-driven exploration can damage capabilities that its narrow reward does not supervise. ExpDis separates exploration from optimization, and the paper reports stronger mathematical reasoning without the general-capability degradation seen when novelty is added directly to DAPO.

1 Introduction

  • RLVR can reinforce new solutions absent from training data, but discovery depends on exploration because it can reinforce only sampled solutions.Source: §1 · p. 1
  • A novelty bonus applied directly to the trained model can degrade its behavior because verifiable rewards supervise only a narrow slice of its knowledge.Source: §1 · p. 1
  • ExpDis trains a separate novelty-rewarded explorer and transfers only filtered trajectories to a student trained with standard RLVR.The student never receives the novelty reward, while the two policies communicate through filtered trajectories rather than parameters.Source: §1 · p. 1
  • Parallel explorers and repeated rounds scale exploration breadth and depth, with the composition yielding sustained gains under wall-clock-matched experiments.Source: §1 · p. 2
  • At the same compute, ExpDis outperforms DAPO in pass@1 and pass@64 on mathematical reasoning benchmarks across three base models.The evaluation uses Qwen3-1.7B, Qwen3-4B, and Ministral-3-3B-Instruct-2512.Source: §1 · p. 2
  • ExpDis is compatible with different exploration mechanisms, including mechanisms too aggressive to apply directly to the deployed model.Source: §1 · p. 2

2 Related Work

Related work examines whether RLVR discovers new solutions, how exploration pressure can be increased, and how stronger or self-generated reasoning traces can be distilled. ExpDis differs from prior decoupling approaches by using a separate explorer network and passing only filtered trajectories to the student.

2 Related Work

  • Prior studies disagree on whether RLVR expands the set of solvable problems or mainly sharpens existing solution distributions.Some report gains in majority-vote accuracy without pass@k gains, while others report expansion through prolonged training or composition of existing skills.Source: §2 · p. 2
  • Methods addressing entropy collapse include changing clipping, regulating entropy, or adding exploration bonuses, including classical count-based, curiosity, and RND bonuses.Source: §2 · p. 3
  • Reasoning-trace distillation spans expert iteration, stronger-model distillation, self-training on filtered samples, and multi-teacher distillation.ExpDis instead trains its explorer and student from the same base model, with exploration as the distinct objective.Source: §2 · p. 3
  • Earlier RL methods separated exploration from exploitation, while LLM approaches include discarded auxiliary policies and self-distillation of correct rollouts.R2PO and ICE share weights between explorer and student; ExpDis uses a separate network and transfers filtered trajectories.Source: §2 · p. 3

3 Method

ExpDis trains a novelty-seeking explorer, filters its trajectories, and distills them into a student optimized without a novelty bonus. It scales through parallel explorers and iterative rounds, with prompt shards and annealed novelty pressure across rounds.

3 Method

  • ExpDis trains an explorer with a novelty bonus, filters its trajectories for quality and correctness, and distills the selected traces into a student.Source: §3 · p. 3
  • Every correct explorer rollout receives more reward than any incorrect one, and the novelty bonus applies only when the correctness reward is positive.Applying novelty to all rollouts can destabilize training because novel low-quality outputs can raise reward.Source: §3 · p. 3
  • The trajectory filter requires a correct final boxed answer, termination within a fixed token budget, and no repetition loops.After filtering, the method keeps only the shortest trajectory per problem and caps the set at 500 per round.Source: §3 · p. 4
  • The student is first SFT-trained on filtered traces, then trained with DAPO using correctness reward only.This combines off-policy trajectory distillation with on-policy student training.Source: §3 · p. 4

3.1 Parallel Explorer Policies

  • Parallel independent explorers pool trajectories before filtering and distillation, increasing the diversity of traces available to the student.Compute is divided among explorers, without additional wall-clock time over a single explorer.Source: §3.1 · p. 4

3.2 Iterative Exploration-Distillation

  • Each iterative round initializes explorers and a student from the previous student, uses a separate prompt shard, and lowers novelty weight from 0.75 to 0.25 over four rounds.The schedule applies stronger novelty pressure early and lower pressure later to consolidate discovered solutions.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

4 Experimental Setup

The experiments use three base models, RND novelty, a fixed RL budget, and math and general-capability benchmarks. Figure 3 summarizes the same-budget comparison of ExpDis with baselines.

4 Experimental Setup

  • Experiments use Qwen3-1.7B, Qwen3-4B, and Ministral-3-3B-Instruct-2512, with RND as the primary novelty bonus.RND uses predictor networks trained to match a frozen random target on language-model hidden-state features.Source: §4 · p. 5
  • The novelty-weight sweep found performance roughly peaks at λ=0.5, which is used for the experiments.The standard math verifier supplies the correctness reward.Source: §4 · p. 5
  • Evaluation covers seven math benchmarks and focuses on the mean of five because GSM8K and, for Qwen3, AMC23 are near saturation.The reported unbiased pass@k estimates use up to 64 completions per problem, with fewer for AMC23 and GSM8K.Source: §4 · p. 5
  • General-capability evaluation covers knowledge, logical reasoning, and instruction following as a proxy for abilities outside the training data and reward.Source: §4 · p. 5
  • Figure 3 reports that ExpDis outperforms all baselines under the same RL compute budget, including DAPO and single-round ExpDis.Source: Figure 3 · p. 6
  • The compute-matched protocol uses 300 RL steps, and ExpDis divides updates between explorer and student stages.The comparisons include correctness-only GRPO, Dr. GRPO, and DAPO, as well as DAPO with novelty and DAPO trained for four times as many steps.Source: §4 · pp. 5–6

5 Results

ExpDis outperforms DAPO across three models, while compute allocation, trajectory filtering, and repeated exploration shape the performance and diversity trade-offs. The results also show how student policies evolve across rounds and where novelty-driven exploration can affect prior capabilities.

5 Results

  • Across all three models, ExpDis beats DAPO in mean mathematical accuracy, and all runs beat 4×-step DAPO in average performance and pass@k through 64 generations.Source: §5 · p. 6
  • At 300 RL updates, adding rounds improves pass@1 more than adding explorers, while combining both gives the largest pass@1 and pass@64 gains; gains saturate and show regression signs beyond three explorers and four rounds.Source: Figure 4 · p. 6, §5 · p. 7
  • Adding novelty to DAPO raises lexical and answer diversity but slightly lowers math performance; ExpDis can push explorer diversity much higher without harming the student.Source: §5 · p. 7, Figure 5 · p. 7
  • On Qwen3-1.7B, unfiltered-trajectory distillation followed by RL and filtered-trajectory distillation without subsequent RL each score slightly below DAPO on AIME24; shortest-valid-trajectory filtering beats keeping every correct trajectory.Replacing RND with kNN-based or elliptical novelty lowers mean accuracy by at most 1.6 points, while both variants still outperform DAPO.Source: §5 · p. 7
  • DAPO with RND novelty falls below the base model on prior-capability benchmarks despite improving on math, whereas ExpDis obtains exploration benefits without that degradation.Observed drops are modest at this scale; larger-scale accumulated degradation is speculative.Source: Figure 6 · p. 8, §5 · p. 8
  • Across rounds, student diversity and accuracy rise together; by round four students nearly close the explorers’ semantic-diversity gap, while lexical diversity declines and reasoning faithfulness typically falls mildly.The student inherits broader solutions as more correct trajectories enter the distillation set.Source: §5 · pp. 8–9, Figure 7 · p. 8

6 Discussion

ExpDis moves exploration into a separate policy, allowing it to try strategies that standard RLVR’s need to remain correct constrains. The discussion argues that this separation may matter more as training scales and may support exploration beyond mathematical reasoning.

6 Discussion

  • Standard RLVR limits exploration by requiring the policy to remain correct, while ExpDis lets a separate explorer try new ideas and pass only filtered reasoning traces to the student.The explorer is not deployed and can be rewarded for novelty, pushed to extremes, or discarded after one round.Source: §6 · p. 9
  • As models master familiar strategies, problems with rare or unfamiliar correct solutions may make aggressive exploration increasingly valuable, including in agentic tasks, theorem proving, and open-ended science.The authors present this as an expectation and identify these settings as possible extensions beyond mathematical reasoning.Source: §6 · p. 9
  • ExpDis focuses on one novelty bonus and explorers sharing the student’s architecture, while different novelty signals or sampling strategies could let explorers specialize across the solution distribution.The authors speculate that methods too destabilizing for direct use on a trained model may become usable when explorers only need to produce useful traces.Source: §6 · p. 9

A Training & Hyperparameters

The appendix reports shared RL training choices and compute allocations, then specifies the evaluation sampling and pass@k estimator used for the compute-budget analysis.

A Training & Hyperparameters

  • All methods use dynamic sampling, overlong filtering, clip-higher, no KL penalty, and Dr. GRPO loss normalization by a fixed constant rather than token count.The shared settings are listed in Table 1; Figure 8 shows continued-training dynamics for Qwen3-1.7B under ExpDis with λ=0.5.Source: A Training & Hyperparameters · p. 13, Table 1 · p. 13, Figure 8 · p. 13
  • Each RL stage uses on-policy updates, reloading trainer weights after every step; batches contain 4 prompts × 16 samples for the next gradient step.Table 2 allocates each ExpDis run 200 explorer and 100 student updates, with 19,200 selected rollouts; parallel explorers run concurrently for wall-clock-matched rows, except DAPO (4× steps).Source: A Training & Hyperparameters · p. 13, Table 2 · p. 14

A.1 Compute budget

  • Each run uses one TPU v5litepod-64 slice, with 4 hosts training via FSDP and 12 serving vLLM rollouts in bf16 at a 40,960-token context.The compute setup is described as 16 hosts per slice.Source: §A.1 · p. 14
  • Evaluation samples 32 completions per AMC23 problem and 8 per GSM8K problem, and estimates pass@k unbiasedly from correct counts across N problems.The estimator uses n samples per problem, with c_i correct completions for problem i, averaged over N problems.Source: Table 3 · p. 14, §A.2 · p. 14

B Extended Results

The extended results provide full pass@k results across models and methods, with a mean pass@k table covering the five primary benchmarks.

B Extended Results

  • The paper reports pass@k for all models and methods in Figure 9 and Table 4.The passage identifies both visuals as the locations for those results.Source: B Extended Results · p. 14, Figure 9 · p. 15
  • Figure 9 presents the full pass@k results.The figure caption identifies the results as full pass@k results.Source: Figure 9 · p. 15

C Novelty Bonus Analysis

The analysis finds that adding an RND novelty bonus directly to DAPO degrades mathematical reasoning, while the downstream ExpDis student does not inherit explorer degradation. It also examines alternative novelty metrics and the effects of increasing rounds.

C Novelty Bonus Analysis

  • With an RND novelty bonus, DAPO’s mean mathematical reasoning performance degrades relative to DAPO without the bonus.The analysis reports three runs with different seeds and studies bonus weights from λ=0.25 to 2; Figure 10 addresses accuracy degradation.Source: C Novelty Bonus Analysis · p. 15, Figure 10 · p. 15
  • Even when novelty bonuses hurt the explorer, the downstream student does not inherit that degradation, including at λ=2 where explorer pass@k regresses.For Qwen3-1.7B, Figure 11 indicates λ=0.5 roughly maximizes downstream student performance, with several λ values outperforming DAPO.Source: C Novelty Bonus Analysis · p. 16, Figure 11 · p. 16

C.1 Novelty metric (rnovelty)

  • The alternative-metric experiment tests two single-round Qwen3-1.7B explorers using mean-pooled, L2-normalized final-layer trajectory features.The kNN bonus uses mean cosine distance to 16 nearest neighbors among the 4,096 most recent correct completions; elliptical novelty uses a ridge-regularized running covariance over past correct completions.Source: §C.1 · pp. 16–17
  • Table 5 ablates the novelty metric, while Figures 12 and 13 report DAPO novelty-bonus degradation and the impact of increasing rounds, respectively.The supplied captions identify the scope of these visuals without giving their detailed outcomes.Source: Table 5 · p. 17, Figure 12 · p. 17, Figure 13 · p. 17

D Diversity Analysis

The section defines several complementary measures of generation diversity, spanning lexical variation, distinct final answers, token uncertainty, and reasoning-approach clusters.

InterDistinct-4

  • InterDistinct-4 measures the ratio of distinct 4-grams to total 4-grams across a problem’s generation pool.The metric is averaged across problems.Source: D Diversity Analysis · p. 17

AnswerDistinct@n

  • AnswerDistinct@n counts distinct extracted final answers among n generations for each problem, then averages across problems.Source: D Diversity Analysis · p. 17

Token entropy

  • Token entropy is the mean per-token entropy across eight sampled rollouts for each of 30 AIME24 problems.The evaluation sampler is used.Source: D Diversity Analysis · p. 18

Semantic diversity

  • Semantic diversity is the number of clusters of reasoning traces, normalized by the number of generations.Clusters are based on pairwise similarity scores from the Li et al. [21] classifier.Source: D Diversity Analysis · p. 18

E Ablations

The ablations examine quality filtering, the combination of student distillation and RL, λ scheduling, and training-data choice. They show that filtering and the full student-training sequence matter, while the tested data pools differ in their benchmark results.

Quality filtering

  • 49.47% with quality filtering exceeds 48.95% from pooling every correct trajectory and 48.93% from a single explorer.Source: E Ablations · p. 18

Student distillation vs student RL

  • 52.76% on AIME24 requires filtering followed by RL, versus 49.79% for unfiltered distillation plus RL, 49.84% for filtered distillation without RL, and 50.05% for DAPO.Table 8 reports the ablation.Source: E Ablations · p. 18, Table 8 · p. 20

Annealing λ

Training data

  • The full 40K DeepScaleR pool performs better on every benchmark than DAPO-Math-17K or its 17K DeepScaleR subset, which perform similarly without a consistent winner.Each run samples about 1,200 prompts; the main results use DAPO-Math-17K.Source: E Ablations · p. 18, Figure 16 · p. 20, Table 9 · p. 20
Loading 2610.10536v1…