PaperlayerView exact revision on arXiv

Source-linked AI summary

ExpRL: Exploratory RL for LLM Mid-Training

Violet Xiang, Amrith Setlur, Chase Blagden, Nick Haber, Aviral Kumar

arXiv:2606.17024v1cs.LG

TL;DR

Sparse-reward RL for LLM reasoning is bottlenecked when base models rarely sample productive paths. ExpRL uses on-policy RL with dense, reference-guided rewards to build broader coverage before sparse-reward RL, yielding stronger initializations and improved downstream RL across answer-based math benchmarks.

  • Problem

    Sparse-reward RL is limited when base models rarely sample productive reasoning paths, motivating broader coverage beyond manually taught primitive skills.

  • Method

    ExpRL uses on-policy RL with dense reference-guided rewards to reinforce partial progress and useful reasoning behaviors before sparse outcome-reward RL.

  • Results

    Across answer-based math reasoning benchmarks, ExpRL yields stronger RL-ready initializations and improves downstream sparse-reward RL.

  • Takeaways & Limitations

    ExpRL can serve as a reference-answer-based priming interface that broadens productive reasoning coverage before subsequent sparse-reward RL.

  • Takeaways & Limitations

    ExpRL requires reference solutions or similar auxiliary information, which may be unavailable in domains where good references are difficult to obtain.

Abstract

from arXiv · show

Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through \emph{mid-training} on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: \emph{RL-based mid-training} using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as \emph{reward scaffolds}: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.

1. Introduction

ExpRL uses hidden reference solutions as reward scaffolds for dense, on-policy RL mid-training rather than as imitation targets. It improves reasoning coverage and RL initialization, with gains in pass@1, pass@k, attempt diversity, and productive reasoning behaviors across math and mixed-domain settings.

  • Motivation: RL success depends on base-model coverage, because sparse rewards provide little learning signal when useful reasoning paths are rarely sampled.Initialization is therefore a central bottleneck for scaling RL-based reasoning.
  • Motivation: Mid-training aims to increase probability mass on productive reasoning attempts by improving primitive skills and their composition across hard problems.The targeted skills include decomposition, verification, backtracking, and self-correction.
  • Method: ExpRL hides reference solutions from the policy and uses them to construct problem-specific rubrics that provide dense rewards for on-policy reasoning traces.The policy samples only from the original problem prompt, preserving exploration while providing richer feedback than sparse correctness rewards.
  • Method: ExpRL-Outcome scores full rollouts densely, while ExpRL-Process assigns dense rewards to intermediate prefixes to credit partial progress and useful solution structures.These rewards can reinforce promising decompositions and correct intermediate reductions even before the model fully solves a problem.
  • Results: Compared with SFT, sparse-reward GRPO, and self-distillation, ExpRL produces a stronger Stage-I policy and better initialization for subsequent sparse-reward RL.Improvements extend to pass@1, pass@k, reasoning-attempt diversity, verification, self-correction, and backtracking, with additional mixed-domain evaluation.

2. Preliminaries, Definitions, and Notation

This section defines the mid-training setup, the goal of expanding productive reasoning coverage for downstream RL, and the metrics used to evaluate exploratory capability. It also specifies the dense-reward RL algorithms and token-level process-reward updates used in the framework.

  • Problem setup: The setup uses a mid-training dataset of problems paired with step-by-step reference solutions, while the policy is trained to broaden productive reasoning paths for a downstream dataset.The reference solutions are typically human-written and may differ substantially from LLM-generated reasoning traces.
  • Evaluating exploratory capabilities: The primary downstream metric is pass@1 after Stage-II sparse-reward RL, while pass@k measures the probability of obtaining at least one correct rollout among k independent attempts.Pass@k is also measured on the mid-training dataset to diagnose exploratory capability and serves as a proxy for coverage under sampling.
  • RL priming for downstream RL: RL priming prepares a base model for later binary-reward RL by helping it compose primitive skills into productive paths for difficult downstream problems.A model is considered well primed when its downstream pass@k is large.
  • RL algorithms: Outcome-level RL uses GRPO with normalization across n rollouts per problem, whereas process-level rewards use a REINFORCE update over sampled rollouts.The method applies dense rewards at both the final outcome and intermediate process points.
  • RL algorithms: With process rewards, advantages are computed at the token level because rewards and advantages can differ across positions in a rollout.The section later specifies how the advantage function is constructed for this setting.

3. ExpRL: Reference-Guided Dense Rewards for RL Priming

ExpRL is an online RL mid-training method that uses hidden reference solutions to construct dense outcome- and process-level rewards for sampled reasoning traces. By reinforcing partial progress and productive reasoning paths rather than imitation or final correctness alone, it aims to broaden coverage before downstream sparse-reward RL.

  • Summary: ExpRL’s central contribution is building stronger coverage for later sparse-reward RL by rewarding partial progress rather than pure correctness.The method uses dense outcome/process rewards instead of treating reference solutions as imitation targets.
  • Design and reward construction: ExpRL uses reference solutions as reward scaffolds, with an LLM judge scoring sampled traces against fixed rubric criteria rather than asking the policy to imitate references.The judge is instructed to verify only what the generated text supports, without filling missing steps or correcting errors.
  • ExpRL-Outcome: ExpRL-Outcome assigns graded rewards to full traces, preserving distinctions among partially correct rollouts that sparse final-answer rewards treat alike.This provides learning signal even when fully correct solutions are rarely sampled.
  • ExpRL-Process: ExpRL-Process scores rollout prefixes to localize credit, rewarding improvements in judged alignment and penalizing regressions relative to prior prefixes.Centered segment-level advantages emphasize relative partial progress and prevent the first segment from dominating updates.
  • Optimization and downstream RL: The dense rewards are optimized with on-policy RL during mid-training, after which downstream RL resumes with unchanged binary final-answer rewards from the ExpRL-primed policy.The intended effect is to shift probability mass toward productive trajectories before sparse-reward training begins.

4. Experiments

Experiments show that ExpRL provides stronger RL priming than imitation-based and sparse-reward alternatives, improving both immediate coverage and downstream sparse-reward RL performance. Mixed-domain and calibration studies indicate that its benefits extend beyond math and depend on correct problem-matched references, although coding remains better served by execution-based rewards.

  • Main results: ExpRL variants outperform SFT, sparse GRPO, and self-distillation on held-out answer-based benchmarks after downstream sparse-reward RL.On AIME-2026, ExpRL-Process reaches 63% after downstream RL.
  • Main results: ExpRL improves held-out pass@1 and pass@k after RL priming alone, increasing coverage that downstream sparse-reward RL can amplify.The reported gains include improved pass@k coverage on IMO-AnswerBench and stronger pass@k curves on HMMT.
  • Training dynamics: During priming, ExpRL maintains higher token-level entropy than sparse-reward GRPO, while ExpRL-Process unlocks solvable prompts fastest.Sparse-reward GRPO collapses entropy fastest and unlocks the fewest prompts; ExpRL-Outcome and ExpRL-Process exhibit distinct dynamics.
  • Reasoning behaviors: ExpRL expands verification, self-correction, and backtracking coverage relative to the base model, preserving or expanding useful search behaviors better than SFT and sparse-reward GRPO.The behavioral changes support reference-guided RL priming as a way to shift sampled trajectories toward productive solution paths.
  • Scope and limitations: On a 4,001-example mixed-domain mixture, ExpRL-Outcome improves the larger Qwen3-8B base policy on every pass@1 evaluation and leads Math-Aggregate and STEM-Aggregate.The mixture spans math, science QA, and coding, using a smaller Qwen3-4B-Instruct judge.
  • Scope and limitations: Correct-reference judging gives the lowest misplacement rate for 4B-and-larger judges across math and science, whereas coding shows similar rates across reference conditions and favors sparse GRPO.These results indicate that ExpRL requires a minimally capable judge and problem-matched references, while coding benefits especially from execution-based rewards.

5. Related Work and Discussion

Prior work distinguishes skill-inducing from coverage-building mid-training, while sparse-reward RL can under-explore hard problems. ExpRL targets coverage-building through on-policy rewards for useful behaviors and partial progress relative to references rather than cloning correct traces.

  • Mid-training before RL: Mid-training primes models for RL through skill-inducing behaviors such as self-correction, backtracking, and verification, or by expanding coverage of productive reasoning paths.Coverage-building is motivated by hard downstream tasks with sparse outcomes.
  • Mid-training before RL: ExpRL targets coverage-building mid-training by rewarding useful reasoning behaviors and partial progress relative to references instead of cloning traces with correct final answers.The method uses on-policy RL, addressing concerns that offline training can cause optimization instabilities and that cloning may narrow exploration.
  • Exploration bottleneck in LLM RL: Sparse-reward RL becomes brittle on hard problems when correct rollouts are rare, causing under-exploration and sometimes degraded pass@k after RL.Prior responses include intrinsic bonuses, entropy regularization, count-based rewards, pass@n-aware objectives, and verification-based signals.

6. Conclusion

The paper frames RL priming as expanding coverage over productive reasoning paths. ExpRL uses dense reference-guided rewards to reward partial progress before sparse-reward RL, but depends on auxiliary reference information that may be unavailable.

  • ExpRL frames RL priming for LLM reasoning as improving coverage over productive reasoning paths.
  • ExpRL uses on-policy RL with dense reference-guided rewards to reward partial progress before sparse outcome-reward RL, yielding stronger initialization and improved downstream RL.
  • ExpRL requires auxiliary information such as reference solutions to identify useful techniques and partial progress, which may be unavailable in some domains.

A. Appendix · A.1. Additional Implementation Details

The appendix details the RL infrastructure and optimization settings for both training stages. Stage I uses modified verl1 with on-policy sampling, while Stage II uses asynchronous Pipeline-RL2 on 8-GPU nodes.

  • A.1. Additional Implementation Details: Stage-I experiments modify verl1 to support on-policy sampling and learning, with up to one-step off-policy updates.ExpRL-Process instead uses fully on-policy updates because they performed better.
  • A.1. Additional Implementation Details: ExpRL-Process is the sole Stage-I exception, using fully on-policy updates after performing better under that configuration.
  • A.1. Additional Implementation Details: The upper clipping threshold is 0.28 for ExpRL-Outcome runs and 0.26 for sparse-reward RL (GRPO) runs.
  • A.1. Additional Implementation Details: Stage-II experiments use asynchronous RL pipelines, specifically Pipeline-RL2.Asynchronous RL speeds experimentation and supports the large number of training steps required in Stage II.
  • A.1. Additional Implementation Details: The answer-based Stage-II experiments require 500 training steps.
  • A.1. Additional Implementation Details: Each Stage-II run is performed on a single node with 8 NVIDIA H100 GPUs.

A.1.1. Slicing steps for process rewards

ExpRL-Process uses the base model’s native ### delimiter to construct semantically meaningful reward prefixes. Although delimiter counts collapse after training with length clipping, unclipped training suggests this is primarily a clipping artifact and remains an open implementation issue.

  • Delimiter-based prefix construction: The ### delimiter matches Qwen3-4B-Instruct’s default format and provides a practical heuristic for splitting reasoning traces into semantically meaningful prefixes.About 98.3% of base-model rollouts contain at least one ### delimiter.
  • Delimiter-based prefix construction: After ExpRL-Process training with length clipping, responses shift sharply toward very few ### delimiters, often containing none or only one.Figure 6 reports the post-training distributional collapse in delimiter counts.
  • Length-clipping interaction: Without length clipping, ExpRL-Process training on Qwen3-4B NoThink preserves a much broader step-count distribution instead of producing the same collapse.This indicates that reduced delimiter usage is driven mainly by the interaction between process rewards and the clipped-length penalty.
  • Length-clipping interaction: The interaction among process rewards, delimiter usage, and length clipping remains an open implementation issue for future work.The NoThink model was not used in the main experiments because Qwen3-4B-Instruct was more capable and better suited to the study.

A.1.2. LLM Judge Prompts

ExpRL uses two judge prompts for its dense reward variants: one evaluates complete reasoning traces, while the other evaluates individual intermediate steps for more localized credit assignment. Both prompts instruct the judge to verify alignment with the reference solution rather than solve or repair the reasoning.

  • ExpRL uses two judge prompts corresponding to its two dense reward variants.The variants are ExpRL-Outcome and ExpRL-Process.
  • The ExpRL-Outcome prompt scores a full generated reasoning trace against a reference solution.
  • The ExpRL-Process prompt scores one intermediate step or segment against the reference solution to provide more localized credit assignment.
  • Both prompts instruct the judge to verify alignment with the reference solution without filling in missing reasoning or correcting it.The judge is instructed to verify rather than solve.

A.2. Ablation: advantage centering for process rewards

The ablation compares three advantage-centering schemes for converting prefix process scores into segment-level rewards. Across held-out benchmarks, all three produce broadly similar Stage-I pass@k curves, indicating limited sensitivity to the exact centering scheme.

  • Advantage normalization: The variants emphasize different aspects of partial progress, including relative improvement over final outcome, local step-to-step gains, and trajectory-centered normalization.All variants operate on the same underlying process-reward signal.
  • Advantage normalization: ExpRL-Process EndNorm, DeltaNorm, and GroupNorm convert prefix scores {s_t} into segment-level advantages.The three methods correspond to Eq. 4–6.
  • Results: Across held-out benchmarks, DeltaNorm, EndNorm, and GroupNorm yield broadly similar Stage-I pass@k curves.This indicates that ExpRL is not highly sensitive to the exact centering scheme for process rewards.
  • Results: GroupNorm is slightly stronger at low k on some benchmarks, but the differences are modest and not uniform across tasks.The advantage does not consistently generalize across benchmarks.

A.3. Stage-I Behavior Analysis · A.4. Stage-I pass@k curves on held-out benchmarks · A.5. The LLM judge provides a useful dense learning signal

Stage-I analysis uses LLM-based annotations to characterize solution strategies and reasoning behaviors, while held-out pass@k curves show that ExpRL improves RL-ready sampling before sparse-reward RL. Judge calibration further indicates that reference-conditioned outcome scores separate correct from incorrect rollouts and process scores broadly track eventual success.

  • A.3. Stage-I Behavior Analysis: Stage-I rollouts are classified by an external Claude Sonnet 4 annotator using a detailed rubric rather than by asking the annotator to solve each problem.The annotator reads each full generated response and returns structured JSON annotations.
  • A.3. Stage-I Behavior Analysis: The rubric distinguishes high-level solution archetypes from process-level reasoning behaviors, with most labels binary and self-interruption and self-correction recorded as counts.Archetypes include coordinatization, casework, recursion, and contradiction; behaviors include verification, backtracking, exploration, and restart.
  • A.3. Stage-I Behavior Analysis: The behavior analysis is fully LLM-judged, aggregates rollout annotations by problem, and compares model-level statistics against the base Qwen3 model.It uses neither regexes, keyword matching, nor hand-written heuristics.
  • A.4. Stage-I pass@k curves on held-out benchmarks: ExpRL improves held-out answer-based pass@k immediately after Stage-I priming, before downstream sparse-reward RL, with the strongest effects on AIME25, AIME26, and HMMT.Gains are especially visible at low to moderate k, while improvements on IMO-AnswerBench are smaller.
  • A.5. The LLM judge provides a useful dense learning signal: Reference-conditioned outcome judging gives substantially higher scores to correct than incorrect rollouts, with clearer separation than judging without the reference.This supports informative dense feedback when fully correct solutions are rare.
  • A.5. The LLM judge provides a useful dense learning signal: Process-level prefix scores are noisier than outcome scores but broadly track the likelihood that a prefix eventually leads to success.Figure 11 compares judge-score trajectories with downstream success estimated from 32 sampled continuations per prefix.
  • A.4. Stage-I pass@k curves on held-out benchmarks: ExpRL-Outcome and ExpRL-Process rewards consistently outperform imitation-based and sparse-reward-only baselines on AIME25, AIME26, and HMMT.The figure describes the clearest gains at low to moderate k, where finding a correct rollout is hardest.
Loading 2606.17024v1…