PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training

Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard

arXiv:2605.12483v4cs.LGcs.AI

TL;DR

Scarce labeled data raises the question of which model should receive each example and reward density. This paper evaluates a teacher-first workflow that uses sparse reward for teacher-side discovery and dense transfer for student compression, improving the deployment student across MATH and AIME 2024 over direct student RL.

  • Problem

    Labeled verifiable data is scarce, leaving unclear which model should receive each example, with which reward density, and in what order.

  • Method

    The workflow applies sparse-reward RL to a stronger teacher, then uses forward-KL warmup, on-policy distillation, and optional post-bridge student RL.

  • Results

    79.3% versus 75.9% on MATH and 25.2% versus 19.8% on AIME 2024 favor RL-teacher transfer over direct GRPO at fixed Qwen3-1.7B student size.

  • Takeaways & Limitations

    The supported operational lesson is to allocate sparse reward to teacher-side discovery, use dense supervision for student compression, and apply student RL after the bridge.

  • Takeaways & Limitations

    Evidence is limited to verifiable math, two student-teacher model families, and relatively small deployment scales, while the bridge requires a shared tokenizer.

Abstract

from arXiv · show

In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative. We identify a reward-density principle that governs this allocation: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model. The principle yields a simple allocation rule: use scarce labeled data upstream on the strongest available teacher, then transfer the reward-shaped behavior downstream as dense supervision. We evaluate this rule through a four-stage workflow -- teacher RL, forward-KL warmup, on-policy distillation, optional post-bridge student RL -- on verifiable math with Qwen3 and Llama models. At fixed Qwen3-1.7B deployment-student size, an RL-improved 8B teacher distilled through the dense bridge outperforms direct GRPO on the same student ($79.3\%$ vs.\ $75.9\%$ on MATH; $25.2\%$ vs.\ $19.8\%$ on AIME~2024, avg@16), while transfer from the same teacher \emph{before} RL underperforms. A component ablation confirms that each stage is load-bearing: replacing the RL-improved teacher with a raw teacher costs $7.8$ MATH points, removing the forward-KL warmup costs $1.7$, and removing on-policy distillation costs $3.3$. The teacher-quality ordering -- raw-teacher transfer $<$ direct GRPO $<$ RL-teacher transfer -- replicates on Llama-3.1-8B-Instruct with a Llama-3.3-70B-Instruct teacher. The operational lesson is to avoid spending scarce labeled data on the least prepared policy: use sparse reward for teacher-side discovery, dense transfer for student compression, and student-side sparse reward only after the bridge.

1 Introduction

Scarce labeled data should first train a stronger teacher with sparse rewards, then be transferred through a dense two-stage bridge to the deployment student. This teacher-first allocation outperforms direct student GRPO and can make subsequent student RL effective.

  • Motivation: Labeled verifiable training data is the bottleneck, making model, reward density, and training order central allocation decisions.Pretraining text and teacher rollouts scale with compute more readily than checked examples with reliable graders.
  • Reward-density principle: The reward-density principle assigns sparse reward to stronger teachers for target discovery and dense supervision to students for behavior compression.Direct GRPO asks the deployment student to discover the target from sparse rollouts, whereas teacher-first transfer supplies a dense autoregressive proxy.
  • Two-stage dense bridge: The two-stage bridge uses forward KL to move the student onto teacher support, then OPD to transfer the teacher under student occupancy.The pipeline is teacher RL, forward-KL warmup, on-policy distillation, and optional post-bridge student RL.
  • Empirical result: Teacher RL plus the two-stage bridge outperforms direct student GRPO for the same Qwen3-1.7B deployment student across MATH and AIME.The headline comparison uses avg@16 accuracy and allocates the same labeled training data to both approaches.
  • Post-bridge student RL: Post-bridge student GRPO lifts the bridge endpoint above direct GRPO and a matched replay control, indicating improved student trainability.The post-bridge stage uses any remaining labeled data after dense transfer.

2 The Workflow

The workflow uses sparse-reward RL to improve a larger teacher, then transfers its behavior through forward-KL warmup and on-policy distillation to a deployment student, with optional held-out-data student RL. Stages 1–2 form the core bridge, while Stage 3 applies only when labeled data is held out.

  • The Workflow: The four-stage workflow runs teacher-side sparse-reward RL, forward-KL distillation on teacher rollouts, on-policy reverse-KL distillation on student rollouts, and optional student RL.Stages 1–2 constitute the core bridge; Stage 3 uses held-out labeled data when available.
  • Stage 1: Teacher-side sparse-reward RL: Teacher-side RL uses labeled data D to produce a reward-shaped teacher whose distribution concentrates on high-reward trajectories.The teacher is intended as a source of dense supervision and differs qualitatively from an SFT-shaped teacher.
  • Stage 2a: Forward-KL warmup on teacher rollouts: Forward-KL warmup trains the student on teacher rollouts by matching the teacher’s next-token distribution under teacher occupancy.This mode-covering, off-policy training is well-conditioned at cold start because it uses teacher-supported states.
  • Stage 2b: On-policy distillation under student rollouts: On-policy distillation minimizes reverse-KL on student rollouts, querying the frozen teacher at the student’s sampled prefixes.It corrects the student on its own state distribution.
  • Stage 3: Post-bridge student-side sparse-reward RL: Held-out labeled data enables optional post-bridge GRPO on the bridged student, which adds value over skipping it or reusing bridge data.Stage 3 applies only when labeled data is split between teacher and student.

3 Theory: A Reward-Density Principle

The theory frames sparse verifier reward as a teacher-side discovery signal and on-policy distillation as dense student-side compression. This bridge is effective when the teacher is reward-shaped and sufficiently close to the student for stable token-level transfer.

  • Reward-shaped target: A sequence-level verifier reward defines a KL-regularized reward-shaped target, which direct student RL must recover from the student’s sparse rollouts.The workflow instead applies sparse reward to a larger model and uses the resulting teacher as a proxy.
  • OPD as a local implicit-reward update: OPD converts the teacher signal into a dense, on-policy, teacher-induced implicit reward and performs a local trust-region update around the current student.Its update maximizes expected dense implicit reward while constraining divergence from the current student policy.
  • Proxy conditions: The teacher must be near-optimal on verifier reward; otherwise, OPD pushes the student toward a larger but unshaped, non-reward-aware distribution.Stage 1 sparse-reward training is the mechanism enforcing this optimality condition.
  • Proxy conditions: Stable OPD also requires small teacher–student KL; poor coverage overlap causes high-variance token rewards and gradients dominated by rare outliers.The theory highlights the cold 1.7B student and post-RL 8B teacher as an example of potentially problematic overlap.
  • Falsifiable predictions: The theory predicts that removing teacher RL makes transfer no better than direct student GRPO, while removing warmup or on-policy feedback weakens the bridge.Warmup addresses cold-start ill-conditioning, and on-policy distillation exposes the student to feedback on its own states.

4 Experiments

Experiments across Qwen3 and Llama models support allocating sparse reward to teacher-side discovery, followed by dense bridging and optional student RL. Component ablations show that teacher RL, forward-KL warmup, OPD, and post-bridge RL each contribute under fixed labeled-data budgets.

  • Experimental design: The experiments use Qwen3-1.7B as deployment student, raw or RL-trained Qwen3-8B/14B teachers, and replicate teacher-quality ordering with Llama models.The workflow tests allocation, bridging, and post-bridge student RL through baseline and ablation comparisons.
  • Teacher-side allocation: 71.5% and 72.8% MATH from raw 8B and 14B teachers are 7.8 and 5.8 points below full-workflow endpoints, confirming teacher-side RL is load-bearing.The dense implicit reward becomes useful when its source distribution has been shaped by sparse reward; scale alone does not suffice.
  • Dense bridge: 1.7 points and 1.5 points are lost by removing forward-KL warmup at 8B and 14B teacher sizes, respectively.Without warmup, coverage mismatch makes the implicit reward high-variance and OPD poorly conditioned.
  • Dense bridge: 76.0% and 76.5% result when OPD is removed at the 8B and 14B teacher sizes, showing off-policy teacher-sample SFT alone is insufficient.OPD supplies feedback on student-only states, and both one-stage variants underperform the two-stage bridge on MATH and AIME 2024.
  • Post-bridge student RL: 78.5% and 78.7% MATH follow post-bridge student RL from 75.4% and 76.3% at 8B and 14B teacher sizes, while replay improves by no more than 0.3 points.Cold direct GRPO reaches only 75.9%, so the gains reflect new held-out labeled examples after the bridge rather than extra updating alone.
  • Cross-family replication: 55.4% vs. 59.8% MATH shows that a 9× larger raw teacher remains worse than direct GRPO, while raw-teacher transfer < direct GRPO < RL-teacher transfer replicates on Llama.The replication uses Llama-3.1-8B-Instruct as student and Llama-3.3-70B-Instruct as teacher.

5 Discussion

The discussion reframes post-training as a reward-density allocation problem: use labeled sparse-reward data preferentially on larger teachers, then retain student-side RL after the bridge when it improves results. The evidence remains limited to verifiable math and relatively small deployment scales, with larger-scale persistence unresolved.

  • What changes operationally: The reward-density principle reframes post-training methods as an allocation problem over models, signal density, and execution order.Direct sparse-reward RL on the deployment model is described as inefficient placement.
  • Implication for model-family training: Labeled post-training data should be allocated preferentially to the larger model first when training a model family.The recipe assumes parallel larger-teacher and smaller-student targets, preferably pretrained on the same distribution with a shared tokenizer.
  • Why student-side reward still matters: 3.1-point lift on MATH comes from post-bridge student RL, which outperforms additional bridge-data updates.The result supports teacher-first training with post-bridge student RL rather than either always or never training the student.
  • Limitations: The evidence covers verifiable math with 1.7B and 8B students and teachers up to 14B and 70B, leaving larger-scale persistence open.The discussion specifically raises a 70B student with a 400B+ teacher as an unresolved setting.

6 Related Work

Prior work applies sparse rewards directly to deployment models or transfers teacher behavior through dense supervision, including on-policy distillation from student rollouts. This paper instead studies how to allocate fixed labeled data between teacher-side and student-side training and which bridge connects them.

  • Sparse-reward post-training: Sparse-reward methods apply PPO, GRPO, or SFT-warmup-then-PPO directly to deployment models, while verifier-filtered SFT uses reward only to filter data.Recent approaches also increase reward density through self-distillation or reference-guided trajectories.
  • Distillation and OPD: Knowledge distillation transfers teacher behavior through dense supervision, with teacher-sample SFT as its off-policy form.OPD instead corrects students on their own rollouts using dense on-policy teacher-logprob reward.
  • Reasoning teachers and data allocation: RL-improved models can teach smaller ones via SFT, while multi-teacher OPD integrates domain specialists through on-policy token-level rewards.These examples establish reasoning-teacher transfer and multi-teacher extensions.
  • Reasoning teachers and data allocation: This paper focuses on allocating a fixed pool of labeled training data between teacher-side and student-side training, and on the bridge connecting them.It studies allocation rather than merely whether an RL-improved model can teach.

7 Conclusion · A OPD as a Local Implicit-Reward Update

The four-stage workflow improves the fixed-budget Qwen3-1.7B student on MATH and AIME 2024, while OPD admits a local dense implicit-reward interpretation. This equivalence is local, requires adequate teacher–student coverage overlap for informative updates, and does not imply global optimization of the sparse task reward.

  • 7 Conclusion: 79.3% vs. 75.9% on MATH and 25.2% vs. 19.8% on AIME 2024: the four-stage workflow improves Qwen3-1.7B at fixed labeled-data budget.The workflow comprises teacher RL, forward-KL warmup, on-policy distillation, and optional post-bridge student RL.
  • 7 Conclusion: Each on-policy distillation step is a local trust-region update under a dense teacher-induced implicit reward, informative when the teacher is reward-shaped.The conclusion connects the workflow’s empirical gains to its reward-density principle.
  • A OPD as a Local Implicit-Reward Update: Reverse-KL OPD minimizes KL(πθ∥πT) by comparing student-sampled sequences under the student and teacher likelihoods.This objective is stated for a fixed prompt x.
  • A OPD as a Local Implicit-Reward Update: At the current student πk, OPD defines a per-step implicit reward β[log πT(y | x) − log πk(y | x)].The reward is anchored at the current student and uses the teacher–student likelihood ratio.
  • A OPD as a Local Implicit-Reward Update: The policy-gradient update maximizing this implicit reward equals the negative reverse-KL gradient at θk.The appendix notes that the differentiated log-probability’s +1 term vanishes in expectation against the score function.
  • A OPD as a Local Implicit-Reward Update: The policy-gradient interpretation is local at πk and does not claim that OPD globally optimizes the original sparse task reward.The local identity is a fact about linearization at the current student, independent of the global reference used to define the reward-shaped fixed point.
  • A OPD as a Local Implicit-Reward Update: Forward-KL warmup addresses poor teacher–student coverage overlap, where outlier implicit-reward terms destabilize OPD updates.Teacher-favored tokens are rarely sampled under low overlap, while unlikely teacher tokens can receive large reward magnitudes.

B Half-Split Experiments: SFT-Teacher and Bridge-Protocol Controls

The half-split controls separate teacher shaping from bridge-data reuse and compare SFT-trained teachers with RL-trained teachers. They show that teacher quality and transfer protocol both affect student-side GRPO initialization.

  • Half-split construction: The DAPO-Math-17K set is randomly split into 1H and 2H, with Stages 1–2 on 1H and Stage 3 using either held-out 2H or replayed 1H.The held-out split is the full workflow, while reuse of 1H is the replay control.
  • SFT-teacher control: SFT-trained Qwen3-8B and Qwen3-14B teachers learn from gpt-oss-120B responses, providing a non-RL counterfactual to verifier-based sparse-reward training.The protocols share the starting checkpoints and prompts but differ in supervised next-token training versus sparse-reward RL.
  • SFT-teacher control: The teacher-quality ordering is raw < SFT < RL, with SFT-teacher bridges reaching lower MATH and AIME 2025 endpoints than RL-teacher bridges.This pattern holds for Stage 3 with a Qwen3-1.7B student evaluated using avg@16.
  • Bridge-protocol controls: Under half-split controls, the same RL-trained Qwen3 teacher and Stage 3 GRPO data isolate transfer protocol, with the two-stage bridge strongest before student-side GRPO.Only the transfer protocol varies across rows.

C Where Should the Held-Out Half Go? · D Extended Related Work

The held-out labeled-data half performs slightly better when allocated upstream to teacher training, though student-side RL remains a competitive lower-cost alternative. Extended related work situates this allocation question among sparse-reward optimization, distillation and on-policy distillation, and reasoning-teacher research.

  • C Where Should the Held-Out Half Go?: The allocation comparison asks whether the held-out second half, 2H, should be assigned to Stage 1 teacher training or Stage 3 student training.This is the residual allocation question after Section 4.2 under a fixed labeled-data pool.
  • C Where Should the Held-Out Half Go?: 79.3% MATH results from allocating both labeled halves upstream to train the teacher and bridge, versus 78.5% when the second half trains the bridged student.Both placements use the same total labeled data; only the placement of 2H changes.
  • C Where Should the Held-Out Half Go?: 0.8 MATH points favor teacher-side allocation, while AIME differences remain within standard error.When teacher-side compute is binding, student-side RL remains a competitive lower-cost alternative.
  • C Where Should the Held-Out Half Go?: Table 6 reports transfer-only endpoints for Qwen3-1.7B students without Stage 3, including raw, SFT, and RL’d teachers across 1.7B, 8B, and 14B.The 1.7B RL’d-teacher rows isolate dense-reward effects from teacher scale.
  • D Extended Related Work: The appendix supplies per-paper detail omitted from the shorter related-work section.This extended section therefore complements the main discussion with more granular literature coverage.
  • D Extended Related Work: Sparse-reward post-training updates policies directly through PPO, GRPO, or SFT-warmup-then-PPO, while verifier-filtered SFT uses reward to construct training data.Systems such as verl/HybridFlow support practical RLHF dataflows through flexible algorithms and distributed execution.
  • D Extended Related Work: Knowledge distillation transfers stronger-model behavior to smaller models, with reasoning-distillation work emphasizing intermediate traces and on-policy distillation extending dense behavior compression.The cited work also studies KL schedules, informative student-state tokens, self-distillation, offline OPD, and unified OPD recipes.
  • D Extended Related Work: DeepSeek-R1 demonstrated that large-scale RL can elicit reasoning behavior that smaller models inherit through supervised fine-tuning on generated traces.ORBIT instead discovers reasoning-effort policies under context constraints and fuses them into one controllable model through OPD.

E Method Classification

This section classifies representative methods by where sparse reward enters and which signal is used for transfer.

  • Representative methods are classified by sparse-reward entry location and transfer signal.

F Implementation Details

The experiments fix deployment-student size within each model family, use matched training protocols across GRPO contrasts, and implement transfer as forward-KL warmup followed by on-policy distillation. Evaluation reports avg@16 accuracies with standard errors over evaluation problems.

  • Model and transfer setup: Qwen3 comparisons fix the student at Qwen3-1.7B, while Llama comparisons use Llama-3.1-8B-Instruct with a Llama-3.3-70B-Instruct teacher.OPD is restricted to within-family transfer because token-level KL requires a shared tokenizer and vocabulary.
  • Data splits: The Qwen allocation experiment splits DAPO-Math-17K into equal halves, assigning 1H to teacher RL and bridging while holding 2H out for full-workflow Stage 3 GRPO.The full-workflow and replay rows begin from the same bridge checkpoint and use the same Stage ...
  • Matched training protocol: Direct GRPO, Stage 3 GRPO, and replay GRPO share verifier rewards and matched optimization, rollout, length, learning-rate, KL, and update settings within each contrast.Full-workflow and replay rows are also matched in checkpoint initialization, data count, rollout count, update count, and sequence-length limit.
  • Bridge protocol: The bridge applies forward-KL warmup to cached teacher rollouts, then computes OPD teacher signals on prefixes sampled from the current student.Cached logits may support audit and replay, but the teacher checkpoint remains ...
  • Evaluation and error bars: All reported accuracies are avg@16: each problem averages 16 independent completions, and table values average correctness across evaluation problems.Reported ± values are standard errors over evaluation problems, not standard deviations across independently retrained checkpoints.
Loading 2605.12483v4…