PaperlayerView exact revision on arXiv

Source-linked AI summary

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Yang Li, Semih Yavuz, Shafiq Joty

arXiv:2609.05295v1cs.AI

TL;DR

OPD’s dense supervision is limited by teacher quality: external teachers suffer distribution mismatch, while privileged self-distillation depends on in-context learning. RISE extrapolates the model’s own RLVR trajectory in logit or parameter space to create a refreshed token-level teacher, and it consistently outperforms RLVR and privileged-conditioning baselines across four task families. Its main scope limitation is reliance on sufficiently low-dimensional trajectories and non-hackable reward signals.

  • Problem

    OPD provides dense token-level supervision, but external teachers suffer distribution mismatch and privileged self-distillation is limited by in-context learning capacity.

  • Method

    RISE extrapolates the displacement between an RLVR-updated policy and an anchor in logit or weight space, then distills the resulting future teacher into the student.

  • Results

    RISE consistently outperforms RLVR and privileged-conditioning baselines across math, STEM, code, and agentic tasks, with the largest margins on challenging competition benchmarks.

  • Takeaways & Limitations

    Refreshing the teacher from each RLVR update turns OPD into a recursive improvement loop rather than one-shot compression.

  • Takeaways & Limitations

    RISE relies on sufficiently low-dimensional trajectories for meaningful linear extrapolation and can amplify spurious directions when RLVR rewards are hackable.

Abstract

from arXiv · show

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose \textbf{RISE} (\textbf{R}ecursive \textbf{I}mprovement via \textbf{S}elf-\textbf{E}xtrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.

1 Introduction

RISE addresses OPD’s teacher-quality bottleneck by extrapolating the model’s own RLVR trajectory into a dense token-level teacher. RLVR grounds the direction in verified outcomes, while OPD refines token decisions through a recursively refreshed teacher.

  • Motivation: OPD supplies dense per-token supervision, but external teachers face distribution mismatch and privileged self-distillation is limited by in-context learning.These limitations motivate constructing a teacher from the student’s own trajectory.
  • RISE: RISE extrapolates the displacement between a current checkpoint and an earlier anchor to synthesize a future teacher in parameter or logit space.The displacement captures recent improvement, and extrapolation projects beyond the current state along the training trajectory.
  • RISE: When β = 1, the teacher equals the current checkpoint; when β > 1, extrapolation amplifies the most recent update.The method uses β > 1 to continue along the training direction.
  • Training loop: RLVR grounds extrapolation in verified improvement, while OPD converts the extrapolated teacher into fine-grained token-level refinement.The two signals are complementary: RLVR supplies a meaningful direction and OPD supplies dense supervision.
  • Contributions: Across math, STEM, code, and agentic tasks, RISE outperforms RLVR and OPSD baselines with improved sample efficiency and 1.3–1.6× wall-time overhead.The teacher is refreshed from each latest update, making distillation a recursive improvement loop rather than one-shot compression.

2 Background and Related Work

RLVR provides sparse outcome-level learning signals, whereas OPD provides dense token-level supervision but depends critically on teacher quality. RISE builds on low-dimensional training dynamics to replace static or mismatched teachers with a non-stationary extrapolated teacher.

  • RLVR: RLVR assigns outcome-derived advantages across response tokens, creating a credit-assignment bottleneck because helpful and harmful steps receive indistinguishable signals.Policy-gradient methods optimize objectively checkable rewards but do not distinguish token-level contributions within a response.
  • OPD: OPD minimizes per-token divergence to a teacher distribution, providing relative desirability over vocabulary alternatives rather than a single outcome signal.Full-vocabulary divergence is expensive, motivating top-K or sample-based approximations.
  • Teacher limitations: External teachers become unreliable on increasingly diverse student prefixes, while privileged self-distillation can fail to use or transfer privileged context effectively.The two approaches introduce different forms of distribution mismatch or depend on limited in-context learning.
  • Linear trajectories: Post-training updates are concentrated in low-dimensional, approximately linear directions, supporting extrapolation beyond the current checkpoint along the training trajectory.The paper reports that three directions capture approximately 87% of variance in its runs.
  • Joint training: RISE’s non-stationary teacher advances with RLVR improvement, avoiding the static teacher ceiling associated with external reward extrapolation.This converts OPD from one-shot compression into a recursive improvement loop.

3 Method

RISE alternates RLVR updates with OPD against a teacher extrapolated from the current policy and an anchor. It supports logit-space and weight-space constructions, uses trust-region-oriented refinement, and adapts extrapolation as training progresses.

  • Teacher construction: The self-extrapolated teacher extends the displacement between the post-RLVR policy and an anchor through a representation map φ.The optimal teacher is unknown, so RISE approximates it using the RLVR-grounded training trajectory.
  • Extrapolation control: The extrapolation scale satisfies β = 1 for the current post-RLVR policy and β > 1 for continued movement along the improvement direction.The method uses a decreasing β schedule because the safe range narrows as training approaches the optimum.
  • Instantiations: Weight-space extrapolation applies the displacement to parameters, whereas logit-space extrapolation applies it to per-token log-probabilities.Weight-space yields a coherent materialized model; logit-space avoids parameter manipulation and fixes the teacher before OPD.
  • Implementation: RISE uses top-K approximations for practical divergence computation and holds the extrapolated teacher fixed during OPD gradient steps.Logit-space constructs the teacher from cached distributions, while weight-space evaluates a materialized extrapolated model.
  • Training loop: Each iteration first samples rollouts and applies RLVR to obtain a post-RLVR policy, then constructs an extrapolated teacher and distills it into the student.The same rollouts are reused for both phases, so OPD adds no additional sampling cost.
  • Anchor dynamics: An EMA anchor can smooth and enlarge the displacement, with the previous-checkpoint anchor recovered when η = 1.Anchor choice is treated as an ablated design option.
  • Role of OPD: OPD acts as a trust-region projection toward the extrapolated teacher while keeping the update anchored near the post-RLVR policy.Directly adopting the extrapolated policy could amplify noise or cause degenerate behavior at large β.

4 Experiments

Across mathematical, multi-domain, agentic, and code tasks, RISE consistently improves performance, sample efficiency, or convergence over RLVR and other baselines. Ablations show that RLVR grounding and OPD refinement are complementary, while extrapolation requires conservative schedules and adds moderate compute overhead.

  • RISE consistently outperforms baselines across mathematical reasoning, multi-domain STEM, and agentic tasks.
  • +16.7 AIME’24 accuracy: RISE (logit) improves OLMo3-7B from 30.2 to 46.9, while Math Avg rises from 47.6 to 56.4 (+8.8).
  • RISE preserves or improves out-of-distribution performance, with Qwen3-8B OOD Avg rising from 70.6 to 72.0 and OLMo3-7B from 51.2 to 55.5.
  • RISE learns faster than GRPO, reaching higher accuracy earlier, with the largest gap during early training before RL advantage estimates stabilize.
  • In mixed math and STEM training, RISE (weight) reaches Math Avg 44.8 versus 40.2 and STEM Avg 47.5 versus 45.5 for GRPO.
  • In agentic tasks, RISE (weight) exceeds GRPO by +9.4 on ALFWorld and +10.9 Acc on WebShop.
  • Removing RLVR causes collapse, whereas removing OPD leaves training stable but eliminates consistent gains; both phases are therefore necessary for reliable improvement.
  • The safe extrapolation range narrows during training: β ≤1.2 is safe in the tested operating range, while β0 = 2.0 diverges or becomes catastrophic near optimality.

5 Conclusion

RISE combines RLVR-grounded extrapolation with OPD to turn a model’s training trajectory into a recursively refreshed teacher. Its theory characterizes when extrapolation contracts suboptimality, while experiments and limitations delimit the conditions under which this mechanism is reliable.

  • Conclusion: RISE extrapolates the model’s RLVR trajectory in logit or weight space, then distills the resulting future policy into the current student.The approach uses outcome rewards to ground the direction and OPD to apply dense token-level refinement.
  • Theory: The decomposition separates distillation error, which vanishes as the student approaches the teacher, from a persistent teacher gap to the optimum.Because the bound has a T-dependent prefactor, the result is directional rather than a tight numerical certificate.
  • Theory: Under a linear trajectory, the extrapolated teacher is closer to the optimum when 1 < β < 2α∗/α_n −1, so the safe range narrows as training converges.Values below α∗/α_n remain between the student and optimum, while larger values can overshoot yet remain closer within the stated range.
  • Theory: After OPD, the suboptimality bound contracts by γ(β) < 1 in the idealized zero-distillation-error limit.In practice, OPD acts as a trust-region projection toward the extrapolated teacher rather than converging fully to it.
  • Limitations: RISE relies on sufficiently low-dimensional, approximately linear trajectories; large extrapolation factors can leave this regime and degrade teacher quality.The authors describe the theory as a mechanism characterization rather than an exact convergence guarantee.

A.3 How Low-Dimensional Are Our Trajectories?

Measured RLVR trajectories are not rank-one, but they occupy a low-dimensional subspace: three components explain roughly 87% of variance across model scales. This supports modest extrapolation while motivating caution about approximation and top-K implementation effects.

  • Trajectory dimensionality: The leading trajectory component explains 68.2% of variance at both Qwen3-1.7B and Qwen3-8B, while the top three explain 86.6% and 88.5%.The near-identical leading-component share despite a 5× parameter-count difference suggests a property of training dynamics rather than model size.
  • Trajectory dimensionality: Three components capture approximately 87% of variance, leaving a substantial but bounded orthogonal residual rather than an exactly linear path.Within the rank-3 subspace, the leading direction explains about 78% of subspace variance.
  • Extrapolation range: For β ≤ 1.2, the residual orthogonal variance is argued to control linear-approximation error, and empirical gains persist to β ≈ 1.3.The safe range still contracts as the policy converges, consistent with the theoretical prediction.
  • Top-K approximation: The logit-space implementation selects the current top-K support before extrapolation, so tokens promoted from the tail can be omitted from the extrapolated support.This differs from full-vocabulary extrapolation followed by selecting TopK(π_future).
  • Top-K approximation: With K = 100, the selected tokens carry more than 99% of typical distributional mass, making the induced bias negligible in practice for moderate β.The text notes that this does not formally bound support-set differences, whereas weight-space extrapolation applies top-K to the exact forward distribution.

B.1 Training Setup

The experiments use matched training configurations across RLVR and distillation methods, evaluate broad reasoning and coding benchmarks, and report complementary measures of average quality and sampling coverage.

  • Training configuration: All methods share configuration-specific hyperparameters, so performance differences are attributed to teacher construction rather than training dynamics.Training uses VeRL on one node with 8 GPUs for one epoch, AdamW, constant learning rate, and no warmup.
  • Benchmarks: The evaluation suite covers mathematical reasoning, out-of-distribution STEM and instruction benchmarks, multi-domain tasks, code generation, and agentic tasks.The mathematical suite includes MATH-500, AIME, AMC, Minerva, OlympiadBench, GPQA-Diamond, IFEval, MMLU-Pro, SuperGPQA, and TheoremQA.
  • Metrics: avg@N measures mean accuracy across sampled completions, while pass@N estimates the probability that at least one of N independent draws is correct.pass@N is estimated using 1,000 bootstrap subsets per problem before averaging across prompts.
  • Baselines: OPSD baselines use a privileged teacher and differ in how its token-level signal enters the update.GRPO+SDPO adds a KL loss, whereas SDAR and RLSD reweight token-level advantages.
  • Baselines: The privileged teacher is available only when a rollout group contains a correct solution, whereas RISE derives its teacher from the training trajectory and applies it uniformly.The baselines’ teacher is active on approximately 85% of Qwen3-8B prompts and 70% of Qwen3-1.7B prompts on average.

C.1 Qwen3-1.7B-Base Results

On Qwen3-1.7B-Base evaluated on DAPOMath, RISE achieves the strongest reported in-domain or selected benchmark results, though its margin over GRPO is comparatively narrow.

  • Overall results: 29.2 Math Avg is the best in-domain average achieved by RISE (logit) on Qwen3-1.7B-Base.RISE (logit) also shows competitive out-of-distribution performance.
  • Benchmark results: 62.4 on MATH-500 and 20.7 on Minerva are the leading scores achieved by RISE (weight).These results show that the weight-space variant leads on these two benchmarks.
  • Comparison with GRPO: +1.2 Math Avg is RISE’s margin over GRPO, the narrowest margin among the reported configurations.The passage attributes this smaller margin to the capability ceiling of a base model that has not been instruction-tuned.

C.2 Per-Metric Convergence Curves

Across representative benchmarks and metrics, both RISE variants reach higher accuracy in fewer training steps than GRPO, with code-generation curves showing faster early convergence.

  • Per-metric convergence: Both RISE variants consistently reach higher accuracy in fewer training steps across the representative benchmarks and model configurations.Figure C.1 presents per-metric curves with four representative benchmarks in each row.
  • Code generation: Both RISE variants converge faster than GRPO early in training on HumanEval+, MBPP+, and LiveCodeBench under avg@4 and pass@4.The comparison uses Qwen3-8B-Base trained on Skywork-OR1-Code.
  • Average versus coverage: RISE improves both avg@16 and pass@16 on the AIME benchmarks rather than improving mean quality while leaving coverage unchanged.At 1.7B on AIME’24, pass@16 gains exceed avg@16 gains for RISE (logit): +9.6 versus +6.3.

C.5 Grounding and OPD Ablation

The ablations show that RLVR grounds RISE against collapse and that OPD is necessary to realize consistent gains, while anchor smoothing can materially affect performance in some settings.

  • Grounding ablation: Without RLVR, training reward falls from 0.56 to 0.00 by step 60, while policy entropy spikes to 0.78 before dropping near zero.The resulting behavior is consistent with repetitive non-terminating generation; GRPO and RISE maintain stable entropy.
  • OPD ablation: Removing OPD leaves Math Avg essentially unchanged relative to GRPO across the reported model scales, forfeiting RISE’s gains.The matched comparisons are 60.0→60.3 at 8B, 45.4→45.6 at 1.7B, and 28.0→27.7 at 1.7B-Base.
  • Anchor ablation: On OLMo3-7B-Instruct-SFT, removing EMA smoothing causes drops of −6.3 for logit-space and −2.7 for weight-space extrapolation.The reported explanation is that large, stable per-step GRPO displacements do not benefit from smoothing.

C.9 Compute-Matched Comparison

The appendix compares RISE with compute-matched GRPO-2× and examines rollout reuse and anchor choices across math benchmarks. It also reports sensitivity to the β0 and decay schedule.

  • Compute-Matched Comparison: GRPO-2× uses a second gradient pass on the same rollouts, matching RISE’s total gradient budget at equal sampling cost.This comparison isolates RISE’s RLVR + OPD phases from a doubled inner-loop update.
  • Schedule Sensitivity: The β0 and decay-schedule ablation varies β0 and schedule settings on Qwen3-8B and β0 on Qwen3-1.7B-Base.
  • Rollout Strategy: Rollout reuse reuses RLVR rollouts for OPD, while resampling generates fresh rollouts from πθ′.

C.10 Reproducibility Across Random Seeds

Across three random seeds, RISE’s DAPOMath gains over GRPO exceed seed-to-seed variance at both Qwen3 scales. The appendix also reports anchor and compute-matching comparisons relevant to interpreting these results.

  • Multi-Seed Results: +4.5 Math Avg at 1.7B: RISE (logit) reaches 49.6 versus GRPO’s 45.1, roughly 10× the per-method standard deviation.The standard deviation is ≤0.6 across the reported methods.
  • Multi-Seed Results: +2.2 Math Avg at 8B: RISE (weight) reaches 62.4 versus GRPO’s 60.1, and is +2.8 over SDAR’s 59.6.
  • Additional Comparisons: The anchor ablation finds that EMA helps on Qwen3-1.7B but degrades on OLMo3-7B-Instruct-SFT.
  • Additional Comparisons: The compute-matched comparison pairs RISE with GRPO-2×, which performs a second inner-loop pass on the same rollouts.
  • Multi-Seed Results: Table C.8 reports mean ± standard deviation across three random seeds for Qwen3-1.7B and Qwen3-8B on DAPOMath.
Loading 2609.05295v1…