Source-linked AI summary
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, Vladimir Braverman
TL;DR
OPD can collapse into repetition-saturated, truncation-dominated rollouts that bias gradients and destabilize validation performance. The paper proposes Stable-OPD, combining a reference-based divergence constraint with rollout mixture distillation, and reports a 7.2% average accuracy improvement over standard OPD baselines across mathematical reasoning benchmarks.
Problem
OPD can undergo abrupt length inflation and repetition saturation, causing truncation-dominated training data, biased gradients, and severe instability.
Method
Stable-OPD combines a reference-based divergence constraint with rollout mixture distillation to stabilize on-policy training.
Results
7.2% average accuracy improvement over standard OPD baselines is reported across multiple mathematical reasoning benchmarks.
Takeaways & Limitations
The combined constraints prevent truncation-repetition collapse and stabilize OPD training across the evaluated settings.
Abstract
from arXiv · showhide
On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.
1. Introduction
OPD can develop abrupt repetition-driven length inflation: truncated rollouts dominate the data, bias gradients, and destabilize training. Stable-OPD addresses this pathology with divergence control and rollout mixture distillation, improving accuracy across datasets and models.
- Failure mode: OPD rollouts can abruptly become much longer, causing truncated trajectories to dominate training as repetition saturation emerges.The pathology is described as distinct from generic length bias in GRPO-style reinforcement learning.
- Mechanism: Repetitive tokens receive disproportionately large reverse-KL advantages, creating a feedback loop that favors further repetition and length expansion.As repetition frequency rises, these token-level advantages increasingly dominate gradient updates.
- Consequences: Repetition saturation and rollout truncation coincide with biased gradients, severe training instability, and sudden validation-accuracy drops.The collapse occurs even though the teacher and loss formulation remain fixed, implicating OPD’s on-policy dynamics.
- Stabilization: Stable-OPD combines a reference-based divergence constraint with rollout mixture distillation to limit policy drift and preserve complete, non-truncated trajectories.The mixture maintains a stable fraction of reference trajectories while the divergence constraint curbs excessive rollout expansion.
- Evaluation: Across six datasets and three LLMs, Stable-OPD consistently improves accuracy and reduces repetition saturation.The evaluation summary reports consistent gains across the tested datasets and model configurations.
2. Preliminary
The preliminary section contrasts sequence-level reinforcement learning and fixed-sequence distillation with OPD’s student-generated training states. It establishes the objectives and limitations motivating token-level, on-policy supervision.
- Group Relative Policy Optimization: GRPO samples groups of responses, assigns sequence-level rewards, and applies group-normalized advantages to every token in each response.Its clipped objective inherits the PPO trust-region formulation.
- Group Relative Policy Optimization: GRPO provides limited token-level error guidance and can produce no effective update when all sampled responses are correct or incorrect.The latter case yields zero advantages despite the cost of group sampling.
- Knowledge distillation: Knowledge distillation trains a student by matching a stronger teacher’s output distribution on fixed sequences such as teacher responses or demonstrations.The standard setup minimizes a divergence between teacher and student next-token distributions.
- Knowledge distillation: Fixed-sequence distillation creates a training-inference mismatch because students condition on self-generated prefixes at inference time.Those prefixes may differ from the fixed distillation sequences.
- On-policy distillation: OPD addresses this mismatch by training on responses sampled from the student policy, aligning training states with the student’s test-time states.Its loss is formulated over student-generated responses with teacher guidance.
3. Length Inflation in OPD
OPD training exhibits a robust truncation-repetition inflation failure mode: rollouts abruptly become long and repetitive, truncation dominates, and validation accuracy collapses. Token-level reverse-KL advantages and student-induced dynamics help explain how repetitive continuations gain disproportionate influence.
- Metrics: OPD monitors truncation and repetition on training and validation rollouts to analyze training dynamics.Truncation marks responses that exhaust the fixed generation budget; repetition uses compression-based detection of highly compressible tails.
- Stable early training stage: Across three student-teacher groups, training begins stably with low repetition and gradually improving validation accuracy.Early rollout truncation is around 0.5 for the 1.5B student and 0.23 for the 7B students, while repetition remains near zero.
- Phase transition: Within about 30 OPD steps, rollout truncation rises toward one while repetition spikes from near zero to about 0.3-0.6.The transition indicates that most generations hit the maximum length budget and develop long, highly compressible repetitive suffixes.
- Phase transition: Validation truncation and repetition jump at nearly the same step as rollout inflation, coinciding with a sudden drop in MATH500 accuracy across all three groups.The shared transition across model pairs and dataset evaluation suggests a robust OPD failure mode rather than an artifact of one configuration.
- Rollout-level evidence: Around inflation onset, response length, student and teacher log-probabilities, and reverse-KL advantage shift together, with the teacher log-probability increasing more.The rollout-level pattern is consistent across all three student-teacher groups.
- Token-level and mechanistic evidence: Repetitive tokens receive larger reverse-KL advantages, and after inflation their increased frequency lets them disproportionately influence OPD updates.After collapse, repetitive tokens account for roughly 30% of tokens while their average advantage is about 4–9× that of non-repetitive tokens.
4. Mitigating Repetition Saturation
The paper proposes mixture distillation and reference-based KL regularization to prevent OPD from being dominated by long, repetitive, truncated trajectories. Together, these methods rebalance supervision and constrain policy drift.
- Mixture Distillation: Mixture distillation combines on-policy student rollouts with high-quality golden solutions to maintain complete, non-truncated trajectories during training.The mixed objective reduces the influence of degenerate on-policy rollouts while preserving the original teacher-derived OPD signal.
- Mixture Distillation: Golden data anchors OPD to complete, non-repetitive reasoning by adding off-policy supervision for the same prompts as student trajectories.This creates a mixture of the student-induced distribution and a fixed golden-data distribution.
- Motivation: Repetitive tokens receive larger reverse-KL advantages throughout training, so their sharply increasing frequency can make them dominate OPD updates after collapse.This pattern motivates explicitly combining distributional anchoring with policy-drift control.
- KL-Regularized Mixture Distillation: The reference-based divergence constraint penalizes deviations from a reference policy at visited prefix states, limiting uncontrolled policy drift and rollout expansion.The regularization strength is controlled by βKL.
5. Experiment
Experiments evaluate Stable-OPD on mathematical reasoning benchmarks, comparing it with supervised, reinforcement-learning, and standard OPD baselines. Stable-OPD improves accuracy and keeps training dynamics stable across tested settings.
- Performance: Standard OPD reaches 43.8% average accuracy on the 7B backbone, below SFT at 44.1% and GRPO at 45.5%.The results indicate that standard OPD does not match these baselines despite on-policy samples and dense token-level supervision.
- Performance: Stable-OPD achieves 47.6% average accuracy on Qwen2.5-Math-7B, outperforming the evaluated zero-style RLVR methods.The comparison covers six mathematical reasoning benchmarks.
- Training Dynamics: Across both student-teacher groups, standard OPD develops sharp and persistent truncation and repetition spikes, whereas Stable-OPD training remains stable.The dynamics are tracked on training rollouts and MATH500 evaluation over training steps.
- Ablation: Combining KL regularization with mixture distillation improves the 1.5B ablation from 29.7% to 35.7%, exceeding KL regularization alone.KL regularization alone improves OPD from 28.0% to 29.7%, while the combined variant is strongest.
6. Related Work
The paper distinguishes OPD’s repetition-driven length inflation from sequence-level length bias in GRPO-style reinforcement learning. It situates Stable-OPD within knowledge distillation and on-policy distillation research.
- Length Bias in LLM Reasoning: Unlike GRPO-style length bias, OPD’s failure arises from repetitive tokens receiving larger token-level advantages that become dominant through on-policy sampling.The paper characterizes this mechanism as specific to OPD dynamics.
- Experimental Comparisons: The reported experiments use benchmark accuracy tables and ablations to compare Stable-OPD with OPD and other training approaches.Table 1 covers six mathematical benchmarks for Qwen2.5-Math-7B, while Table 2 studies KL regularization and mixture distillation.
- Knowledge Distillation: Knowledge distillation trains a student under guidance from a stronger teacher, commonly by matching conditional next-token distributions.The cited background describes token-level distillation using forward KL divergence.
- On-Policy Distillation: OPD trains on trajectories sampled from the current student policy while using teacher-provided per-token guidance.This aligns supervision with the student’s visitation distribution and reduces off-policy distribution shift.
7. Conclusion
The paper identifies OPD’s truncation-repetition collapse as a failure mode caused by student-induced sampling interacting with likelihood-based distillation objectives. Stable-OPD combines divergence control and mixture distillation, stabilizing training and improving average performance by 7.2% over standard OPD baselines.
- Conclusion: OPD can undergo rollout length inflation, truncation collapse, and repetition saturation as repetitive tokens dominate gradient updates.The paper describes this as a self-reinforcing feedback loop under student-induced data collection.
- Conclusion: Stable-OPD combines a reference-based divergence constraint with rollout mixture distillation to stabilize OPD training.These mechanisms target policy drift and the dominance of degenerate on-policy trajectories.
- Conclusion: 7.2% average performance improvement is reported for Stable-OPD compared with standard OPD baselines across mathematical reasoning benchmarks.The conclusion presents this as the aggregate outcome across multiple benchmarks.
C. Baseline Methods.
The benchmark suite compares Stable-OPD with supervised and reinforcement-learning baselines across six mathematical reasoning benchmarks and two student scales.
- Stable-OPD is benchmarked against SFT, GRPO, SimpleRL-Zero, Oat-Zero, PRIME-Zero, OpenReasonerZero, and standard OPD.Experiments use Qwen2.5-Math-1.5B and Qwen2.5-Math-7B backbones.
D.1. Additional Experiment Results on More Base models
On the 1.5B backbone, Stable-OPD improves average accuracy over standard OPD and achieves the best performance across the evaluated benchmarks.
- 7.2 points: Stable-OPD raises average accuracy from 28.9% to 36.1% on the 1.5B backbone.The method achieves the best performance in this comparison.
- Standard OPD fails to match the improvements achieved by SFT and GRPO, suggesting that training instability limits its effectiveness.
D.2. Additional Experiment Results on More Teacher models
Stable-OPD remains effective with different teacher models, with the strongest average result reported using OpenThinkerV3 and benchmark-specific wins for both teachers.
- 36.1% average accuracy: Stable-OPD with OpenThinkerV3 achieves the best average performance.It also reaches 73.9% on MATH-500 and 37.4% on Olympiad.
- Stable-OPD distilled from DeepSeek-R1-Distill-7B performs best on Minerva, AIME24, and AIME25.The reported accuracies are 32.7%, 14.6%, and 17.2%, respectively.
- OpenThinkerV3 improves average accuracy over R1-Distill-7B for the same student, consistent with stronger teacher supervision.
E. More Dynamics Analysis of OPD
OPD’s accuracy fluctuations align with abrupt changes in teacher-guided training signals, particularly advantage estimates and teacher log-probabilities, alongside truncation and repetition shifts.
- Increased truncation or repetition often co-occurs with synchronized spikes in teacher-guided signals and sudden performance fluctuations.
- The analysis tracks truncation and repetition dynamics for rollout and evaluation against accuracy during OPD training.
- Sudden accuracy changes often align with abrupt shifts in teacher log-probabilities and advantage estimates.