Source-linked AI summary
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, Lijie Wen
TL;DR
On-policy self-distillation provides dense supervision, but privileged conditioning can shift its token-level signal toward style rather than task correctness, destabilizing training or shrinking responses. RLCSD contrasts correct and incorrect privileged hints to suppress this drift, and experiments across Qwen3 and Olmo reasoning tasks show consistent gains over GRPO and prior OPSD methods while maintaining stable training dynamics. The contrastive principle also improves existing OPSD methods and extends to broader on-policy distillation.
Problem
Privilege-induced style drift concentrates on-policy self-distillation’s learning signal on stylistic rather than task-bearing tokens, producing training instability or response-length shrinkage.
Method
RLCSD contrasts teacher–student gaps under correct and incorrect privileged hints and integrates the resulting signal into RLVR as verifier-anchored token-level advantage modulation.
Results
RLCSD consistently outperforms GRPO and prior OPSD methods across mathematical and logical reasoning tasks while maintaining stable entropy and response length.
Takeaways & Limitations
The contrastive principle improves existing OPSD methods and provides a general way to purify privileged token-level signals in on-policy distillation.
Takeaways & Limitations
The verifier-grounded trajectory-level update direction does not distinguish which intermediate tokens are responsible for the final outcome.
Abstract
from arXiv · showhide
On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs. We term this pathology \emph{privilege-induced style drift}, which destabilizes training or causes response length to shrink. To address this, we propose \textbf{RLCSD} (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing the style shift that conditioning on a hint tends to induce regardless of correctness, and yielding a signal that is more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to the broader cross-model on-policy distillation setting.
1 Introduction
The paper identifies privilege-induced style drift as a failure mode of on-policy self-distillation and proposes RLCSD to isolate correctness-relevant token supervision. Across reasoning benchmarks, RLCSD improves over GRPO and prior OPSD methods while maintaining stable training and response length.
- On-policy distillation addresses GRPO’s sparse outcome-level reward by providing dense token-level supervision along student-sampled trajectories.
- Privilege-induced style drift shifts the signal toward stylistic tokens, causing instability, entropy or length explosion, and premature response-length shrinkage.
- RLCSD contrasts teacher–student gaps under correct and incorrect privileged hints, subtracting shared stylistic effects to emphasize task correctness.
- RLCSD is evaluated on Qwen3 models at 1.7B, 4B, and 8B scales and Olmo-3-7B-Think across mathematical and logical reasoning tasks.
- RLCSD consistently outperforms GRPO and prior OPSD baselines while preserving stable training and response length, unlike competing methods that destabilize or shrink responses.
2 Related Work
Related work develops dense on-policy distillation to complement RLVR’s sparse rewards, but practical teacher-access constraints and privilege-induced style drift limit existing approaches. RLCSD targets the latter through contrastive cancellation of correctness-orthogonal effects.
- RLVR methods such as GRPO estimate advantages from grouped outcome rewards without a critic network, but these rewards remain sparse for long reasoning trajectories.
- On-policy distillation supplies dense per-token supervision by measuring teacher–student distribution gaps along the student’s sampled trajectories.
- Existing methods explore alternative divergences and black-box or training-efficient variants to trade off mode coverage, mode seeking, and resource demands.
- OPD commonly requires teacher logits, shared vocabulary, and a separately served teacher, creating access, compatibility, memory, and latency constraints.
- Existing OPSD methods rely on one-sided positive privileged contexts and therefore inherit privilege-induced style drift; RLCSD removes it through symmetric contrastive cancellation.
3 Method
RLCSD identifies privilege-induced style drift in OPSD and counters it by contrasting correct and incorrect hinted teachers under matched templates. The resulting signal suppresses shared stylistic effects, emphasizes task-bearing tokens, and modulates verifier-anchored RL updates.
- Problem: Privilege-Induced Style Drift: Privilege-induced style drift shifts OPSD influence toward stylistic tokens, including shorter and more assertive phrasing, rather than correctness-bearing tokens.The style-token mean is 0.263 versus 0.083 for task tokens, roughly a 3× difference.
- Problem: Privilege-Induced Style Drift: Existing OPSD methods can exhibit entropy-driven instability with response-length explosion or premature response-length shrinkage.The skewed token signal is reported as the source of these two failure modes.
- Verifier-Anchored Modulation of the Advantage: RLCSD converts the contrastive signal into bounded modulation and applies it to the verifier-derived GRPO advantage through masking, clamping, and a two-path clipped loss.The mask selects tokens with sufficiently strong contrastive signal for modulation.
- Contrastive Token-Level Signal: RLCSD forms a contrastive token signal by comparing teacher–student gaps under correct and incorrect hints drawn from the same rollout group.Correct and incorrect rollouts are identified by a binary verifier, while matched prompt templates make the comparison symmetric.
- Contrastive Token-Level Signal: Averaging K independently sampled negative hints stabilizes the contrast against heterogeneous error types.A single incorrect hint may represent a mismatched error type and provide an unreliable counter-signal.
- Contrastive Token-Level Signal: The contrastive signal substantially reduces the style–task separation and shifts weight toward task-bearing tokens.The effect is reported in the task/style analysis of |e_ctr,t| relative to |e_c,t|.
4 Experiments
Experiments across Qwen3 and Olmo-3 models evaluate RLCSD against GRPO and OPSD baselines on mathematical and logical reasoning, including training stability and contrastive-hint ablations. RLCSD consistently improves performance while preserving stable optimization, and contrastive hints generalize across OPSD methods.
- Experimental setup: Experiments use four reasoning models across mathematical and logical reasoning tasks, with in-domain and out-of-domain Knight-and-Knaves evaluation.The models are Qwen3-1.7B, Qwen3-4B, Qwen3-8B, and Olmo-3-7B.
- Main results: RLCSD consistently outperforms GRPO and prior OPSD methods across model scales and task settings.Average gains over the Base model range from +1.8 to +4.3 on math and +6.8 to +14.4 on logical reasoning across the reported model families.
- Training dynamics: Existing OPSD methods fail through entropy explosion and training collapse or through premature response-length shrinkage.OPSD, SDPO, and SRPO exhibit entropy growth and collapse, whereas RLSD exhibits response-length shrinkage.
- Training dynamics: RLCSD maintains stable entropy and response length while augmenting GRPO’s outcome signal with dense contrastive token-level supervision.Its stable optimization behavior is reported alongside stronger final validation performance.
- Contrastive hints: Contrastive hints improve existing OPSD methods as a general-purpose component, while one-sided hint-source changes have little effect.Contrast improves every evaluated method on nearly every metric; OPSD gains +2.3 on the K&K average and RLSD gains +2.2 on the math average.
- Ablations and broader insights: Contrastive construction mitigates both entropy-explosion and response-length-shrinkage failure modes by purifying privileged token-level signals.Removing K-marginalization or two-path aggregation also reduces performance, showing that these design choices preserve reliable contrastive supervision.
- Ablations and broader insights: KL analysis indicates that privileged token-level signals can concentrate on discourse and structural tokens rather than task-bearing content.The paper extends this concern to cross-model on-policy distillation and proposes per-token KL ranking as a training-free teacher-selection diagnostic.
5 Conclusion
RLCSD addresses privilege-induced style drift by contrasting correct and incorrect hints and integrating the resulting signal into verifier-anchored RLVR. Across mathematical and logical reasoning tasks, it consistently outperforms GRPO and prior OPSD methods while maintaining stable training dynamics.
- RLCSD contrasts correct and incorrect privileged hints under a shared template to mitigate privilege-induced style drift.
- Across Qwen3 and Olmo models on mathematical and logical reasoning, RLCSD consistently outperforms GRPO and prior OPSD methods.
- The contrastive principle improves existing OPSD methods and extends to broader cross-model on-policy distillation.
A Additional Analysis of Reference-Hint Selection
This appendix develops two refinements to the contrastive method: K-marginalization of negative hints and exclusion of the target rollout from the hint pool. Together, they address unreliable vanilla contrastive signals and restore verifier-aligned behavior.
- The appendix introduces K-marginalization of negative hints as a refinement to the contrastive signal.This refinement is analyzed in Appendix A.1.
- It also examines excluding the target rollout from the hint pool as a second refinement.This refinement is analyzed in Appendix A.2.
- Together, the refinements explain why vanilla contrastive signals are unreliable and how to restore a clean, verifier-aligned signal.
A.1 Negative-Hint Selection: Case Analysis and K-Marginalization
The appendix identifies dissimilar-error negative hints as a failure mode for the naive contrastive signal and uses K-marginalization to recover verifier-aligned behavior. It also shows that self-conditioning can make teacher probabilities over-confident, motivating exclusion of the target rollout from the hint pool.
- Case Analysis: Correct rollouts and incorrect rollouts with similar-error negative hints produce contrastive signals whose signs align with the verifier.Correct rollouts yield strongly positive signals, while similar-error incorrect rollouts yield strongly negative signals.
- Case Analysis: Incorrect rollouts paired with dissimilar-error negative hints can produce contrastive signals of either sign.Because the negative hint makes a different mistake, its branch loses the expected positive sign.
- Case Analysis: A substantial fraction of incorrect rollouts can therefore be encouraged rather than penalized, injecting noise into optimization.Selecting error-matched negative hints online with an LLM judge is described as prohibitively expensive at RL-training scale.
- K-Marginalization: K-marginalization averages probability estimates across K independently sampled negative hints instead of selecting an error-matched hint online.This spreads the teacher’s mass across the negative branch and avoids the expense of online LLM-as-judge selection.
- Empirical Validation: Correct rollouts produce a positive signal in 0.989 of rollouts, while least-similar negative hints center the signal near zero with roughly half the rollouts having the wrong sign.The least-similar hints isolate the dissimilar-error case, whereas K-marginalization restores a cleanly negative-skewed signal.
- Reference-Hint Selection: Using the target rollout as its own hint shifts teacher probabilities toward extreme over-confidence, unlike using a sibling hint.Figure 9 reports that sibling hints leave the distribution close to the no-hint case, motivating target-rollout exclusion.
B Implementation Details
This section supplies implementation details that were not fully described in the main text. It supplements the earlier description of the method with additional operational information.
- The section presents full implementation details for the method.
- These details were not fully described in Section 4.1 of the main text.
- The section therefore extends the implementation description beyond the main-text account.
B.1 Shared training setup
All six methods share the same training infrastructure and within-configuration settings, while logical-task learning rates differ for RLCSD because other methods collapse more severely at the higher rate.
- All methods use veRL’s PPO trainer with FSDP2 actor sharding, vLLM rollouts, full-parameter training, and 8×H20 GPUs.
- Within each task and backbone configuration, GRPO, OPSD, SDPO, SRPO, RLSD, and RLCSD share settings; only the per-step loss differs.
- Mathematics tasks use a learning rate of 1 × 10^-6 for all six methods.
- Logical tasks use 5 × 10^-6 for RLCSD but 1 × 10^-6 for other methods, which experience more severe training collapse at the higher rate.
B.2 Method-specific hyperparameters
Method-specific hyperparameters are listed separately, with values taken from the original papers where those papers specify them.
- Table 8 lists the hyperparameters that vary across methods and follows the original papers wherever values are specified.
C Vocabulary Partitioning for Task / Style Statistics
The vocabulary partition classifies decoded math-task tokens into style, task, or neutral categories using ordered normalization and matching rules.
- Each decoded token is stripped of Qwen3’s leading-space marker, lowercased, and assigned by the first matching rule.
- Empty or whitespace-only tokens are labeled style, while digits, math operators, LaTeX commands, and selected mathematical symbols are labeled task.
- A math wordlist containing terms such as mod, prime, gcd, sqrt, frac, and named Greek symbols is labeled task.
- Pure punctuation and literal newline tokens are labeled style.
- Discourse connectives, hedges, scaffolding terms, and closed-class function words are labeled style; all remaining tokens are neutral.
D Computational Cost Analysis
RLCSD adds teacher-side computation for wrong-hint scoring, but rollout generation dominates runtime and its total step time remains comparatively favorable.
- 13.46s per step is spent on RLCSD teacher scoring, compared with about 9s for one-sided methods such as RLSD and OPSD.
- 473.89s per step is spent on rollout generation, making the additional teacher scoring a minor fraction of generation cost.
- RLCSD is the third fastest method overall, behind GRPO and RLSD, and faster than dense-distillation baselines OPSD, SDPO, and SRPO.
- RLCSD’s favorable end-to-end runtime is attributed to sampled-token modulation, whereas dense baselines perform vocabulary-level token distillation during optimization.