PaperlayerView exact revision on arXiv

Source-linked AI summary

SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation

Chenyang Jiang, Changhan Huang

arXiv:2608.14647v1cs.LGcs.AI

TL;DR

Dirty-history rollouts make multi-turn on-policy self-distillation brittle because later turns condition on erroneous intermediate replies. SMOPD masks low-entropy middle-turn tokens, improving SHARDED accuracy by 1.0–2.5pp across evaluated Qwen3 scales, while correctness scaling is inconsistent or harmful. These results support token-level uncertainty as the more consistent stabilization signal in the evaluated setting.

  • Problem

    Multi-turn language-model training requires stabilization methods beyond single-turn evaluation and costly turn-level demonstrations.

  • Method

    SMOPD selectively masks the lowest-entropy middle-turn tokens in a clean-teacher/dirty-student distillation loss.

  • Results

    SMOPD improves SHARDED accuracy by 1.0–2.5pp across evaluated 1.7B–8B scales, whereas correctness scaling is harmful without masking and inconsistent when combined with it.

  • Takeaways & Limitations

    In the evaluated LiC/Qwen3 setting, entropy masking is the most consistent positive intervention among tested configurations.

  • Takeaways & Limitations

    Evidence is limited to LiC, Qwen3-Instruct models, math-heavy held-out evaluation, mostly 100-step training, and largely single-seed cross-scale results.

Abstract

from arXiv · show

Dirty-history rollouts make multi-turn on-policy self-distillation (OPSD) brittle: once a student emits an erroneous intermediate reply, later turns are conditioned on that reply, and uniform distillation can spend loss on tokens that carry little corrective signal. We introduce SMOPD (Selective Masking for On-Policy Distillation), a loss-only stabilization method for multi-turn OPSD. For each generated middle-turn reply, SMOPD ranks token positions by student entropy and removes the lowest-entropy 20% from the clipped generalized Jensen-Shannon distillation loss; final-answer and FULL-preservation losses are unchanged. This design targets token-level uncertainty rather than coarse trajectory outcomes, adds no parameters, and has zero inference-time overhead. We compare SMOPD with a correctness-scaling variant that multiplies a common detached reliability proxy using final-answer correctness. On LiC with Qwen3 models, SMOPD improves SHARDED-view accuracy by 1.0-2.5 percentage points in single-seed 1.7B, 4B, and 8B comparisons, and a small 4B multi-seed check shows a +1.7pp mean SHARDED gain over baseline (two-tailed p = 0.022). Adding the outcome scalar is harmful without masking at 1.7B (-4.0pp) and remains scale-dependent when combined with masking (+1.3pp at 4B, neutral at 1.7B, and -0.5pp at 8B). These archived aggregate results suggest that token-level uncertainty is a more reliable stabilization signal than scalar final-answer correctness in this evaluated dirty-history OPSD setting, while leaving causal mechanism tests and broader benchmark validation to future work.

1 Introduction

Multi-turn OPSD is challenged by dirty history, where student errors compound across turns and distort later conditioning. SMOPD addresses this with loss-only masking of low-entropy middle-turn tokens, and entropy masking shows the most consistent positive evidence across evaluated Qwen3 scales.

  • Motivation: Dirty history compounds student errors across turns, pulling multi-turn OPSD away from the reference distribution.Each intermediate response conditions the next prompt, so errors distort subsequent rollout contexts.
  • Method: SMOPD applies percentile-based student-entropy selection inside the middle-turn GJS distillation loss.It is a multi-turn clean-teacher/dirty-student stabilization component adapted from entropy-guided dynamic sampling.
  • Results: −4.0pp SHARDED-view accuracy occurred when outcome correctness scaling was used without masking.When combined with masking, the effect was scale-dependent: neutral at 1.7B, +1.3pp at 4B, and −0.5pp at 8B.
  • Implication: The findings support prioritizing token-level uncertainty before adding coarse outcome signals in multi-turn OPSD pipelines.The paper presents this as a practical starting point based on the evaluated ablations, while separating the mechanism hypothesis from archived aggregate accuracy evidence.

2 Related Work

SMOPD is positioned as a narrow loss-side intervention within MAIGO-style clean-teacher/dirty-student multi-turn OPSD, rather than a replacement for broader OPD or outcome-guided frameworks. Its distinguishing mechanism is selective per-token entropy masking of middle-turn distillation, alongside unchanged answer-turn and FULL-preservation losses.

  • Positioning: SMOPD is a narrow loss-side intervention within MAIGO-style clean-teacher/dirty-student training, not a replacement for broader OPD frameworks or richer outcome-guided credit assignment.MAIGO extends OPSD to multi-turn LiC by cleaning teacher history and preserving FULL-view competence; SMOPD instead studies selective masking inside the GJS loss.
  • Multi-turn OPSD: SMOPD averages all eligible middle-turn losses while selectively masking tokens inside the GJS distillation loss.The student generates dirty-history trajectories, while the EMA teacher supplies reference logits under clean history on the same response tokens.
  • Stabilization signals: Entropy-based token selection places SMOPD closer to token-level OPD methods such as REOPOLD, EOPD, and AOPD than to global regularization approaches.Related methods select high-entropy tokens or alter KL direction using entropy, whereas SMOPD uses entropy specifically for middle-turn distillation.
  • Outcome guidance: Outcome-guided methods use richer steering, dual-path weighting, or process supervision, whereas SMOPD’s outcome variant uses only a scalar correctness-dependent multiplier.CREDIT argues that coarse trajectory outcomes are insufficient for OPD credit assignment, motivating comparison with a weaker outcome signal.
  • Loss scope: SMOPD leaves answer-turn and FULL-preservation RKL losses unmasked while applying entropy masking only to middle-turn GJS losses.This separates the proposed loss-side intervention from broader changes to answer competence or FULL-view preservation.

3 Motivation: Token-Level Confidence in Multi-Turn OPSD

SMOPD motivates token-level entropy masking because middle-turn tokens differ in corrective value under dirty-history OPSD. It retains uncertain positions using a per-reply percentile, removing the lowest-entropy tokens from the middle-turn loss while avoiding a global threshold.

  • Motivation: Under dirty history, low-entropy tokens often represent predictable or locally committed choices, whereas high-entropy tokens more often mark decision points.Examples of decision points include operators, variable bindings, and logical connectors; confidence may also become miscalibrated after earlier student errors enter the context.
  • Entropy masking: SMOPD retains uncertain token positions and removes the lowest-entropy positions from the middle-turn loss as a stabilization hypothesis.The method targets positions where correction is more likely to matter.
  • Entropy masking: β = 0.8 retains the highest-entropy 80% of tokens within each generated middle-turn reply.A per-reply percentile is used because entropy scales shift across tasks, turns, and training stages.

4 Method

SMOPD trains a dirty-history student against an EMA teacher evaluated in clean contexts, using aligned token-level distillation losses. It selectively retains high-entropy middle-turn tokens while preserving answer-turn and FULL-branch objectives, with optional reliability and outcome weighting.

  • Clean-teacher / dirty-student framework: The student generates dirty-history multi-turn rollouts, while a frozen EMA teacher supplies logits from clean, single-turn contexts.Middle turns use only user-turn context, whereas the answer turn uses the canonical FULL-view specification.
  • Token alignment and teacher forcing: Student and teacher distributions are evaluated on the same generated token sequence despite differing dirty-history and clean-history contexts.This shared sequence avoids variable-length alignment between independent continuations.
  • Distillation losses: Middle turns use clipped GJS with βmid = 0.5, while the answer turn uses clipped RKL with βfinal = 1.0.All per-token GJS and RKL terms are clipped at ϵclip = 0.5; FULL-branch RKL is additionally computed with probability ρ = 0.2.
  • Entropy masking: SMOPD retains the top β fraction of middle-turn tokens by student entropy and zeroes the lowest-entropy positions, using β = 0.8.The operation is loss-side only: it adds no parameters, changes no generation, and has zero inference-time overhead.
  • Reliability and outcome weighting: A detached reliability proxy weights all reported configurations, with optional outcome scaling at η = 0.2 that multiplies incorrect outcomes by 1.2 and correct outcomes by 0.8.The implementation averages all eligible middle-turn losses in each rollout rather than sampling one eligible turn.
  • Training schedule: Entropy masking is disabled during the first 33% of training steps by setting β = 1.0, without a reward-filtering phase.The EMA teacher is updated after each optimizer step with λ = 0.99.

5 Experimental Setup

The experiments use LoRA-fine-tuned Qwen3-Instruct models at 1.7B, 4B, and 8B scales on the LiC multi-turn benchmark. Evaluation emphasizes SHARDED accuracy under dirty-history conditions, comparing four OPSD stabilization configurations with archived aggregate evidence.

  • Models and training: Qwen3-Instruct models at 1.7B, 4B, and 8B parameters are fine-tuned with LoRA applied to all attention and feed-forward linear projections.LoRA uses rank r = 64, α = 128, and dropout 0.0.
  • Models and training: 100 training steps use AdamW at learning rate 5 × 10^-6, with EMA decay λ = 0.99 and FULL preservation probability ρ = 0.2.Entropy masking is disabled during the first 33% exploration phase, and all experiments use seed 42 unless noted otherwise.
  • Benchmark and evaluation: 12,165 multi-turn LiC training examples cover math, database, code, and action-planning domains, while evaluation uses 200 held-out math examples under FULL-view and SHARDED-view protocols.SHARDED accuracy is the primary metric for multi-turn reasoning under dirty-history conditions.
  • Evidence scope: All tabulated and plotted accuracies are aggregate metrics traceable to archived run artifacts, while token-level diagnostics, per-turn slices, prompt examples, and exploratory parsing outputs are excluded.β = 0.8 is therefore treated as a fixed operating point rather than a proven optimum, and the source package excludes weights, checkpoints, logs, and raw evaluation output.
  • Compared configurations: Four configurations compare baseline, +entropy, +outcome, and combined stabilization, with β = 0.8 for entropy masking and η = 0.2 for the correctness scalar.Two archived training runs with identical hyperparameters are averaged to estimate run-to-run variation.

6 Results

Entropy masking produced the strongest and most consistent SHARDED gains across the ablations and evaluated model scales, while outcome-scalar combinations were weaker and scale-dependent. Multi-seed results support a +1.7pp entropy-masking gain at 4B, but limited seeds and run-to-run variation temper interpretation.

  • Core ablation: +2.5pp SHARDED accuracy over baseline was the largest ablation gain for entropy masking, alongside a +2.5pp FULL improvement.Adding the outcome scalar without masking lowered SHARDED by 4.0pp.
  • Core ablation: 2.5pp separated the two identical combined-configuration runs, which achieved 64.5% and 67.0% SHARDED.This illustrates substantial run-to-run variation under identical hyperparameters.
  • Core ablation: 0.7pp below baseline was the 1.7B combined mean, at 65.8%, indicating no clear benefit over masking alone.The combined mean may slightly degrade performance at the 1.7B scale.
  • Cross-scale results: +2.5pp, +1.5pp, and +1.0pp were the SHARDED gains from entropy masking at 1.7B, 4B, and 8B, respectively.The gains were positive at every evaluated scale but small and descriptive.
  • Cross-scale results: −0.7pp, +1.3pp, and −0.5pp were the combined SHARDED deltas at 1.7B, 4B, and 8B, respectively.The combined configuration therefore remained scale-dependent; an initial 4B +7.0pp outlier was reduced to a +1.3pp mean gain after replication.
  • Multi-seed reliability: +1.7pp was the 4B entropy-masking mean SHARDED gain across three seeds (two-tailed p = 0.022), whereas combined achieved +1.3pp (two-tailed p = 0.101).With only three seeds, the p-values are rough reliability checks rather than precise population estimates.

7 Analysis

In the evaluated LiC/Qwen3 settings, entropy masking is the most consistent positive intervention, but the aggregate results do not establish its general optimality or causal mechanism. Outcome correctness scaling is noisy and unstable, while single-seed effects warrant replication with multiple seeds or within-configuration checks.

  • Empirical scope: Entropy masking is the most consistent positive intervention in the evaluated LiC/Qwen3 configurations, without establishing general optimality or a unique causal explanation.The archived results do not show that β = 0.8 is optimal or that overconfidence suppression uniquely causes the gains.
  • Mechanistic interpretation: Three mechanisms are compatible with the aggregate pattern: suppressing confidently wrong updates, filtering routine tokens, and emphasizing harder decision points.These interpretations fit the smaller gains at larger scales but require token-level calibration logs for verification.
  • Mechanistic interpretation: Alternative explanations include reduced supervision, altered gradient scale, and confidence-unrelated token artifacts, so mechanism claims remain interpretive rather than causal.The proposed controls are random, divergence-based, fixed-denominator, and direct entropy-error calibration masking.
  • Outcome scaling: Outcome correctness scaling applies a noisy delayed scalar across middle turns, potentially amplifying variance and conflicting with clean-context teacher-student divergence.Intermediate quality can diverge from final-answer correctness because errors alter later-turn input distributions and trajectories can recover.
  • Outcome scaling: Outcome scaling is harmful without masking at 1.7B and inconsistent with masking, consistent with coarse trajectory outcomes being insufficient without more local signals.Possible alternatives include larger-scale use, turn-level correctness, or process feedback.
  • Reproducibility: +7.0pp was the 4B combined single-seed outlier, versus +1.3pp replicated mean; 1.7B combined runs differed by 2.5pp SHARDED under identical hyperparameters.Core OPSD comparisons should therefore include multi-seed results or within-configuration replication.

8 Conclusion · Appendix

SMOPD improves SHARDED accuracy across evaluated Qwen3 scales with no inference overhead, while outcome-guided correctness scaling is harmful or inconsistent. The authors recommend uncertainty-first stabilization but limit conclusions because evidence comes from archived, narrow, mostly single-seed experiments.

  • 8 Conclusion: SMOPD improves SHARDED accuracy by 1.0–2.5pp across evaluated 1.7B–8B scales with no added parameters or inference overhead.It is presented as a selective token-entropy masking variant for dirty-history multi-turn OPSD.
  • 8 Conclusion: Outcome-guided correctness scaling is harmful without masking at 1.7B (−4.0pp) and inconsistent with masking across scales.The combined effect is +1.3pp at 4B, neutral at 1.7B, and −0.5pp at 8B.
  • Practical Takeaways: Start with token-level uncertainty before adding scalar trajectory outcomes, and validate outcome signals at the target scale.Suggested practices also include a small step sweep, β = 0.8 as an initial operating point, and multi-seed reporting.
  • Practical Takeaways: Entropy masking is recommended as the first stabilization check because it is a drop-in loss modification with zero inference cost.The authors caution that β = 0.8 is an operating point to sweep, not a default law, and advise against relying on binary outcome scaling alone.
  • Limitations: The evidence is limited to LiC, Qwen3-Instruct models, math-heavy held-out evaluation, LoRA rank 64, and mostly 100-step training.Cross-scale results are single-seed except for the 4B reliability check, and β = 0.8 is not an established optimum.
  • Data, Code, and Compute Availability: Quantitative claims rely on archived aggregate artifacts, while model weights, checkpoints, raw logs, and raw evaluation outputs are not included.The experiments used one NVIDIA RTX PRO 6000 Blackwell GPU (96GB VRAM), with approximately 40–50 minutes per 100-step run.
  • Broader Impact: If the result transfers, entropy masking could improve multi-turn OPSD data efficiency without changing inference behavior, with no masking-specific risks identified beyond general capable-LLM deployment risks.The authors acknowledge LiC maintainers for standardized multi-turn evaluation protocols.

A. Reproducibility and Experimental Protocol · B. Source-to-Claim Provenance

The paper consolidates its reported protocol and maps quantitative claims to archived aggregate artifacts for auditability. It does not provide checkpoints, raw outputs, or several unarchived diagnostics needed for full reproducibility.

  • A. Reproducibility and Experimental Protocol: Table 8 consolidates the experimental settings used for reported runs, making the arXiv version auditable without reconstructing the protocol.The settings are collected in one place rather than requiring readers to recover them from method and results sections.
  • A. Reproducibility and Experimental Protocol: All reported accuracy values are aggregate evaluation results.The arXiv source package includes the manuscript, style file, and figures needed for compilation.
  • A. Reproducibility and Experimental Protocol: The arXiv source package omits model checkpoints, model weights, raw logs, and raw evaluation outputs.These omissions limit reproduction beyond the archived aggregate results.
  • B. Source-to-Claim Provenance: Table 9 maps the paper’s main quantitative claims to archived aggregate artifacts used for this arXiv draft.It functions as a provenance checklist, not a replacement for a future full reproducibility release.
  • B. Source-to-Claim Provenance: The manuscript excludes unarchived token-level diagnostics, per-turn slices, prompt-level examples, qualitative case studies, and masking-ratio sensitivity rows as quantitative evidence.These analyses are deferred until the corresponding raw artifacts are preserved.
  • B. Source-to-Claim Provenance: Future releases should report deferred analyses only after preserving their corresponding raw artifacts.The stated exclusions cover diagnostics, slices, examples, case studies, and masking-ratio sensitivity rows.

C. Statistical Interpretation Notes

The reported gains should be interpreted cautiously because the 200-example evaluation yields coarse accuracy increments, while the three-seed 4B reliability check has limited statistical power. The combined configuration also excludes a visible +7.0pp SHARDED outlier from its reliability mean.

  • Evaluation granularity: 1.0pp and 2.5pp SHARDED differences correspond to only two and five evaluation examples, respectively, so single-seed cross-scale comparisons are directional descriptive evidence.The held-out evaluation set contains 200 examples, producing 0.5 percentage-point accuracy increments.
  • Statistical power: With n = 3, Welch’s two-tailed p-values are rough indicators that support only a cautious +entropy reliability claim, not a general scaling law or optimal masking ratio.The 4B reliability check uses three seeds per configuration; directional one-tailed values are discussed because the intervention hypothesis is directional.
  • Outlier handling: +7.0pp SHARDED from an early 4B seed-42 combined run is excluded from the multi-seed reliability mean but retained as a warning about single-seed OPSD variance.The combined row in Table 5 estimates the subsequent reliability runs rather than incorporating the outlier.

D. Implementation Notes Relative to Prior Work · E. Per-Seed Breakdown for Multi-Seed Validation · F. Step Sweep Figure

The implementation adapts MAIGO- and REOPOLD-related ideas to multi-turn OPSD without reproducing either algorithm exactly. The validation notes clarify seed provenance, while the step sweep shows divergent SHARDED and FULL behavior as training steps increase.

  • D. Implementation Notes Relative to Prior Work: SMOPD averages over all eligible non-empty middle turns, unlike MAIGO’s single-turn estimator, within a clean-teacher / dirty-student training family.The detached reliability proxy is common across reported configurations; the outcome variant additionally multiplies it by a correctness scalar.
  • D. Implementation Notes Relative to Prior Work: The entropy threshold is computed within each generated middle-turn reply, adapting REOPOLD’s selection rule to multi-turn OPSD rather than single-turn refinement.The schedule delays entropy masking only during the first 33% of training steps and does not implement reward-based filtering.
  • D. Implementation Notes Relative to Prior Work: Teacher forcing aligns student and EMA-teacher divergences on the same student-generated sequence despite differing contexts.This avoids aligning independently generated responses.
  • D. Implementation Notes Relative to Prior Work: Only middle-turn GJS terms are entropy-masked; answer-turn RKL and FULL-preservation RKL remain unmasked.The masking distinction applies to the divergence terms computed through teacher forcing.
  • E. Per-Seed Breakdown for Multi-Seed Validation: Table 11 is a provenance breakdown rather than a strictly paired-seed design for the 4B reliability summary.Baseline and +entropy use paper-canonical Phase 3 seed-42 rows, while combined seed-42 is a subsequent reliability rerun excluding the earlier +7.0pp outlier.
  • E. Per-Seed Breakdown for Multi-Seed Validation: Seed 42 is the cross-scale run for baseline and +entropy, whereas the combined seed-42 row is a subsequent reliability rerun.The earlier combined seed-42 outlier is reported separately in §7.3.
  • F. Step Sweep Figure: SHARDED peaks at 200 steps then decreases, while FULL rises at 400 steps in the 1.7B baseline sweep.The figure suggests possible overfitting to single-turn performance.
Loading 2608.14647v1…