PaperlayerView exact revision on arXiv

Source-linked AI summary

ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

Junming Liu, Jicheng Wang, Yifeng He, Hao Chen, Jianzhong Qi

arXiv:2609.32448v1cs.AIcs.CL

TL;DR

Direct AR-to-DLM distillation compares predictions made with different contexts: a causal teacher sees a left prefix, while a DLM can also use tokens to the right. ForkLeft separates student rollouts from aligned teacher supervision and returns to native decoding at inference. It improves Efficient-DLM-4B on all ten benchmarks, with gains also reported across teachers and a second student family.

  • Problem

    AR teachers predict from a left prefix, whereas DLM students can condition on tokens on both sides, leaving a context mismatch in direct distillation.

    Source: §1 · p. 1

  • Method

    ForkLeft commits uncertain student positions first, routes supervision by answer correctness, distills on the student’s shared left prefix, and uses confidence-first native decoding at inference.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 5

  • Results

    ForkLeft improves Efficient-DLM-4B on all ten benchmarks, including MATH500 from 72.60% to 79.60%, and the full design wins the order–view comparison on every benchmark.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The distilled 4B and 8B students exceed published matched-scale diffusion models on seven benchmarks while retaining native parallel generation.

    Source: §1 · p. 2, §6 · pp. 9–10

    Training requires verified answers, teacher queries, and a shared tokenizer, and a correct teacher answer does not guarantee correct distributions on every student prefix.

    Source: §5 · p. 9

Abstract

from arXiv · show

Autoregressive Next-Token Prediction (NTP) has enabled strong reasoning capabilities in language models, while Diffusion Language Models (DLMs) offer flexible token orders and parallel generation. We ask whether DLMs can acquire NTP-style reasoning through distillation without giving up their native generation process. Direct distillation, however, faces a fundamental mismatch: an autoregressive teacher predicts from a left prefix, whereas a DLM can condition on tokens on both sides. We introduce ForkLeft, a distillation framework that resolves this mismatch by separating the student's rollout from teacher supervision. During training, the student first performs entropy-first rollouts that commit uncertain positions and expose potential forks. We then fix the resulting student prefix and distill an NTP teacher under the same context, with answer correctness determining the supervision source. At inference, the student returns to its native confidence-first parallel decoding. With Qwen3-30B-A3B-Base, ForkLeft improves Efficient-DLM-4B on all ten benchmarks, raising MATH500 from 72.60% to 79.60% and consistently outperforming three alternative designs. The gains scale with teacher strength and generalize to SDAR-4B with only $500$ updates. At matched scale, the distilled 4B and 8B students exceed the published SDAR-Chat and OPDLM models on seven benchmarks, showing that DLMs can learn NTP-style reasoning without sacrificing native parallel generation. Code and datasets will be released upon acceptance.

1 INTRODUCTION

DLMs can generate tokens in parallel and choose their order, but direct AR-to-DLM distillation mismatches the teacher’s left-prefix context with the student’s bidirectional context. ForkLeft exposes uncertain decisions during training, aligns the teacher’s view to a student-written prefix, and retains native decoding; its full design improves the student across ten benchmarks.

1 INTRODUCTION

  • DLMs resolve masked positions iteratively, generate several tokens in parallel, and choose their own generation order, unlike the AR next-token prediction paradigm.The introduction frames transfer of AR reasoning capabilities into this flexible generation paradigm as the motivating question.Source: §1 · p. 1
  • Direct transfer mismatches contexts: the AR teacher predicts from a left prefix, while the DLM may also condition on its own tokens to the target’s right.Thus a disagreement can reflect different available evidence, not simply weaker predictions.Source: §1 · pp. 1–2
  • Existing approaches leave the trajectory and comparison view unresolved: inference-order rollouts defer uncertain forks, while bidirectional comparisons give teacher and student different information.The cited passage also notes that outcome-reward methods assign a terminal scalar across parallel decisions, while OPD supplies token-level distributions but still faces these design choices.Source: §1 · pp. 1–2
  • ForkLeft exposes uncertain forks through entropy-first rollouts, routes supervision by answer correctness, aligns teacher and student on the same student-written left prefix, and restores confidence-first decoding at inference.The teacher does not rewrite the student rollout; selected loss positions are sampled uniformly, so confident errors can also receive supervision.Source: §1 · p. 2
  • Across ten benchmarks, ForkLeft improves Efficient-DLM-4B on every task and the four-way comparison favors entropy-first, prefix-aligned teaching on all ten.The method also adds gains to SFT, scales with teacher strength, generalizes to SDAR-4B, and exceeds matched-scale published diffusion models on seven benchmarks.Source: §1 · p. 2

2 RELATED WORK

Prior work studies post-training, on-policy distillation, and learning from uncertain tokens, but ForkLeft’s related-work distinction is to design both the student trajectory and the teacher’s supervision view.

Post-training diffusion language models

  • Related diffusion post-training includes progressively generated-state training, blockwise SFT, and outcome-reward optimization; JustGRPO constrains RL training to left-to-right order while retaining parallel inference.The passage also connects confidence-driven flexibility concerns with the Confidence Shortcut and harder masked-diffusion training orders.Source: §2 · pp. 2–3

On-policy distillation

  • ForkLeft designs both the trajectory and supervision view for AR-to-DLM transfer, comparing causal teacher and bidirectional student predictions on a shared student-generated prefix.OPDLM supervises student trajectories with an AR teacher; other cited diffusion self-distillation approaches use later denoising states or self-generated answer suffixes.Source: §2 · p. 3

Learning from uncertain tokens

  • ForkLeft uses uncertainty to choose when a token is committed, not to restrict loss positions; uniform sampling can supervise confident errors too.This differs from approaches that concentrate learning signal on high-entropy tokens.Source: §2 · p. 3

3 METHOD

ForkLeft separates response generation from supervision: it exposes uncertain decisions in student rollouts, routes targets using answer correctness, and teaches from aligned prefixes. A training-only confidence signal shapes the priority used by native confidence-first inference.

3 METHOD

  • ForkLeft separates writing from learning: the student generates bidirectionally with uncertain positions first, receives supervision under its own left prefixes, and returns to confidence-first native decoding.Entropy selects rollout commitments, whereas uniform sampling selects loss positions.Source: §3 · p. 3
  • For a target such as C, teacher and student both predict from the completed student prefix AB, with C and its suffix hidden from the teacher’s prediction context.The student’s native generation remains bidirectional; prefix supervision does not require left-to-right generation.Source: §3 · p. 3, §3.2 · p. 4

3.1 EXPOSE THE FORK: ENTROPY-FIRST STUDENT ROLLOUTS

  • High-entropy positions are committed first using the student’s own argmax predictions, leaving any resulting wrong branch visible for on-policy correction.The teacher neither selects nor replaces rollout tokens; under the same prefix-aligned loss, this order adds 2.62 pp on average over low-entropy-first commitments.Source: §3.1 · p. 3

3.2 VERIFY, ROUTE, AND TEACH ON THE SHARED PREFIX

  • Answer-verdict pairs route each response: reinforce the student when both are correct, use teacher distributions when only the teacher is correct, use an available reference when both are wrong, and drop student-only-correct cases.The teacher’s target is predicted under the student’s own prefix; uniformly sampled eligible positions allow confident errors to receive loss.Source: §3.2 · pp. 4–5
  • Under entropy-first rollouts, replacing the student’s bidirectional readout with the shared-prefix view raises mean gain from 0.14 to 4.93 pp.The passage identifies this prefix view as the second comparison axis.Source: §3.2 · p. 5

3.3 SHAPE CONFIDENCE WITH A PREFIX-SENSITIVITY SIGNAL

  • A training-only prefix-sensitivity signal lowers prefix confidence where revealing the prefix changes predictions most, aiming to defer those positions during confidence-first inference.The ranking is a surrogate learned through shared parameters, not an enforced inference rule or token-correctness label.Source: §3.3 · p. 5

3.4 TRAINING OBJECTIVE AND NATIVE INFERENCE

  • At inference, only the student runs: it commits unresolved positions by maximum token probability and recomputes on its native bidirectional state.This confidence-first order reverses training’s entropy-first priority, allowing uncertain positions to use more context before commitment.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate ForkLeft across ten benchmarks, two evaluation protocols, design ablations, teacher strengths, a second student family, and same-scale published diffusion models. The results test whether the method improves students while retaining native evaluation procedures.

4.1 EVALUATION SETUP

  • The evaluation trains EDLM-4B for 2000 updates with Qwen3-30B-A3B-Base and assesses ten benchmarks using native generation or masked-answer-slot likelihood scoring.Generated-answer tasks use each diffusion family's native sampler; both protocols share the OPDLM prompt template and answer scorer.Source: §4.1 · p. 6

4.2 FORKLEFT IMPROVES THE STUDENT ACROSS ALL TEN BENCHMARKS

  • EDLM-4B improves on all ten benchmarks after 2000 updates; MATH500 rises from 72.60% to 79.60%.The student retains its native evaluation procedure, and gains also appear under multiple-choice likelihood scoring.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 PREFIX ALIGNMENT AND ENTROPY-FIRST ROLLOUTS WORK TOGETHER

  • The entropy-first, prefix-aligned configuration wins all ten benchmarks; the prefix view adds 3.80 pp, entropy-first order 1.63 pp, and their descriptive interaction is 1.99 pp.Under the prefix view, entropy-first improves all ten tasks over entropy-last; under entropy-first order, prefix teaching improves all ten over bidirectional teaching.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4 TEACHER DISTRIBUTIONS ADD TO SFT ON THE SAME QUESTIONS

  • On the same teacher-correct, student-wrong questions, adding OPD to SFT improves six benchmarks, while adding SFT to OPD lowers none.GPQA-D is the only larger decrease when OPD is added to SFT; the objectives supply hard targets and teacher distributions over alternatives, respectively.Source: Figure 5 · p. 7, §4.4 · pp. 7–8

4.5 GAINS RISE WITH TEACHER STRENGTH FOR BOTH STUDENT SIZES

  • Mean gains rise with teacher strength for both student sizes, reaching 4.93 pp for EDLM-4B and 3.22 pp for EDLM-8B with Qwen3-30B.Within the tested range, gains had not flattened at the strongest teacher; the authors fit no scaling law.Source: Figure 6 · p. 8, §4.5 · p. 8

4.6 THE RECIPE GENERALIZES TO A SECOND STUDENT FAMILY

  • After 500 updates with Qwen3-30B, SDAR-4B improves on seven benchmarks, including MMLU by 7.06 pp and GPQA-D by 4.55 pp.The other three benchmarks change by at most two items.Source: Figure 7 · p. 8, §4.6 · p. 9

4.7 COMPARISON WITH PUBLISHED DIFFUSION MODELS AT MATCHED SCALE

  • At matched scale, the distilled students score highest among themselves and published SDAR-Chat and OPDLM models on seven benchmarks.The 4B student also exceeds both published 8B models on six of those seven; comparisons use published scores under the study's prompt template and scorer, not matched-budget controls.Source: Table 2 · p. 9, §4.7 · p. 9

5 DISCUSSION AND LIMITATIONS

The discussion reports that distilled students can exceed their teacher on selected benchmarks, while noting that the proposed explanations are not isolated and the method has practical requirements and comparison-scope limits.

5 DISCUSSION AND LIMITATIONS

  • 261/1650 versus 256/1650 on MT-AIME and 79.11% versus 77.79% on MMLU show EDLM-4B and EDLM-8B exceeding their teacher on those benchmarks.The base already exceeds the teacher on AIME25, and the authors do not isolate why students can exceed teacher scores.Source: §5 · p. 9
  • ForkLeft’s scope is bounded by verified-answer, teacher-query, and shared-tokenizer requirements, and reported-score comparisons inherit decoding and training differences.A correct teacher answer does not guarantee correct distributions on every student prefix; ablations isolate rollout order and supervision view, while other components are evaluated only within the complete method.Source: §5 · p. 9

6 CONCLUSION

ForkLeft exposes student forks during training, aligns teacher supervision to shared left prefixes, and returns to native parallel decoding. The accompanying definitions specify its readouts, supervision routes, and update semantics.

6 CONCLUSION

  • ForkLeft improves Efficient-DLM-4B on all ten benchmarks, wins every order–view comparison, adds to SFT on the same questions, and exceeds matched-scale diffusion models on seven benchmarks.The conclusion also reports greater gains from stronger teachers and generalization to a second student family.Source: §6 · pp. 9–10

A.1 NATIVE AND PREFIX READOUTS

  • Native denoising can read student commitments on either side of a target, whereas prefix supervision hides the target and suffix and requires an output-index adapter.Matching textual prefixes does not guarantee matching sequence geometry between a truncated prefix and a fixed-mask canvas.Source: §A.1 · p. 15

A.2 ROLLOUT SCHEDULE AND SUPERVISION ROUTES

  • Entropy priority is recomputed each pass, with selected argmax tokens committed together; correctness of the student and teacher responses routes examples to CE, LEFTOPD, GOLD, or EXCLUDE.CE and LEFTOPD use student prefixes, GOLD uses reference prefixes when available, and EXCLUDE removes both losses; successful teacher answers do not verify every teacher prediction on an erroneous student prefix.Source: §A.2 · p. 15

A.3 ELIGIBLE POSITIONS, EOS, AND NORMALIZATION

  • Selected response positions include eligible tokens and realized EOS, while truncated responses receive no fabricated termination target and padding is never a target.Forced EOS inclusion gives termination explicit weight; the final objective divides by the original question-batch size, leaving excluded questions at zero weight without rescaling retained ones.Source: §A.3 · p. 16

A.4 RANKING READOUTS, GEOMETRY, AND GRADIENTS

  • The ranking score contrasts entropy on a masked canvas with entropy on the completed student prefix, and qualifying pairs prefer lower prefix confidence for greater positive entropy contrast.Scores, selected positions, and pair membership are detached; ranking uses student prefixes even on GOLD, while native-state behavior depends on shared parameters.Source: §A.4 · p. 16

A.5 THE SAMPLING DISTRIBUTION AND THE UPDATE GRADIENT

  • The update samples completed responses under the rollout order and selected positions under the uniform-plus-EOS rule, without conditioning its expectation on retained questions.Rollouts, verdicts, selected indices, and targets are fixed during optimization; gradients pass through live student readouts, not generated tokens, commit order, or routing.Source: §A.5 · pp. 16–17

A.6 HARD AND DISTRIBUTIONAL TARGETS

  • Hard labels and teacher distributions produce different target vectors, and their mixture interpretation applies to one logit vector only when both losses share a prefix.The shared-subset experiment is a matched-example analysis, not a token-normalized efficiency comparison.Source: §A.6 · p. 17

A.7 NATIVE INFERENCE AND THE SCOPE OF THE RATIONALE

  • Inference uses the student’s native masked states and confidence score, without the teacher, verifier, or prefix probes; the fixed-prefix KL rationale does not prove that the full model discards useful right context.The isolated loss has its minimum when the student matches the teacher across supported right contexts; entropy-first ordering and confidence ranking are assessed empirically.Source: §A.7 · p. 17

B TRAINING ALGORITHM AND UPDATE SEMANTICS

Algorithm 1 collects student rollouts, routes each question to its applicable losses, and updates the student using the original batch size; inference uses native student sampling.

B TRAINING ALGORITHM AND UPDATE SEMANTICS

  • For each question, Algorithm 1 generates a student continuation, obtains or reuses the cached teacher response and verdict, and selects a supervision branch.EXCLUDE questions are skipped; retained questions use selected student positions for CE or LEFTOPD, and available reference continuations for GOLD content loss.Source: B TRAINING ALGORITHM AND UPDATE… · p. 18
  • The algorithm computes detached entropy-contrast scores and qualifying pairs, adds the live prefix-confidence ranking loss to content loss, and divides the update by the original batch size.If no questions are retained, it performs no update.Source: B TRAINING ALGORITHM AND UPDATE… · p. 18
  • At inference, the student samples natively with its most confident positions first, without a teacher, verifier, or prefix-sensitivity probe.Source: B TRAINING ALGORITHM AND UPDATE… · p. 18

C.1 TRAINING AND EVALUATION PROTOCOL

The protocol specifies training and evaluation settings, run provenance, and conventions for reporting benchmark counts and averages.

C.1 TRAINING AND EVALUATION PROTOCOL

  • The main EDLM-4B run uses 2000 updates, while the SDAR extension uses 500; generated-answer benchmarks use native student samplers and AR references use greedy decoding.Training uses math-domain questions, 1024-token response budgets, 32-token blocks, and at most 32 passes per block; generation budgets are 2048 tokens or 4096 for MT-AIME.Source: §C.1 · p. 18

C.2 RUN PROVENANCE AND REPRODUCIBILITY

  • The primary, transfer, and checkpoint-curve results correspond to specified completed runs, and the order–view ablation shares initialization, teacher, questions, and evaluation pipeline.The shared-subset experiment fixes selected questions and changes only the supervision objective.Source: §C.2 · p. 18

D COMPLETE NUMERICAL RESULTS

  • Benchmark entries are correct-answer counts with denominator n, accuracy is 100c/n, and cross-task averages weight the ten benchmark accuracies equally.Reported changes use counts before rounding, and pooled totals are not treated as accuracy estimates on independent samples.Source: D COMPLETE NUMERICAL RESULTS · p. 19

D.1 TRAINING PROGRESS

The appendices report the training-progress and configuration comparisons, then extend evaluation to a second student family. They also distinguish reference scores from matched post-training controls.

D.1 TRAINING PROGRESS

D.2 ROLLOUT ORDER, TEACHING VIEW, AND SUPERVISION OBJECTIVES

  • Table 4 reports four rollout-order and teaching-view configurations alongside three objectives evaluated on the same teacher-correct, student-wrong subset.The two blocks answer different questions and should not be pooled.Source: §D.2 · p. 19, Table 4 · p. 19

D.3 TEACHER AND AUTOREGRESSIVE REFERENCE MODELS

  • The AR checkpoint scores in Table 5 are contextual references, not matched post-training controls for the diffusion student; missing cells mean unreported measurements.Source: §D.3 · p. 19, Table 5 · p. 20

D.4 TEACHER CONFIGURATIONS FOR BOTH STUDENT SIZES

  • Qwen2.5-7B adds a cross-generation teacher comparison, with a 3.17 pp mean gain over EDLM-4B; its teacher family changes alongside size, so it is kept off the Qwen3 scaling line.The Qwen3 teacher-scaling comparison is reported separately.Source: §D.4 · p. 20, Table 6 · p. 20

D.5 GENERALIZATION TO A SECOND STUDENT FAMILY

  • After 500 updates, the SDAR-4B comparison reports seven positive changes, two declines, and one unchanged task, including 991 additional MMLU answers and 38 additional CEval answers.The count changes are reported without assigning them to noise.Source: §D.5 · p. 20, Table 7 · p. 21

E COMPARISON WITH PUBLISHED DIFFUSION MODELS

The comparison finds that ForkLeft students lead the published diffusion systems on most benchmarks at matched parameter counts. The comparison is across published systems, not a controlled test of training objectives.

E COMPARISON WITH PUBLISHED DIFFUSION MODELS

  • The published-system comparison is not objective-controlled: SDAR-Chat uses instruction tuning, OPDLM uses on-policy distillation, and ForkLeft trains on math-domain questions for 2,000 updates.The systems also differ in decoding: published scores use greedy static decoding with block size 4, while ForkLeft students use native samplers.Source: E COMPARISON WITH PUBLISHED DIFFUSION… · p. 21
  • ForkLeft leads on seven benchmarks, while SDAR-Chat and OPDLM retain complementary strengths on GSM8K, GPQA-D, and LMB-H.Margins over the better published model on ForkLeft-leading tasks range from 10.5 and 10.1 pp on MT-AIME at 4B and 8B to 1.2 and 0.5 pp on MMLU.Source: E COMPARISON WITH PUBLISHED DIFFUSION… · p. 21, Table 8 · p. 22
  • Each distilled student matches or exceeds each published system on at least eight of ten benchmarks.Table 8 reports the ten-benchmark comparison; the published SDAR-Chat and OPDLM scores are from Su et al. (2026).Source: E COMPARISON WITH PUBLISHED DIFFUSION… · p. 21, Table 8 · p. 22

F.1 USEFUL RIGHT CONTEXT IS COMPATIBLE WITH PREFIX TEACHING

The appendix examines context views and correctness routing, then qualifies how benchmark counts should be interpreted. Its related-work discussion situates ForkLeft among approaches that address other aspects of diffusion learning but leave teacher–student context alignment open.

F.1 USEFUL RIGHT CONTEXT IS COMPATIBLE WITH PREFIX TEACHING

  • Among 19,115 prediction disagreements, the bidirectional view alone matches the reference token at 15,998 positions, versus 519 for the left view alone; these are static predictions, not observed decoding transitions or answer errors.The left-only-correct cases are 519/120,453, approximately 0.43% of all probed positions.Source: §F.1 · p. 22
  • Mean entropy is 0.6134 nats in the left view and 0.0438 in the bidirectional view, but entropy alone does not determine the gradient, and uniform loss sampling covers confident errors independently of rollout entropy.Source: §F.1 · p. 22

F.2 CORRECTNESS ROUTING EXPLOITS RESPONSE-LEVEL DIFFERENCES

  • Teacher and student answer correctness differs question by question: the teacher's net advantage is 101 GSM8K answers and 36 MATH500 answers, smaller than each teacher-only-correct pool.The evaluated response unions are 1297/1319 on GSM8K and 474/500 on MATH500; these are verifier coverage, not distilled scores, and benchmark inference receives no training labels.Source: §F.2 · p. 22

F.3 RESOLUTION, DEPENDENCE, AND UNCERTAINTY

  • MT-AIME's 1,650 instances are 55 translations of the same 30 questions, and marginal before/after totals cannot supply paired-discordance counts for significance testing.One item shifts accuracy by 3.33 pp on a 30-item AIME set, 2.22 pp on LMB-H, and approximately 0.51 pp on GPQA-D.Source: §F.3 · p. 22

G ADDITIONAL RELATED WORK

  • Prior diffusion methods refine reward optimization, trajectory credit, self-distillation, causal structure, or block curricula; those approaches do not by themselves resolve which context bidirectional students and causal teachers should share.ForkLeft explicitly targets that context-alignment choice while retaining student decisions and restoring native confidence-first parallel decoding at inference.Source: G ADDITIONAL RELATED WORK · p. 23

Related papers

Loading 2609.32448v1…