DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models

1. Introduction

• The conversion changes both axes in three stages: attention-to-recurrence distillation, bidirectional denoiser expansion, and masked-diffusion continual pretraining.Staging lets the authors trace where a capability is retained or lost.Source: §1 · p. 1
• On the multi-query recall probe, converted students score 0.000 at both scales versus teachers’ 0.335–0.575; diffusion pretraining alone does not restore retrieval.The curriculum-port result is stochastic under a fixed schedule, motivating the paper’s closed-loop approach.Source: §1 · p. 1
• A closed-loop curriculum raises retrieval success from 1/3 to 3/3 on the same three seeds and transfers unchanged to 8B, where two of three seeds succeed.It advances the gap only while measured retrieval accuracy remains above a threshold.Source: §1 · pp. 1–2
• Language modeling transfers only partially and in-distribution, while in-context retrieval does not transfer at 1.7B or 8B.The introduction characterizes retrieval as absent, not merely degraded.Source: §1 · p. 2
• At 8B, retrieval onset varies across seeds (6.5k, 11k, and >16k steps), consistent with a fixed budget cutting off a late run.The third seed had not learned within its fixed 16k-step budget.Source: §1 · p. 2
• Every model that learns retrieval scores 0.000 on held-out tokens; resampling keys and values each batch rules out memorized bindings as the explanation.The reported boundary is token coverage, and randomized-pool pacing policies also fail through distinct mechanisms.Source: §1 · p. 2
• Separately, a 7B code model is converted into a 3:1 recurrent-attention block-diffusion hybrid, with an auxiliary alignment loss bringing no benefit and a learning-rate ceiling appearing after 50k stable steps.The final checkpoint reaches held-out block-diffusion NLL 2.819 after two rollbacks and a learning-rate reduction.Source: §1 · p. 2

2. Related Work

• Gong et al. (2025) and Ye et al. (2025) adapt pretrained autoregressive Transformers to masked diffusion while retaining attention.These methods establish that the objective change is survivable and far cheaper than pretraining.Source: §2 · p. 2
• RADLADS converts attention to RWKV-7-style recurrence at multi-billion scale while retaining next-token prediction; stage 1 follows its two-phase recipe.RADLADS is attributed to Goldstein et al. (2025).Source: §2 · p. 2
• DiffuMamba shows scan backbones can serve as denoisers, while this paper uses gated delta-rule cells by default and RWKV-7 in one 8B arm.The recurrent cells use the rank-1 state update of Yang et al. (2024) and Peng et al. (2025).Source: §2 · p. 2
• Block diffusion interpolates between autoregressive and diffusion objectives, while prior recall work motivates testing whether recurrence’s associative-recall deficit is a learnability problem at scale.The related-work discussion cites Arriola et al. (2025), Arora et al. (2023; 2024), Boesch & Wee (2026), Okpekpe & Orvieto (2025), and Blouir et al. (2024).Source: §2 · p. 2

3.1. Stage 1: attention to recurrence

• At stage 1, each attention sublayer is replaced by a randomly initialized gated delta-rule cell while inherited parameters stay frozen; two-phase distillation matches attention outputs and then teacher next-token distributions.On the distillation text, top-1 agreement is 0.746 at 1.7B and 0.779 at 8B; 8B held-out text8 agreement is 0.544.Source: §3.1 · pp. 2–3

3.2. Stage 2: bidirectional expansion

• Stage 2 adds copied backward recurrent cells, merge matrices, and a mask-token row without training parameters, enabling bidirectional context for denoising.The forward cells remain frozen at their stage-1 values by default.Source: §3.2 · p. 3

3.3. Stage 3: diffusion continual pretraining, with a retrieval curriculum

• Stage 3 trains with masked discrete diffusion, annealing the mask ratio from 0.15 to 0.65; retrieval episodes place queries after a key–value table and a real-text haystack.Episodes use four queried pairs, while the haystack gap is sampled up to a cap that grows toward the sequence maximum.Source: §3.3 · p. 3
• The open-loop gap cap grows with training steps, whereas the closed-loop ratchet advances only when the retrieval-accuracy EMA meets its threshold; randomized-pool experiments resample keys and values each batch.The randomized-pool setting draws tokens from a shared pool to prevent memorizing particular bindings.Source: §3.3 · p. 3

3.4. Evaluation: the recall probe, its controls, and a teacher check

• The recall probe scores exact match on one masked answer across pair-count and gap settings; shuffled bindings, mismatch queries, and a teacher check test whether retrieval is genuine.The disjoint-band evaluation uses keys and values untouched by retrieval training or randomized pools to test generalization beyond trained tokens.Source: §3.4 · pp. 3–4

4.1. What transfers, and what does not

• Stage-1 top-1 agreement is 0.746 at 1.7B and 0.779 at 8B on distillation text, but 0.544 on held-out text at 8B; converted retrieval remains 0.000.Stage-3 diffusion pretraining without retrieval episodes also leaves retrieval at 0.000.Source: §4.1 · p. 4

4.2. The lottery, and the closed-loop fix

• All three same-seed runs lock in under closed-loop scheduling (per-seed means 0.97, 0.98, 0.97), unlike the open-loop lottery.On wikitext, the closed-loop recipe reaches 0.955–0.995 on every evaluated N × gap cell.Source: Table 2 · p. 4, §4.2 · pp. 4–5

4.3. Scaling to 8B

• Applied unchanged at 8B, the recipe lifts two locked-in seeds from 0.000 to grid means of 0.90 and 0.78; the third had not reached onset by 16k steps.The two onsets were at 6.5k and 11k steps; later onset left less training after the avalanche, and a proposed closed-loop budget remains untested.Source: §4.3 · p. 5, Figure 1 · p. 5

4.4. The token-coverage boundary

• Fresh per-batch keys and values still yield 0.000 on the disjoint band, despite gap-64 accuracy matching gap-0 and gap-160 accuracy remaining above chance.This separates token coverage from fixed-binding memorization and distance limits.Source: §4.4 · pp. 5–6
• Randomized-pool pacing fails by lockout at ratchet 0.6, learn-then-collapse at 0.25, or no onset for a 512-token pool within budget.The 8B pool-annealing run also failed to reach onset on its first tier within 24k steps, so annealing remains untested.Source: §4.4 · p. 6
• Because training-side interventions did not overcome token coverage, the authors retain sparse attention as a third rationale beyond state capacity and length extrapolation.The 7B hybrid retains every fourth attention layer, but its retrieval has not been measured.Source: §4.4 · p. 6

Conversion setup

• This separate conversion applies the stage-1 recipe to a Qwen2.5-Coder-7B-Instruct teacher, omits stage 2, and uses block diffusion with attention retained.Pretraining ran for 85,449 steps, followed by 4,000 response-masked supervised fine-tuning steps; the final checkpoint required two rollbacks and a learning-rate reduction.Source: §5 · p. 6

Outcome

• Held-out causal NLL is 2.49 for the student versus 1.07 for the teacher (≈4.1× perplexity ratio), and free-running generation collapses into repetition.The held-out block-diffusion NLL is 2.819, versus a final training-objective EMA of 2.786.Source: §5 · p. 6

Architecture

• The student replaces 21 of 28 attention layers with gated delta-rule cells and keeps every fourth attention layer, forming a 3:1 recurrent-attention hybrid.It has 2.91B trainable parameters; the retained attention layers are trainable, while only the inherited embeddings are trainable among the other inherited components.Source: §5 · p. 6

Two-stream training

• Within each 64-token block, the noised stream uses clean states from earlier blocks and its own noised context, while each block’s first position is fed clean.The clean state at each block’s start is exposed at no additional cost through the chunked recurrence kernel; outputs are shifted by one position to preserve next-token semantics.Source: §5 · p. 6

Negative result 1

• The representation-alignment auxiliary brought no autoregressive-retention benefit, despite adding 10–15% step time and roughly doubling logit scale.Cosine alignment was already approximately 94% saturated at initialization, so the term was disabled.Source: §5 · p. 7

Negative result 2

• At learning rate 3×10^-4, training diverged twice after tens of thousands of steps, while a per-step loss-spike monitor registered neither failure.After rollback to step 50,729 and reduction to 2×10^-4, training remained stable for about 35k steps.Source: §5 · p. 7

What conversion costs

• Teacher knowledge survives conversion only partially and in-distribution, while in-sequence binding is absent rather than degraded and diffusion continual pretraining does not restore it.The companion mechanism account attributes the deficit to a coverage hole that only training covering it can close (Boesch & Wee, 2026).Source: §6 · p. 7

Closed-loop curricula are the transferable lesson

• On identical seeds, a closed-loop curriculum yields 3/3 retrieval learners versus 1/3 with the open-loop schedule; the companion bench’s open-loop schedule reaches 0/6 at length 512 (Boesch & Wee, 2026).The proposed extension is to gate training budget on the competence signal, rather than use a fixed step count.Source: §6 · p. 7

What the coverage boundary means

• Retrieval trained over a narrow symbol pool scores 0.000 outside that pool, even after pool randomization, three gate policies, and a scale change.The paper interprets this as a circuit over embeddings involved in retrieval training, making token coverage a dimension of the training distribution.Source: §6 · p. 7

Two monitoring lessons from the block-diffusion conversion

• Both block-diffusion training negatives involved decisions made in advance despite later measurements that could have challenged them: alignment was saturated at initialization, and divergence escaped a per-step monitor.The divergence developed over hundreds of steps before the loss spike, while the representation-alignment term’s saturation was measurable from initialization.Source: §6 · p. 7

Token generality

• Every solved model scores 0.000 on held-out token bands, so the paper makes no claim that the curriculum installs general retrieval; its proposed annealing fix is untested.The limitation concerns token generality, not retrieval over the training tokens.Source: §7 · p. 7

Language-modeling transfer

• Stage 1 distills on a 12M-character slice of one corpus; at 8B, held-out agreement is 0.544 and off-domain perplexity retention is near zero.The authors therefore do not characterize the converted students as general-purpose language models.Source: §7 · p. 7

Seed counts

• Conversion-scale results generally rest on three seeds, with one Wikitext arm using one seed, and no pairwise contrast is significance-tested.The reported 3/3, 2/3, and 1/1 rates are small-n outcomes.Source: §7 · p. 7

Fixed budgets

• The 8B seed-2 miss, RWKV-7 no-onset arm, and pool-annealing arm remain consistent with late-onset budget truncation rather than method failure.Until the closed-loop budget is run, those three results remain open.Source: §7 · p. 7

Probe scope

• Retrieval is evaluated with one multi-query recall probe whose format is close to training, so the paper does not claim broad downstream retrieval.Evaluation uses held-out bindings and a held-out token band, but retains the training-like format.Source: §7 · p. 7

Measured language-modeling shortfall

• The model has a measured held-out NLL and an approximately 4.1× perplexity ratio against the teacher; when run freely, its generation collapses into repetition.Source: §6 · p. 8

Generation

• Free-running generation opens coherently, then collapses into repetition.Source: §6 · p. 8

Evaluation scope

• Systematic generation, retrieval-transfer, and instruction-following evaluations are deferred to a forthcoming results paper.Source: §6 · p. 8

• Per-arm training telemetry, including gate cap, retrieval EMA, and avalanche onset, is retained in object storage.Source: Reproducibility Statement · p. 8
• Evaluation outputs are retained for every arm, including failed arms, and checkpoints are retained for most arms but not the unseeded pilot.Source: Reproducibility Statement · p. 8
• The companion benchmark code and results are public, while the conversion code is not yet released.Source: Reproducibility Statement · p. 8

• Reducing the compute needed to re-architect pretrained language models may also lower the barrier to deploying them without scrutiny.Source: Impact Statement · p. 8
• A converted model can appear competent on its training distribution yet show no retrieval just outside it, a safety-relevant boundary.Source: Impact Statement · p. 8
• Perplexity-based evaluation does not surface the reported retrieval failure just outside the training distribution.Source: Impact Statement · p. 8