A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

1 Introduction

• Masked dLLMs predict multiple tokens from shared visible context, but same-step predictions can conflict and later steps revise committed tokens.The central question is whether they can generate a coherent block in one pass without a continuous latent or later token revision.Source: §1 · p. 1
• CONDOR adds a Gaussian noise field to mask embeddings so each ticket can select a coherent block, with different tickets learning different continuations.Winner-take-all supervision selects the lowest-loss prediction; self-distillation trains the one-pass output to match its refined block.Source: §1 · p. 2, Figure 1 · p. 2
• The paper introduces a from-scratch dLLM capable of generating a complete coherent token block per forward pass.Its stated contribution also includes successive blocks that remain readable and one-pass multimodal generation for text-to-image and unconditional text-and-image tasks.Source: §1 · p. 2

2 Related Work

• Autoregressive efficiency methods draft or fill multiple tokens, but conflicts between guesses require self-verification and multiple steps (29), (6), (26, 44).Speculative decoding, Medusa, Jacobi decoding, and Consistency LLMs are examples of these generate-then-verify procedures.Source: §2 · p. 3
• Continuous-latent dLLMs denoise continuous representations built before or jointly with diffusion training and decodable to tokens (12, 14, 19, 30, 33, 55, 57); masked models fill multiple discrete-token positions per step (1–3, 7, 32, 36, 38, 41, 46, 53).Source: §2 · p. 3
• One-step coherence requires an input to select a coherent output: VAE-style pairs can fail through latent ignoring or posterior collapse (25), while flow and distillation methods pair noise with outputs (31), (10), (48), (54), (59).IWAE retains multiple predictions of one label, whereas WTA supervises only the lowest-loss input so inputs can specialize (5), (16, 27, 35, 40).Source: §2 · p. 3

3 Method

• The block-wise Transformer conditions each predicted block on a prefix, with bidirectional attention within a block and attention across blocks limited to the current and earlier blocks.The block input combines ground-truth tokens with a mask-position set; masked positions may include the whole block, and their mask embeddings are noised.Source: §3 · p. 4

3.1 Winner-take-all (WTA) on a partial mask

• With a partial mask, CONDOR scores k noise-conditioned predictions against masked ground truth and sends a gradient only through the lowest-loss draw.The other draws remain free to represent different continuations.Source: §3 · p. 4, §3.1 · p. 4

3.2 Self-distillation on a full mask

• With a full mask, the student’s greedy one-pass block is partly retained and the remaining positions are remasked and iteratively refilled to form a label for that same noise.The one-pass prediction is trained toward its noise-matched refined label; teacher filling commits the most confident remaining tokens at each step.Source: §3.2 · p. 5

3.3 Several blocks, and images

• At inference, each block is generated in one pass, appended to the prefix, and followed by fresh noise for the next block.Each new block conditions on the given prefix and previously generated blocks, retaining one forward pass per block.Source: §3.3 · p. 5
• The framework supports text-to-image and unconditional text-and-image generation, representing image patches as tokens and predicting continuous pixel values within each patch.Image prediction uses mean squared error instead of cross-entropy; patch remasking and teacher-fill selection are random because continuous predictions lack confidence scores for ranking patches.Source: §3.3 · p. 6

4.1 Experimental setup

• Scratch-trained CONDOR is evaluated on TinyStories block continuation and MNIST text-to-image and unconditional generation; MNIST images use sixteen 7×7 patches.Text continuations use eight-token blocks.Source: §4.1 · p. 6

4.2 Single-block text continuation

• 93.9% Validity was the best tested one-pass result, using current-student teaching, ρ = 0.75, shared-token noise, and Winner distilled.These are the best tested settings identified across the ablations.Source: §4.2 · p. 7, Table 2 · p. 8
• 60.4% one-pass Validity was reported for 30k-update CONDOR, whose Dvalid16 of 7.38 exceeded Plain CE, IWAE, IWAE with weight, and one-pass WTA.The 30k-update run aligns training time with the no-self-distillation runs.Source: §4.2 · p. 7, Table 2 · p. 8
• Larger inference-noise angles increase the fraction of outputs differing from the reference while gen-PPL stays stable; excessive noise norm reduces Validity.Both noise direction and magnitude affect the continuation.Source: §4.2 · pp. 7–8, Table 3 · p. 9

4.3 Block-wise text generation

• Across eight 8-token blocks, Free-continuation Validity declines while GT-prefix Validity is largely retained; unsatisfactory blocks can be regenerated with fresh noise.Table 4 reports Validity for the current block, not the entire continuation.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 Multimodal generation

• MNIST demonstrations use one pass per block for digit-word-conditioned image generation and unconditional digit-word generation followed by a conditioned image.These are qualitative multimodal examples.Source: §4.4 · p. 9, Figure 3 · p. 10

• CONDOR is trained from scratch with winner-take-all supervision and same-noise self-distillation.Source: §5 · p. 10
• The model learns to map different noise samples to different continuations, predicting a complete block in one forward pass.Source: §5 · p. 10
• Small-scale experiments demonstrate single-block and block-wise text generation, text-to-image, and unconditional text-and-image generation.Source: §5 · p. 10

• The experiments study TinyStories continuation and a qualitative MNIST demonstration.Source: §6 · p. 10
• Larger models and more complex, diverse datasets are identified as natural directions for follow-up work.Source: §6 · p. 10
• Reliable long-form generation and broader generalization are also left for follow-up work.Source: §6 · p. 10

• The single-block model uses pre-norm layers, learned positional embeddings, tied token embeddings, and an EMA teacher initialized from the student with decay 0.9999.Source: §A.1 · p. 16
• Table S1 gives optimizer details for CONDOR, and its image column applies to both image tasks.Source: Table S1 · p. 16
• Training schedules specify warm-up and cosine-decay learning rates, with distinct distillation-weight ramps for single-block, multi-block, and image runs.For 30k-update single-block runs, the distillation ramp is shortened to 16,000 updates; multi-block full-mask and image losses are normalized by k.Source: §A.1 · p. 16
• Default single-block CONDOR training takes 14.1 GPU-hours on one A100; k = 4 WTA takes 5.3 hours and DiMO about 2 hours per configuration.Source: §A.1 · p. 16

A.2 Baseline training

• Plain CE, IWAE, and WTA use the same base learning rate and weight decay as the single-block basic setting, with uniform mask sampling unless specified otherwise.The schedule uses 2,000 warm-up updates followed by cosine decay; weighted IWAE uses k = 4 and the objective in Eq. (6).Source: §A.2 · p. 16
• DiMO adaptation uses the 150k-update Plain CE checkpoint as a frozen teacher, with student, teacher, and fake models initialized from it.Four adaptations cross fully masked or hybrid inputs with learning rates 10^-5 or 10^-4; Table 2 reports the hybrid-input, 10^-5 setting.Source: §A.2 · p. 16
• DiMO adaptations use AdamW with zero weight decay, temperature 1, and one fake-model update paired with each student update.Each student update uses label smoothing 0.1 for the fake model.Source: §A.2 · p. 17

A.3 Ablation variants

• The independent-target-noise ablation builds its distillation target under fresh ε′ while training the student under the original ε.Source: §A.3 · p. 17

B.1 Frozen-rubric text-validity re-review

• Natural truncation passes, but internal grammar errors, merged words, token loops, control tokens, and punctuation degeneration fail.A close-quote can pass when the visible text already reads as the end of speech; extra or incompatible quote nesting fails.Source: §B.1 · p. 17
• The rubric judges each continuation together with its visible prefix, and boundary items such as a name switch are concrete ambiguities.The reported mixed policy uses each item's recorded decision.Source: §B.1 · p. 17
• A language-model judge had seen earlier rubric discussion, so the review was not an independent human assessment; labels were locked before aggregation.Source: §B.1 · p. 17

C.1 Noise geometry

• A radius-αr0 input uses a unit direction in the D = 2048 completion field, with δ norm equal to αr0.For the default seed-0 student at 150k updates, RMS(e[M]) = 0.720720 and r0 = 16.308026.Source: §C.1 · p. 17
• An angle-θ input has the same norm and angle θ to a reference direction, with its tangent uniform on the orthogonal complement.The 0° and 180° cases each correspond to a single direction.Source: §C.1 · p. 17
• Table S2 presents unedited continuations of “They would swim, splash, and pretend” at norm r0, with one tangent in Panel A and other directions at 30° in Panel B.Source: Table S2 · p. 17