Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models

• Clock Diffusion uses position-dependent noise schedules and efficient training and sampling to support semi-autoregressive continuous DLMs with variable-length generation and KV caching.Source: §1 · p. 2
• ClockDLMs combine Gaussian diffusion over token embeddings with sliding-window Clock Diffusion.Source: §1 · p. 2
• Cache Grab samplers adapt confidence-threshold token commitment and self-speculative decoding from discrete diffusion for sliding-window models.Source: §1 · p. 2
• ClockDLMs set continuous-DLM PPL state of the art on OWT and beat discrete block DLMs; Cache Grab models attain the highest GSM8K accuracy among comparably sized TinyGSM-trained models.Source: §1 · p. 2

• Forward diffusion progressively corrupts data through time-indexed latent variables toward a tractable prior, while learned reverse distributions denoise from noisy inputs using a neural prediction of the clean signal.Source: §2 · p. 2
• For sequences, diffusion is applied independently along the sequence dimension; continuous text models embed one-hot tokens into vectors before applying the process.Source: §2 · pp. 2–3
• The signal and noise schedules vary monotonically over time, and reverse models can predict clean token embeddings through vocabulary logits.Source: §2 · p. 3
• Continuous language models differ by using one-hot, token, or contextual embeddings.Source: §2 · p. 3
• Semi-autoregressive discrete diffusion decomposes sequences into sequential blocks and diffuses each block, tightening likelihood bounds, improving sample quality, and enabling Transformer KV-cache sampling.Source: §2 · p. 3

3.1 PARAMETERIZATION

• Per-position local clocks reparameterize global time with phase offsets, creating equal-duration denoising windows whose overlap or separation determines how positions generate.Positions with a larger phase finish denoising earlier.Source: §3.1 · p. 4

3.2 KEY-VALUE CACHE SAMPLING

• Completed positions are written to the KV cache once and reused, while the sampler advances all currently awake positions in shared model calls.With per-position budget N, the sampler uses T = rN model calls.Source: §3.2 · pp. 4–5

3.3 EFFICIENT TRAINING

• Training vectorizes reverse steps in sequences of length 2L, grouping noisy positions into awake sets with masks that control clean and noisy attention.Each noisy group attends to clean positions strictly preceding that group.Source: §3.3 · p. 5

3.4 SPECIAL CASES OF CLOCK DIFFUSION

• Block generation completes and caches whole blocks in lockstep, whereas sliding-window generation advances an active window of at most S tokens and can cache a completed prefix.Sliding-window training uses causal clean-token attention, allowing a leftmost prefix to be cached without waiting for the rest of the window.Source: §3.4 · pp. 5–6, Figure 2 · p. 5

3.5 CLOCK DIFFUSION LANGUAGE MODELS

• ClockDLMs combine sliding-window Clock Diffusion with Gaussian noise on learnable token embeddings and a continuous-time variational objective.The authors also train a block version, B-ClockDLMs, as an ablation.Source: §3.5 · p. 6

3.6 CACHE GRAB SAMPLERS: ACCELERATED INFERENCE FOR CLOCKDLMS

• Cache Grab samplers accelerate ClockDLM inference by committing additional positions to the cache and skipping their remaining denoising evaluations.Confidence sampling commits eligible contiguous positions from the left; SSD uses an autoregressive verifier and retains uncommitted positions' noisy states and progress.Source: §3.6 · p. 6

4 EXPERIMENTS

• ClockDLMs outperform previous continuous models while matching and exceeding the best SAR discrete models on standard diffusion benchmarks.The experiments report state-of-the-art diffusion likelihood on standard benchmarks.Source: §4 · p. 7

4.1 LIKELIHOOD EVALUATION

• ≈38% reduction in the PPL gap to the AR baseline versus BD3LM with S = 4; both ClockDLM and B-ClockDLM outperform comparable SAR discrete diffusion.The likelihood results report state-of-the-art performance among diffusion models.Source: §4.1 · p. 7

4.2 MATHEMATICAL REASONING

• Up to ∼33% increase in accuracy over previous continuous DLMs; ClockDLM matches SAR discrete diffusion, while Cache Grab-Conf. preserves accuracy with nearly 3× speedups.Cache Grab-SSD exceeds AR accuracy and has the highest throughput among Table 2 configurations.Source: §4.2 · pp. 7–8
• ClockDLMs push the accuracy–efficiency frontier beyond full-sequence continuous diffusion and match or exceed SAR discrete methods.Figure 4 compares zero-shot GSM8K pass@1 with average NFEs/seq for active window sizes S = 32 and S = 64.Source: Figure 4 · p. 8, §4.2 · p. 8
• Cache Grab-Conf. uses ∼3–4× fewer NFEs/seq with minor to no accuracy drop-off; Cache Grab-SSD improves performance with up to ∼5× fewer NFEs/seq.The reported pattern holds across active window sizes S ∈ {16, 32, 64}.Source: Figure 5 · p. 8, §4.2 · p. 8

4.3 SUMMARIZATION

• ClockDLM (S = 32) improves all three ROUGE scores over RePlaid (T = 1024) with ∼8.5× fewer NFEs.Table 3 reports ROUGE-1/2/L and NFE means ± sample standard deviation for models trained from scratch.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 ABLATIONS

• Sliding-window ClockDLMs often outperform block B-ClockDLMs, possibly because their within-window decoding is more left-to-right.Source: §4.4 · pp. 8–9
• Smaller windows generally improve performance as generation becomes more AR-like; lower confidence thresholds increase speed gains but usually degrade quality.The confidence-threshold quality trend is less pronounced in Table 7 than in Figure 6.Source: §4.4 · p. 9

Related work

• Earlier SAR decoding with diffusion over logits lagged discrete diffusion in quality and did not propose efficient training or KV-cache sampling.Source: §5 · p. 9

Position-dependent Noise Schedules

• Figure 6 plots ROUGE-1 against NFEs/seq for ClockDLMs while sweeping the early-commit confidence threshold; hollow markers correspond to no early commit.Source: Figure 6 · p. 9
• Rolling Diffusion formalized a process akin to sliding-window Clock Diffusion, while this work adds vectorized training, KV-cache sampling, and a language-specific variational bound.The cited prior work examined continuous data such as video frames.Source: §5 · p. 9

Discussion & Conclusion

• ClockDLMs achieve state-of-the-art language modeling performance on perplexity and downstream benchmarks, consistently outperforming full-sequence continuous models in performance and efficiency.The framework supports block and sliding-window semi-autoregressive generation and includes accelerated samplers adapted from discrete diffusion.Source: §6 · p. 9

Limitations

• Improved language-modeling performance comes at the cost of decoding flexibility, which may matter in domains without a strict left-to-right bias; experiments so far use relatively smaller-scale models.Source: §6 · p. 9

Future Directions

• Future work plans to test larger scales, explore any-order sliding-window implementations and pretrained AR or discrete-diffusion models, and investigate hybrid discrete–continuous parameterizations.Source: §6 · p. 9

AI USE STATEMENT

• LLMs assisted with derivations, code, tables and figures, literature summaries, formatting, proofs, hypotheses, methodology, implementation, and result interpretation, but not synthetic data, translation, or dataset cleaning.The authors take responsibility for the final content, including AI-assisted text, claims, and artifacts.Source: AI USE STATEMENT · p. 10

A CLOCK DIFFUSION VARIATIONAL BOUND

• The Clock Diffusion variational bound decomposes the negative-likelihood objective into prior, reconstruction, and finite-time diffusion losses.The derivation applies importance weighting and Jensen’s inequality; awake transitions factor over positions conditioned on awake latents and previously revealed tokens.Source: A CLOCK DIFFUSION VARIATIONAL BOUND · p. 17

B.1 CLOCK DIFFUSION WITH GAUSSIAN NOISE

• Clock Diffusion makes Gaussian noise schedules position-dependent, with signal and noise levels governed by α and σ schedules.The base schedules are described as monotonically decreasing and increasing, respectively.Source: §B.1 · p. 18

B.2 LEARNED EMBEDDING PARAMETERIZATIONS

• The learned-embedding parameterization maps one-hot tokens to embeddings, normalizes embedding rows to unit L2 norm, and optimizes both the network and embedding matrix.Source: §B.2 · pp. 18–19

B.3 VARIATIONAL CLOCK DIFFUSION: CONTINUOUS-TIME LIMIT

• In the continuous-time limit, reconstruction loss stabilizes under grid refinement when the finite grid includes every position’s completion time.The prior loss is unaffected by the limit; the diffusion-loss derivation assumes absolutely continuous local clocks and a continuously differentiable base SNR schedule.Source: §B.3 · p. 19

B.4 INHERITED REPLAID COMPONENTS

• ClockDLMs inherit RePlaid techniques including learned monotone noise schedules, self-conditioning, batch-split loss estimation, and an output-logit prior.The schedule can be parameterized by its slope, which the authors found more stable on conditional tasks; the output-logit prior adds the analytic Gaussian log-likelihood of each vocabulary embedding to the denoiser logits.Source: §B.4 · pp. 20–21

B.5 BLOCK CLOCK DIFFUSION LANGUAGE MODELS

• The block formulation partitions sequences into non-overlapping blocks, keeps one block awake at a time, and adds each completed block to the KV cache in lockstep.Sampling uses T = BN model calls.Source: §B.5 · p. 21
• Clock Diffusion recovers BD3LM by replacing masked diffusion with Gaussian corruption while keeping the same priors, reverse kernels, and reconstruction decoders.Source: §B.5 · p. 22

B.6 SLIDING WINDOW CLOCK DIFFUSION LANGUAGE MODELS

• The sliding-window prior loss is unchanged from Block Clock Diffusion because every position has uℓ(1) = 1 and the forward endpoint distribution matches the standard Gaussian prior.Source: §B.6 · p. 22
• Each sliding-window reconstruction factor decodes one token conditioned on its clean prefix and an incoming window containing up to S −1 partially noisy subsequent positions.Unlike block reconstruction, which jointly decodes a block whose positions have reached local time zero, the sliding-window factor uses staggered noise levels.Source: §B.6 · p. 23
• Sliding-window diffusion loss evaluates prediction error along each position’s denoising trajectory, using its active window and clean context.Each active interval maps to local time u ∈[0, 1], canceling the clock derivative against the integration measure.Source: §B.6 · pp. 22–23

C.1 BLOCK TRAINING

• Block training draws one diffusion time per block and broadcasts it across that block, using block-causal clean attention and bidirectional noisy attention within the block.The clean endpoint γ0 conditions the clean half of the canvas.Source: §C.1 · p. 23

C ADDITIONAL TRAINING & EVALUATION DETAILS FOR CLOCK DIFFUSION LANGUAGE MODELS

• Sliding-window training assigns unique diffusion times to positions rather than sharing one time within each active window.The training algorithms use different attention masks for block and sliding-window cases.Source: §C.1 · p. 24

C.2 SLIDING WINDOW TRAINING & EVALUATION

• The sliding-window mask makes clean-to-clean attention token-causal while noisy positions attend within their S-wide window and to clean keys before it.Source: §C.2 · p. 24
• Sliding-window training sorts independently drawn diffusion times within each S-token group, preserving left-to-right local-time order while allowing variable spacings for Cache Grab strides.Source: §C.2 · p. 24
• The clean-endpoint reconstruction proxy uses a strictly cleaner context than the sampler and does not by itself yield a valid bound on L∞,SW.It evaluates token-level cross-entropy with all positions at the clean endpoint.Source: §C.2 · p. 24
• Validation estimates reconstruction with shifted S-wide groups, scoring only group heads and multiplying their contribution by S to recover the L-term sum in expectation.Uniformly sampled offsets make every position a group head with probability 1/S.Source: §C.2 · p. 25
• Validation evaluates diffusion on a separately corrupted input, with random window times yielding each token a uniform local time on [0, 1].Self-conditioning uses two passes per loss term, so reconstruction and diffusion require four model passes per batch with one Monte Carlo sample.Source: §C.2 · p. 25

D ADDITIONAL CLOCK DIFFUSION SAMPLING DETAILS

• Both block and sliding-window generation commit S tokens every N model calls on average, except at the sequence end.Block generation commits a whole block in lockstep; sliding-window generation commits one position every N/S calls.Source: D ADDITIONAL CLOCK DIFFUSION SAMPLING… · p. 25
• When N < S, sliding-window sampling groups S/N consecutive positions into cohorts, admits one cohort per call, and commits it N calls later.Sampling takes ⌈LN/S⌉+ N −1 model calls.Source: D ADDITIONAL CLOCK DIFFUSION SAMPLING… · pp. 25–26
• Fused caching concatenates newly decoded tokens with incoming noisy tokens for the next denoising pass, and the custom mask keeps committed tokens from attending to noisy ones.This makes their cache keys and values the same as those from a separate caching forward; Figure 7 depicts the block and sliding-window masks.Source: D ADDITIONAL CLOCK DIFFUSION SAMPLING… · p. 26, Figure 7 · p. 26

E.1 CACHE GRAB-CONFIDENCE

• Cache Grab-Confidence commits the longest leftmost run whose tokens reach confidence threshold η or whose clocks expire.Later eligible positions wait if an earlier position is neither confident nor finished; η = 1 recovers standard decoding.Source: §E.1 · p. 27

E.2 CACHE GRAB-SSD

• Cache Grab-SSD makes a diffusion draft call and an AR verification call, then commits matching drafts plus the AR correction at the first disagreement.It retains uncommitted positions and their denoising progress; a clock running out alone never commits a position, preserving the sequence produced by AR decoding.Source: §E.2 · pp. 27–28
• For SSD’s active window, newly admitted positions start at the prior while retained positions advance, with surviving new positions rescheduled to preserve clean-to-noisy order.The window is refilled to S positions each iteration.Source: §E.2 · p. 28

F ADDITIONAL EXPERIMENTAL SETUP DETAILS

• The evaluation protocol counts all model forwards in NFE metrics, including prompt ingestion, cache updates, final readout, and SSD verification calls.The reported evaluations use one node with eight H100-80GB GPUs and seed 1.Source: F ADDITIONAL EXPERIMENTAL SETUP DETAILS · p. 28

F.1 OPENWEBTEXT

• OpenWebText likelihood results use variational perplexity upper bounds on the full held-out validation split, with a rolling estimator for sliding-window models.RePlaid and block ClockDLMs use the ordinary validation bound; all evaluations use self-conditioning and EMA weights.Source: §F.1 · p. 29

F.2 MATH REASONING BENCHMARK

• TinyGSM examples pair grade-school math questions with Python programs, and GSM8K evaluation scores zero-shot pass@1 by executing generated programs against reference answers.The test set contains 1,319 examples.Source: §F.2 · pp. 29–30

F.3 SUMMARIZATION

• CNN/DailyMail evaluation generates summaries for 11,490 test articles and reports example-averaged ROUGE-1/2/L without stemming.Generation is capped at 180 tokens, with a minimum of 50 non-[EOS] tokens.Source: §F.3 · p. 30

G.1 OPEN-ENDED GENERATION

• ClockDLM outperforms RePlaid at higher NFE budgets (about 512 or more), but struggles in low-NFE open-ended generation.At a total budget of about 256, ClockDLM applies about 16 denoising steps per token, versus 256 for RePlaid.Source: §G.1 · p. 31, Figure 8 · p. 32

G.2 CACHE GRAB-CONFIDENCE FOR CLOCKDLMS & B-CLOCKDLMS ON GSM8K

• 3.2× fewer NFEs/seq at S = 32 with no accuracy loss, and 2.8× fewer at S = 64 at a cost of 2.0 points, when ClockDLM’s threshold falls to η = 0.7.B-ClockDLMs show smaller reductions: 2.7× at S = 32 and 2.1× at S = 64, with accuracy within one point of standard decoding.Source: §G.2 · p. 32, Table 7 · p. 33

G.3 EFFECT OF SSD TRAINING

• 0.9 to 5.7 points: joint AR and diffusion training improves GSM8K accuracy at every window size, with NFEs/seq essentially unchanged.At S = 4 and 8, jointly trained models reach 66.6% and 67.2%, above the 64.3% AR baseline.Source: §G.3 · p. 33, Table 8 · p. 33

G.4 GENERATED SAMPLE EXAMPLES

• The displayed GSM8K ClockDLM examples correctly compute (16−3−4)×2 = 18 and 2+0.5×2 = 3.The examples use S = N = 32, DDPM with top-1 clean-token predictions, and token-embedding self-conditioning.Source: Figure 11 · p. 37, Figure 12 · p. 37, §G.4 · p. 37
• The appendix shows paired CNN/DailyMail summaries from two models on the same articles, without corrections.The comparisons use the same article inputs for both models.Source: Figure 13 · p. 38, Figure 14 · p. 38, §G.4 · p. 38

H ASSETS

• The asset tables list datasets, tokenizers, architectures, external code, and software with corresponding licenses; OpenWebText packaging is CC0 while source articles retain original rights.The appendix also lists generated OpenWebText samples and sample outputs as part of its examples.Source: Table 9 · p. 39, Table 10 · p. 39, Table 11 · p. 39