Source-linked AI summary
Subliminal Clocks: Latent Time Modelling in Diffusion Language Models
Maximo Eduardo Rulli, Thomas Vaitses Fontanari, Simone Petruzzi, Federico Alvetreti, Giorgio Strano, Donato Crisostomi, Giorgos Nikolaou, Tommaso Mencattini, Andrea Santilli, Emanuele Rodolà, Simone Scardapane, Alessio Devoto
TL;DR
DLMs lack explicit timestep conditioning, leaving it unclear whether they internally represent denoising progress and use it in generation. This paper probes and steers residual-stream representations, finding a decodable, structured signal whose manipulation predictably changes confidence, entropy, and KL divergence.
Problem
It remains unclear whether DLMs internally represent denoising progress during generation and whether that signal influences downstream computations.
Method
The paper trains layerwise probes to recover denoising progress and uses mean activation directions to steer the inferred signal in residual streams.
Results
The denoising-progress signal is probe-decodable across layers and token types, while steering it predictably changes confidence, entropy, and KL divergence.
Takeaways & Limitations
The recovered directions are functionally relevant to DLM computation rather than merely descriptive representations.
Takeaways & Limitations
The analysis is limited to LLaDA and Dream, leaving the denoising-progress phenomenon in other timestep-unconditioned DLMs unexplored.
Abstract
from arXiv · showhide
Diffusion Language Models (DLMs) have recently emerged as a promising alternative to autoregressive models. Unlike standard diffusion-based approaches, DLMs are not explicitly conditioned on a timestep, raising a natural question: do these models internally represent denoising progress, and how is such information used downstream? In this work, we show that DLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams. We find that this signal can be reliably extracted using probes across layers, indicating that denoising progress is decodable from internal activations. We further demonstrate that steering the model along a low-dimensional subspace associated with the inferred timestep allows us to systematically modulate its notion of denoising progress, leading to predictable changes in model confidence and entropy. Finally, we analyse the geometry of the identified representation, showing that it exhibits structured and interpretable properties in activation space, and shedding light on how such a signal is processed by these models.
1 Introduction
This section introduces masked Diffusion Language Models as an alternative text-generation paradigm and frames the paper around whether they internally represent denoising progress, whether that signal affects generation, and what properties it exhibits. The study answers these questions by probing, causally steering, and geometrically characterising the signal in LLaDA and Dream.
- Background: Masked Diffusion Language Models progressively denoise [MASK] tokens in a BERT-style manner rather than generating strictly left-to-right.The section identifies this masked-token family as DLMs and notes that prior work has focused mainly on efficiency and generation quality.
- Motivation: Prior studies examine attention sinks, [MASK]-token manipulation, and the effect of [MASK]-token counts, but the internal dynamics of denoising remain insufficiently understood.These studies respectively address attention-sink movement and pruning robustness, harmful or deceptive generations, and performance sensitivity to [MASK]-token counts.
- Research questions: The paper asks whether DLMs represent a denoising-step signal, whether modifying it affects downstream computations, what characteristics it exhibits, and whether it has a unified model-level representation.These questions are stated as RQ1, RQ2, RQ3, and a further question about unified model-level representation.
- Approach: The analysis proceeds by probing for denoising-time information, causally steering the recovered signal, and characterising its geometry across layers.The three stages correspond to probing, causal intervention, and geometric analysis of the identified signal.
- Models: Experiments analyse LLaDA-1.5, referred to as LLaDA, and Dream as two representative large-scale masked diffusion language models.The section names both models and specifies that the analysis is conducted on them.
2 Background
Diffusion language models learn to recover clean sequences from masked corruptions, with corruption controlled by a scalar noise level. At inference, they iteratively unmask tokens, and τ_t serves as an empirical proxy for denoising progress or time.
- Training formulation: DLMs are trained to recover a clean sequence from a corrupted version, where absorbing-state corruption replaces selected tokens with [MASK].The corrupted sequence is denoted x_s, while the clean sequence is x_0.
- Training formulation: The scalar s ∈[0, 1] controls corruption severity, from no masking at s = 0 to masking every position at s = 1.During training, s is sampled from U(0, 1], and positions are corrupted independently.
- Training formulation: The model predicts an original-vocabulary distribution at every position, while training applies a 1/s-reweighted cross-entropy loss only to masked positions.The conditional distribution is pθ(x_j0 | x_s).
- Inference procedure: Inference begins with L [MASK] tokens and proceeds through T denoising steps, unmasking a subset of currently masked positions according to an unmasking policy.The policy determines how many tokens are revealed at each step.
- Denoising-time proxy: τ_t is interpreted as an empirical measure of denoising progress and used as a proxy for denoising time in masked diffusion language models.Its expected relationship to the continuous diffusion-time variable s is discussed in Section A.
3 Recovering τ
DLM residual streams encode a signal related to the current denoising step τ, which can be accurately recovered with layer-specific MLP probes. This information is distributed across token representations, including both [MASK] and non-[MASK] tokens, even without sequence-level context.
- Probe-based recovery: Layer-specific MLP probes recover the sequence’s current denoising statistic τ from residual-stream hidden states in LLaDA and Dream.Probes are trained separately for every layer and model, using [MASK], non-[MASK], or all tokens.
- Token localization: Both [MASK] and non-[MASK] token activations support accurate τ prediction, with [MASK] tokens yielding marginally better predictions.The result indicates that τ information is not exclusively localized to masked positions.
- Token-level encoding: Single-hidden-state probes achieve high performance without sequence context or neighboring activations, showing that individual token representations encode substantial information about sequence-level τ.Each probe operates independently on one hidden state.
- Conclusion: The findings answer RQ1 affirmatively: DLMs internally model a denoising-step-related signal that can be accurately recovered.The recovered signal is subsequently used to construct an inference-time steering method.
4 Assessing the Signal’s Importance
The denoising-time signal τ can be extracted from mean activation vectors and used to causally steer model behaviour. Steering toward later or earlier denoising progress predictably changes confidence, entropy, and KL divergence, while models can correct many early-layer perturbations.
- Extracting the Signal: Mean activation vectors grouped by denoising step recover the probe-identified τ signal with high correlations in both models.Correlations reach 0.976 Pearson and 0.980 Spearman for LLaDA, and 0.962 Pearson and 0.974 Spearman for Dream.
- Steering Method: Steering swaps the current-step mean vectors for target-step vectors, applying the resulting perturbation across tokens and layers.The perturbation l := µt̂,l − µt,l cancels the layerwise mean component independent of denoising progress; norm-matched random covariance directions provide a control.
- Downstream Effects: Steering effects are strongest in the final transformer layers of both LLaDA and Dream.The reported examples use layer 29 of LLaDA and layer 25 of Dream, with similar trends across layers shown in supplementary analyses.
- Downstream Effects: Targeting larger τ makes the model more confident with lower entropy, whereas targeting smaller τ lowers confidence and raises entropy.KL divergence grows approximately proportionally to |t̂ − t|, linking the magnitude of the downstream change to the steering distance.
- Specificity of the Signal: τ steering produces coherent downstream trends that random perturbations lack, with approximately twice the KL divergence at comparable perturbation norms.This indicates that the learned τ directions form a particularly sensitive and semantically meaningful subspace.
- Propagation and Correction: Models recompute τ across depth and correct most perturbations injected into early layers, though extreme targets such as t̂ = 100 can leave persistent discrepancies.The resulting signal has direct causal implications for entropy, confidence, and KL divergence.
5 Characterising the Signal
The inferred timestep signal is concentrated in a low-dimensional, ordered geometry: two principal components recover steering behavior, while representation alignment varies across layers and model components. LLaDA largely shares semantics across layers, whereas Dream forms representation blocks, and self-attention and the MLP encode the signal oppositely.
- Low-dimensional geometry: A single principal component captures a large fraction of intra-layer variance in the discovered mean vectors, with explained-variance distributions heavily concentrated above 90%.This motivates analyzing the signal through a low-dimensional principal-component subspace.
- Low-dimensional geometry: In LLaDA, steering within the top two principal components closely matches unrestricted steering, whereas equal-norm orthogonal perturbations produce incoherent effects.The result indicates that the orthogonal directions carry little of the inferred τ representation.
- Shared model-level representation: The standardised 2D projections of mean vectors form a stable, coherent parabolic trajectory shared across layers.The trajectory is constructed by averaging binned 2D principal-component projections across layers for each τ index.
- Cross-layer alignment: Across depth, LLaDA mostly preserves strong alignment for the same mean-vector index except at layer 32, while Dream aligns representations mainly within specific layer blocks.Representations outside Dream’s aligned blocks are largely uncorrelated.
- Within-layer computation: Within layers, operations before and after the MLP remain highly correlated, but MLP representations are nearly orthogonal to the signal representations in expectation.The takeaway characterizes self-attention and the MLP as having opposite representations of the mean-vector subspace.
6 Related Work
Prior interpretability work studies activation subspaces and representation geometry, while this work applies mean vectors to a distributed denoising-time signal across layers. Related DLM studies examine stage-specific steering and length-awareness, whereas this work characterizes an internal representation tied to denoising progress.
- LLM Interpretability: Prior LLM interpretability work studies activation subspaces, representation geometry, and concept- or token-level representations.Gurnee et al. (2025) use mean activation vectors to analyze and steer a character-counting task.
- LLM Interpretability: This work similarly uses mean vectors but studies a distributed denoising-time signal and its propagation across layers.
- DLM Denoising Dynamics and Interpretability: Unlike stage-specific concept steering, this work’s τ-related signal persists throughout diffusion and is repeatedly recomputed.Zhou et al. (2026) find that concepts are steerable only at particular denoising stages.
- DLM Denoising Dynamics and Interpretability: Whereas Rossi et al. (2026) study DLM length-awareness through downstream performance, this work characterizes an internal representation tied to denoising progress.
7 Conclusion and Discussion
DLM residual streams encode a probe-decodable, causally steerable denoising-progress signal in a structured low-dimensional subspace. The computation that constructs and updates this signal remains unidentified, motivating future circuit-level investigation.
- Contributions: DLM residual streams contain a denoising-progress signal that is probe-decodable across layers and token types, causally steerable, repeatedly corrected, and structured in a low-dimensional subspace.The signal is represented through mean vector directions and undergoes repeated correction across depth.
- Contributions: Steering the recovered representation predictably changes entropy, confidence, and KL divergence, showing that its directions are functionally relevant to model computation.These effects indicate the directions are not merely descriptive.
- Open questions: DLMs internally track information closely related to the fraction of unmasked tokens, but the circuit that constructs or updates this representation remains unknown.Possible sources include explicit mask-ratio estimation, distributed sequence-level statistics, or attention–MLP interactions.
8 Limitations
The analysis is limited to two DLMs trained with the same [MASK]-token cross-entropy loss, leaving broader generality and inference-time uses of the τ signal unresolved. Token-level effects of steering also remain unexplored because the study focuses on sequence-level statistics.
- Model and training scope: The analysis covers only LLaDA and Dream, both trained with the same cross-entropy loss over [MASK] tokens.How the τ phenomenon extends beyond these models and training conditions remains unexplored.
- Inference-time exploitation: Whether the identified τ signal can enable more efficient decoding or remasking strategies at inference time remains unclear.The authors leave these potential performance improvements as a promising direction for future work.
- Steering effects: Token-level effects of steering remain unexplored because the analyses focus on sequence-level statistics rather than which tokens change and how.The limitation concerns the behavior of individual tokens as the signal is steered.
10 Ethical considerations
The study analyses internal representations of publicly released diffusion language models without involving human subjects, collecting new data, or releasing new models, and foresees no ethical concerns.
- 10 Ethical considerations: The work analyses publicly released LLaDA and Dream models without human subjects, new data collection, or releasing new models.The authors foresee no ethical concerns arising from the work.
A Denoising Progress and the Training Noising Level
The training noising level s is the independently sampled probability that response tokens are masked, whereas τ_t measures the realised fraction of response tokens unmasked during inference. Their relationship is an analogy between mask ratios, not an equality between training and inference distributions.
- Training noising level: During training, each token is independently replaced by [MASK] with probability s, so s controls the expected fraction of masked tokens.The corresponding visibility probability is 1 − s.
- Connection to the training noising level: The realised inference-time mask ratio 1 − τ_t serves as an effective noising level, while τ_t measures effective denoising progress.This correspondence is defined through response-local mask ratios.
- Interpretation: The correspondence between s and 1 − τ_t is only at the level of mask ratios, because training samples independent corruption whereas inference generates states through the model’s unmasking policy.Both procedures nevertheless produce partially masked responses with well-defined response-local mask ratios.
- Interpretation: Recovering τ_t from the residual stream can be interpreted as recovering the model’s internal representation of its position along the denoising trajectory.The paper therefore uses τ_t as its empirical measure of denoising progress.
B Probe Architectural and Training Details · D Low-dimensional Steering · E Depth Correction
The supplementary analyses specify the probe architecture and training procedure, extend τ decoding and steering across models and conditions, and show that low-dimensional subspaces capture most τ-relevant geometry while perturbation effects vary with denoising depth.
- B Probe Architectural and Training Details: The probes use five Linear→LayerNorm→GELU blocks around one residual connection, a sigmoid output in (0, 1), and hidden width capped at 1024.They are trained for 20 epochs on 300 training and 100 validation examples with dynamically varied generation lengths and denoising steps.
- B Probe Architectural and Training Details: MLP probes decode τ accurately across Dream, although accuracy slightly degrades in deeper layers.This extends the corresponding observation from the main analyses to Dream.
- B Probe Architectural and Training Details: Linear probes track τ through early and middle layers, then degrade more steeply than MLP probes and collapse to the mean-τ baseline in final layers.The contrast indicates that token-level linear decoding becomes harder with depth while MLP probes retain more information.
- B Probe Architectural and Training Details: Mean-vector steering in Dream decreases entropy and increases confidence when targeting later denoising steps, with the same trajectory preserved across generation lengths.The generation-length ablation uses L = 64 and L = 128 with denoising steps matched to L.
- B Probe Architectural and Training Details: Steering does not collapse LLaDA’s downstream performance, whereas Dream is more sensitive and degrades substantially on GSM8K at ˆt = 100.Dream’s GSM8K degradation is largely associated with premature EOS emission, which can cause generation to collapse.
- D Low-dimensional Steering: A 1D projection of the mean vectors recovers similar steering effects to untouched mean vectors, while 2D subspaces reproduce those effects with high precision.The results imply that the 2D parabolas encode most of the τ-relevant geometry.
- D Low-dimensional Steering: The orthogonal dimensions ∆t→ˆt l,⊥ have low-impact effects without a clear trend, suggesting little relevance to modelling τ.This contrasts with the strong effects recovered from the one- and two-dimensional mean-vector subspaces.
- E Depth Correction: Early denoising steps are markedly more sensitive to injected perturbations, with substantial absolute probe drift across nearly the network’s depth.For ˆt = 100, some perturbations penetrate the entire model and distort predictions in the final layers.
F Dream’s Mean Activation Analysis
Dream’s mean activations encode τ in a structured, low-dimensional space with a smooth ordered trajectory across denoising time. Unlike LLaDA, Dream shows larger boundary deviations and heterogeneous cross-layer organisation, including three computational stages and qualitatively distinct extreme steps.
- Low-dimensional geometry: Dream’s τ representations concentrate in a structured low-dimensional subspace and follow a smooth, ordered trajectory across denoising time.This mirrors the mean-activation pattern reported for LLaDA.
- Shared 2D geometry: The shared 2D geometry forms a clear parabola, but its spread is larger at t = 1 and t = 100, indicating stronger boundary variation across layers than in LLaDA.At these endpoints, the geometrical organisation changes significantly across layers.
- 3D geometry: Dream’s layer-25 3D trajectory is low-dimensional and ordered, while the initial and final denoising steps appear as outliers.These extreme-step outliers suggest qualitatively different model behaviour and help explain the boundary deviations in the shared 2D geometry.