Source-linked AI summary
Feature Information Dynamics in Diffusion
Jia-Shu Pan, Tao Zhang, Yufei Huang, Yanjun Sheng, Tailin Wu
TL;DR
Evidence for when features emerge during diffusion has largely been qualitative, despite the relevance of their temporal organization. The paper uses I-MMSE-based denoising-loss gaps and a chained decomposition to quantify feature information. It finds different dynamics across representations, with only RAE showing a clean class →mask →Canny ordering.
Problem
Accounts of when and in what order features emerge during diffusion remain mostly qualitative, leaving a general quantitative account missing.
Source: §1 · p. 1Method
The framework uses I-MMSE to relate feature mutual-information change to unconditional and feature-conditional denoising losses, then chains the decomposition to separate incremental information in feature hierarchies.
Source: §1 · pp. 1–2Results
Across pixel, SDVAE, VAVAE, and RAE diffusion, feature information dynamics differ; only RAE achieves a strict class →mask →Canny ordering of density peaks.
Source: §7 · p. 10Takeaways & Limitations
Representation choice is a controllable knob over feature information dynamics.
Source: §7 · p. 10The comparison covers four representations and one conditioning chain, and the information profiles are distribution-level averages rather than descriptions of every image.
Source: §6.2 · p. 10
Abstract
from arXiv · showhide
Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class $\to$ mask $\to$ Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at https://github.com/AI4Science-WestlakeU/feature-information-dynamics.
1 Introduction
The paper develops a quantitative account of when features emerge during diffusion, extending qualitative observations of hierarchical generation. It links feature information to denoising losses and uses a chained decomposition to study frequency and visual-feature hierarchies.
Feature information dynamics via I-MMSE
- The framework defines SNR-resolved feature information through the I-MMSE identity and relates its rate to unconditional versus feature-conditional denoising losses.It also derives equivalent estimators for feature information profiles.Source: §1 · pp. 1–2
Chained information decomposition
- In pixel diffusion, the chained framework finds spectral autoregression and shows that structure can enter earlier and more strongly than class identity.This challenges a simple semantic-first account of the denoising trajectory.Source: §1 · p. 2
- A chained decomposition separates shared from incremental information across a feature hierarchy.The paper applies it to frequency bands and semantic–structural visual cues.Source: §1 · p. 2
Representations shape feature information dynamics
- Across pixel, SDVAE, VAVAE, and RAE models, feature information dynamics differ; only RAE shows a clean class →mask →Canny ordering.The conditioning chain and evaluation protocol are held fixed across representations.Source: §1 · p. 2
2 Related Work
Prior work connects information theory with deep learning and studies the emergence of visual information during diffusion. The paper positions its framework as a quantitative, feature-general account of this information over the diffusion timeline.
Information theory and diffusion models
- Prior information-theoretic diffusion studies interpret diffusion loss or estimate static mutual information, but do not study information dynamics across the diffusion timeline.Kong et al. [2023] generalized I-MMSE to interpret diffusion loss; Franzese et al. [2024] and Yu et al. [2025] estimate static quantities.Source: §2 · p. 2
- Handke et al. [2026] study progressive class specification, but their analysis is limited to categorical features and the SDE formulation.Source: §2 · p. 2
Feature information dynamics in diffusion
- Earlier accounts describe coarse-to-fine or concept emergence qualitatively, and do not readily generalize beyond pixel-space diffusion.The paper notes this gap matters as latent representations are widely used and representation choice can affect training efficiency.Source: §2 · p. 2
3 Feature Information Density
The section defines feature information density by connecting feature mutual information to the difference between unconditional and feature-conditional denoising errors. It then maps that relationship to diffusion objectives and practical estimators.
3 Feature Information Density
- For clean data X and a feature Y, the framework quantifies how information about Y is distributed across noise levels and defines feature information density through an MMSE gap.The framework can also be instantiated with diffusion models and applies beyond them.Source: §3 · pp. 2–3
3.1 Gaussian channel and MMSE
- The MMSE is the minimum squared-error denoising error from a noisy observation and side information; conditioning on Y cannot increase that error.The feature-conditional and unconditional MMSE difference measures how much Y helps denoising at a given SNR.Source: §3.1 · p. 3
3.2 I-MMSE relation and feature information density
- The I-MMSE identity makes the feature information density the rate of feature–observation mutual information change, measured by the unconditional–conditional MMSE gap.Integrating the density over SNR recovers the feature mutual information.Source: §3.2 · pp. 3–4
3.3 Diffusion and flow-matching parameterization
- For diffusion and flow matching, the MMSE corresponds to the denoising objective, so the gap comes from optimal unconditional and feature-conditional losses without changing training.The paper states this correspondence for Gaussian Flow Matching with independent coupling.Source: §3.3 · p. 4
3.4 Practical estimators
- At high SNR, finite-capacity denoisers can overestimate MMSE and bias direct gap integration; the method fits the gap on a reliable interval and analytically extrapolates its tail.Non-independent endpoint coupling is outside the scope because it breaks the stated Gaussian-channel equivalence for GFM.Source: §3.4 · p. 4
4 Chained Information Decomposition
The chained decomposition addresses shared information that would be double-counted when each related feature is analyzed separately. It attributes incremental information to successive features in a chosen order, localizing when each contributes along the diffusion trajectory.
4 Chained Information Decomposition
- Analyzing related features separately can count their shared information repeatedly, obscuring the full information structure.Source: §4 · p. 5
- The method conditions successively on cumulative feature bundles and attributes each feature's incremental information relative to the preceding bundle.For example, the bundles can progress from class to (class, mask) to (class, mask, Canny).Source: §4 · p. 5
- The chained density localizes when each feature is most informative; its peak ordering is an empirical signature of how the chosen representation organizes feature information.The attribution depends on feature order, and the paper uses class →mask →Canny as its canonical semantic-to-local order.Source: §4 · p. 5
5 Experiments
The experiments use chained feature information dynamics to localize when frequency bands and visual features are generated. Pixel diffusion shows a coarse-to-fine frequency progression, while feature ordering and convergence differ across representations.
5.1 Chained decomposition reveals spectral AR in pixel diffusion
- In the controlled MNIST hierarchy, single-band densities overlap because shared information is counted repeatedly, while forward chaining reveals a coarse-to-fine progression.The experiment compares independent conditioning on each band with cumulative conditioning from coarse to fine.Source: §5.1 · p. 5
- On CIFAR-10, chained feature densities show low frequencies concentrated at lower log-SNR and higher frequencies shifting toward cleaner denoising states.This quantitatively supports approximate spectral autoregression and localizes where each incremental band is resolved.Source: §5.1 · p. 5
5.2 Representations reshape hierarchical information dynamics
- The representation comparison uses the same class →mask →Canny conditioning chain and paired images, classes, masks, and edges across pixel, SDVAE, VAVAE, and RAE.The ImageNet-256 subset retains 485 classes with mask success rates above 95%.Source: §5.2 · p. 6
5.2.1 Training-speed test under matched recipes
- Under a matched recipe, best FID over 200 XL epochs was 78.51 for pixel (epoch 200), 30.83 for SDVAE (200), 21.16 for VAVAE (200), and 8.39 for RAE (180).Source: Figure 3 · p. 8
5.2.2 Hierarchical information-dynamics diagnostic
- Only RAE yields a strict class →mask →Canny ordering; pixel space prioritizes mask over class, while VAVAE prioritizes class but leaves mask and Canny overlapping.The feature-density ordering correlates with convergence speed, but the authors do not claim it causes faster convergence.Source: §5.2.2 · p. 8
6 Discussion
The discussion frames diffusion feature profiles as implicit generation orders that vary across representations, while identifying limitations in the estimator, chained attribution, and empirical scope. It also outlines possible uses for representation selection and diffusion design.
6.1 Generation order in vision and language
- VAR and MaskGIT hard-code coarse-to-fine order, while XLNet and insertion-based language models expose factorization or generation order as design choices [Tian et al., 2024] [Chang et al., 2022] [Yang et al., 2019] [Stern et al., 2019].The discussion connects these approaches to observing diffusion's implicit generation order through the placement of chained information-density peaks.Source: §6.1 · pp. 8–9
- Only RAE produces a clean class →mask →Canny ordering; the four representations realize different implicit orders, and the cleanest order coincides with the fastest training.The same conditioning chain is compared across pixel, SDVAE, VAVAE, and RAE.Source: §6.1 · p. 9, Figure 4 · p. 9
6.2 Limitations
- Measured information gaps can depend on denoiser optimization quality, and pretrained initialization versus training from scratch can yield different measured conclusions.Order, checkpoint, and model-size checks provide direct checks on this estimator dependence.Source: §6.2 · p. 10
- Chained attribution is order-dependent because each increment conditions on the preceding feature bundle, rather than assigning information independently of order.Class →mask →Canny is the canonical semantic-to-local order; an alternative order tests the effect of that choice.Source: §6.2 · p. 10
- The comparison covers four representations and one conditioning chain; profiles are distribution-level averages, and the association with convergence speed is not causal evidence.The analysis also assumes an additive Gaussian channel with independent coupling.Source: §6.2 · p. 10
6.3 Possible uses
- Peak locations and widths could inform candidate timestep schedules or conditioning orders, while profiles may also serve as a representation-selection signal without being treated as a proven predictor.The proposed applications also include feature- or SNR-dependent guidance and diagnosing unresolved information.Source: §6.3 · p. 10
7 Conclusion
The paper presents a quantitative framework for locating feature information along diffusion and extends it to feature hierarchies. Its experiments find representation-dependent information dynamics, with only RAE showing a strict class →mask →Canny peak order.
7 Conclusion
- Feature information dynamics localizes when feature information enters diffusion using I-MMSE and unconditional versus feature-conditional denoising losses.The chained decomposition separates each hierarchy level's incremental contribution.Source: §7 · p. 10
- Pixel diffusion follows spectral autoregression in the chained sense, while only RAE has a strict class →mask →Canny ordering of density peaks across the four representations.Pixel and SDVAE deviate from that ordering.Source: §7 · p. 10
- The findings recast representation choice as a controllable knob over feature information dynamics and potentially diffusion training efficiency.The paper presents this interplay between representation and generation as a topic for further exploration.Source: §7 · p. 10
A Equivalent Estimators
The feature-information density admits several equivalent estimators based on denoising losses and score or Fisher-information quantities. Estimator choice matters for continuous features: feature noise regularizes divergent information, while the analytic tail fit can become degenerate without it.
A Equivalent Estimators
- Equivalent feature-information-density estimators use denoiser gaps, score gaps, Fisher-information gaps, and likelihood scores.Their practical trade-offs include numerical stability, model count, and SNR regime.Source: A Equivalent Estimators · p. 13
- The unconditional-minus-feature-conditional MMSE gap equals the expected squared difference between their optimal denoisers.The cross term vanishes by conditioning on the noisy image and feature.Source: A Equivalent Estimators · p. 13
- Learning a noisy classifier p(Y | Xγ) is not necessarily easier than learning generators for general continuous features, although it may be for categorical features with few classes.Estimator difficulty can differ across the bidirectional information-estimation approaches.Source: A Equivalent Estimators · p. 14
- For general continuous features, I(Y;X) can be infinite when Y is a deterministic function of X; adding feature noise α > 0 restores finiteness.The passage also connects this regularization to finite-precision feature measurement.Source: A Equivalent Estimators · p. 14
- At α = 0, the analytic Lorentzian fit for CIFAR-10 band b10 collapses to b = 10^-12 and a peak at µlg = +12.0, beyond support ending at +1.46.At α = 0.5, the b10 fit is well-conditioned; finite-SNR summaries are close for many bands but do not make the settings interchangeable.Source: A Equivalent Estimators · p. 14, Figure 6 · p. 15, Figure 5 · p. 15
B Analytic Fitting of the Single-Feature MMSE Gap
High-SNR denoising errors can make empirical MMSE gaps unreliable, so the method fits the gap on a reliable interval and extrapolates its tail. Fitting the difference directly avoids severe residual errors, and a single double-pole component fits the reported MNIST bands closely.
B Analytic Fitting of the Single-Feature MMSE Gap
- At high SNR, finite-capacity denoisers can overestimate MMSE, contaminating the gap tail and destabilizing direct numerical integration.The method fits the empirical gap on a reliable region and extrapolates the high-SNR tail.Source: B Analytic Fitting of the… · p. 15
- The fit uses the reliable prefix before the empirical gap’s post-decay minimum, chosen by the “argmin minus two” rule, then extrapolates analytically.The boundary excludes the tail where denoiser underfitting and numerical artifacts are most visible.Source: B Analytic Fitting of the… · p. 16
- Fitting unconditional and conditional MMSE curves separately produced estimates around -105 nats, whereas fitting their difference directly avoids that residual-dominated failure.The individual curves are large, smooth, and nearly parallel, so their fitting residuals can dominate the difference.Source: B Analytic Fitting of the… · p. 16
- For multiband MNIST, one double-pole component per band achieved R2 ≥ 0.98, with most bands above 0.999 in the reliable region.The model uses two free parameters per band.Source: B Analytic Fitting of the… · p. 16
C Spectral Hierarchy Experiment Details
The appendix documents frequency-hierarchy measurements using independent, forward-chained, and reverse-chained conditions across MNIST and CIFAR-10. The reported diagnostics include empirical MMSE gaps and Lorentzian fits used to characterize their decay.
C Spectral Hierarchy Experiment Details
- The MNIST control decomposes images into 17 geometric frequency bands, comparing independent band conditioning with forward-chained cumulative bundles at feature noise α = 0.5.Checkpoints use 100k training steps; evaluation uses 10k test images, four noise draws per image, and 61 diffusion times up to t = 0.995.Source: C Spectral Hierarchy Experiment Details · p. 16
- The CIFAR-10 study uses 12 ordered DCT bands and compares single-band, forward-chained, and reverse-chained conditions; the plotted curve is the pretrained-checkpoint fine-tuning line.The earlier from-scratch models show the same qualitative monotone trend but are visibly limited by denoiser underfitting; the final two high-frequency shells are merged for readability.Source: C Spectral Hierarchy Experiment Details · pp. 16–17
- Double-pole fits track empirical single-feature MMSE gaps across bands, including the decay region that dominates the mutual-information integral.Figure 7 shows K = 1 fits and marks the adaptive reliable cutoff used for fitting.Source: Figure 7 · p. 17
D Representation Experiment Details
The representation experiments use paired data and shared conditioning artifacts to compare chained feature measurements across pixel, SDVAE, VAVAE, and RAE. The appendix specifies the representations, condition construction, evaluation protocol, and implementation checks that delimit this comparison.
D Representation Experiment Details
- The hierarchy uses the same ImageNet-256 SR95 data, paired image lists, condition artifacts, and train/validation whitelists across representations.SR95 contains 485 classes; 90% of images are used for training and 10% for validation/FID reference.Source: D Representation Experiment Details · p. 17
- Masks are fixed foreground artifacts, and Canny maps are computed on masked images, making the edge condition a local boundary cue inside object support.Canny uses grayscale images and thresholds (100, 200), rather than detecting edges across the full image background.Source: D Representation Experiment Details · p. 18
- Under the corrected force-null protocol, all four selected XL unconditional trajectories improve monotonically or near-monotonically across the evaluated checkpoint range.The correction avoids evaluating unconditional checkpoints through a conditional label path that had caused artificial late-epoch FID degradation.Source: D Representation Experiment Details · p. 19
- Each hierarchy stage initializes from the preceding checkpoint: class, class+mask, then class+mask+canny, with validation MSE selecting the evaluation checkpoint.The resulting adjacent chained MMSE gaps are converted to log-SNR information densities using shared post-processing.Source: D Representation Experiment Details · pp. 19–20
- FiLM conditioning improves validation loss for VAVAE and RAE, gives small or negligible gains for pixel space, and does not improve SDVAE in the tested online setting.The comparison is an implementation diagnostic for the hierarchy experiments.Source: D Representation Experiment Details · p. 20, Table 6 · p. 20
- The comparison is representation-level under matched experimental plumbing, not a claim that one architecture and optimizer setting is universally optimal across data spaces.Native patch size, channel dimension, encoder family, prediction parameterization, and stable learning-rate range differ by representation.Source: D Representation Experiment Details · p. 20
- The alternative-order evaluation uses the RAE DiTDH-XL configuration, while the checkpoint sweep uses RAE-S; pretrained initialization and from-scratch training can yield gaps supporting different qualitative conclusions.The sweep compares approximately 132M-parameter RAE-S checkpoints with the approximately 842M-parameter paper model.Source: D Representation Experiment Details · p. 20
NeurIPS Paper Checklist
The checklist asks whether the abstract and introduction accurately reflect the paper’s contributions and scope; the authors answer yes. Its guidance emphasizes matching claims to results and stating important assumptions and limitations.
NeurIPS Paper Checklist
- The checklist asks whether the abstract and introduction accurately reflect the paper’s contributions and scope, and the authors answer yes.The justification states that the contributions and scope are presented accurately.Source: NeurIPS Paper Checklist · p. 22
- The checklist guidance says claims should match theoretical and experimental results and reflect how far findings are expected to generalize.Source: NeurIPS Paper Checklist · p. 22
- The guidance says the abstract and introduction should clearly state contributions and important assumptions and limitations.Source: NeurIPS Paper Checklist · p. 22
2. Limitations
The paper identifies single-seed density trajectories as a limitation because multi-seed runs were computationally infeasible. The checklist also records reproducibility materials and marks broader societal impact as not applicable.
2. Limitations
- The authors state that a dedicated Limitations paragraph discusses the work’s limitations.Source: §2 · p. 22
- The theoretical results state their assumptions inline, with full derivations and equivalence proofs in Appendix A.The stated assumptions include a Gaussian channel and mild regularity for de Bruijn and chain-rule steps.Source: §2 · p. 23
- The checklist reports that code and reproduction instructions will be released and that the experimental settings are documented in the appendices.The reproducibility description covers the data construction, conditioning chain, evaluation procedure, and selected hyperparameters.Source: §2 · pp. 23–24
- The reported feature densities are single-seed trajectories because multi-seed runs were computationally infeasible within the project’s compute budget.The qualitative ordering of chained density peaks is robust across checkpoint cadence, but not across seeds.Source: §2 · p. 25
- The paper marks broader impact not applicable, describing the work as foundational analysis of existing public diffusion models without new models or training data.Source: §2 · p. 26