Source-linked AI summary
Deconstructing Denoising Diffusion Models for Self-Supervised Learning
Xinlei Chen, Zhuang Liu, Saining Xie, Kaiming He
TL;DR
The paper asks which parts of generation-oriented DDMs actually support self-supervised representation learning. It deconstructs DDMs into recognition-oriented DAEs and finds that a low-dimensional latent space is critical, producing a simplified l-DAE while revealing remaining performance gaps and scope limits.
Problem
It remains unclear whether DDM representation capability comes from denoising or diffusion, because prior studies mainly evaluated generation-oriented models.
Source: §1 · p. 1Method
The study progressively deconstructs DDMs into classical DAEs and compares components including different tokenizers and noise designs.
Source: §1 · p. 2, §4.2 · p. 4Results
Takeaways & Limitations
Denoising-based methods resembling classical DAEs merit renewed study for modern self-supervised learning.
Source: §6 · p. 8Under this protocol, autoencoder-based methods still trail contrastive learning methods, especially when the model is small.
Source: §5 · p. 8
Abstract
from arXiv · showhide
In this study, we examine the representation learning abilities of Denoising Diffusion Models (DDM) that were originally purposed for image generation. Our philosophy is to deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore how various components of modern DDMs influence self-supervised representation learning. We observe that only a very few modern components are critical for learning good representations, while many others are nonessential. Our study ultimately arrives at an approach that is highly simplified and to a large extent resembles a classical DAE. We hope our study will rekindle interest in a family of classical methods within the realm of modern self-supervised learning.
1. Introduction
The study deconstructs generation-oriented DDMs into recognition-oriented denoising autoencoders to identify which components support representation learning. It finds that a low-dimensional latent space is critical, while diffusion-specific complexity is largely nonessential, yielding a simplified l-DAE that remains competitive but trails contrastive and masking-based methods.
Motivation and approach
- The study deconstructs generation-oriented DDMs step by step into recognition-oriented models to examine which modern components matter for representation learning.The procedure compares DDM components while moving toward a classical DAE.Source: §1 · pp. 1–2
Findings
- A low-dimensional latent space, rather than tokenizer specifics, is the main critical component for good DAE representations.The authors test standard VAE, patch-wise VAE, patch-wise AE, and patch-wise PCA tokenizers.Source: §1 · p. 2
- The l-DAE projects images into a patch-wise PCA latent space, adds noise there, reconstructs pixels with inverse PCA, and predicts the denoised image.This architecture largely resembles a classical DAE, differing mainly in where noise is added.Source: Figure 1 · p. 1, §1 · p. 2
- Single-level noise achieves a decent l-DAE result, while multiple noise levels act like beneficial data augmentation rather than an enabling factor.This supports the view that DDM representation capability mainly comes from denoising, not diffusion.Source: §1 · p. 2
- The resulting methods substantially outperform off-the-shelf counterparts but remain below contrastive and masking-based baselines, although the gap is reduced.The authors identify further research opportunities for DAE- and DDM-based self-supervised learning.Source: §1 · p. 2
2. Related Work
Related work connects generative modeling with self-supervised representation learning, but strong generation quality does not necessarily yield useful downstream representations. DDM representation studies therefore motivate a deeper analysis, while additive-noise classical DAEs remain comparatively underexplored and weak.
Generative models
- High-fidelity generation has often been treated as evidence of representation potential, but it does not necessarily produce representations useful for downstream tasks.This concern motivates examining DDMs beyond their generation quality.Source: §2 · p. 2
DDM representation learning
- Pioneering studies evaluate existing pre-trained DDM networks for recognition and report encouraging representation results, but leave the source of that capability unclear.The unresolved issue is whether representations arise from denoising or diffusion.Source: §2 · p. 2
Denoising autoencoders
- DAEs established autoencoding-based representation learning, while recent successful variants mainly use masking-based corruption such as MAE and related methods.Classical additive Gaussian-noise DAE variants have received little recent study.Source: §2 · p. 2
- A simple classical DAE with additive Gaussian noise performs poorly, with reported accuracy around 20%, helping explain the limited recent attention to this setting.The passage attributes this observation to prior DAE baseline efforts and reports it as an example rather than a general bound.Source: §2 · p. 2
3. Background: Denoising Diffusion Models
DDMs progressively corrupt clean data with Gaussian noise and train a network to remove it, often operating in a tokenizer-defined latent space. The background establishes DiT as the study’s Transformer-based starting point and distinguishes its generation objective from representation evaluation.
Diffusion process
- At timestep t, diffusion forms z_t by scaling clean z_0 and adding Gaussian noise, with γ_t and σ_t controlling signal and noise.The process uses ϵ∼N(0, I) and a noise schedule across timesteps.Source: §3 · p. 2
Denoising objective
- Unlike a classical DAE, a modern DDM commonly predicts the noise ϵ and is trained across scheduled noise levels conditioned on t.During generation, the trained model is applied iteratively until reaching the clean signal z_0.Source: §3 · pp. 2–3
Input spaces
- DDMs can operate directly on pixels or on latent representations produced by a tokenizer, such as a VQVAE mapping x_0 to z_0=f(x_0).The study focuses on the latent-space formulation used by modern models.Source: §3 · p. 3, Figure 2 · p. 3
DiT baseline
- The study begins with DiT because its Transformer architecture enables fairer comparisons with Transformer-based self-supervised baselines and separates encoder and decoder components more clearly than UNets.The baseline uses DiT-L and evaluates its encoder with linear probing.Source: §3 · p. 3
- The DiT-L starting baseline reports 57.5% linear-probe accuracy and FID-50K of 11.6.These values define the initial point of the study’s deconstructive trajectory.Source: §3 · p. 3
4. Deconstructing Denoising Diffusion Models
The study progressively simplifies a generation-oriented DDM into a classical-style DAE, finding that a low-dimensional latent space is central while many modern components are unnecessary for representation learning.
Reorienting DDM
- Removing class conditioning raises linear-probe accuracy from 57.5% to 62.1%, despite worsening FID from 11.6 to 34.2.The authors hypothesize that removing labels forces the model to encode more semantic information.Source: §4.1 · p. 3
- Eliminating supervised perceptual loss produces the first legitimate tokenizer, reducing accuracy from 62.5% to 58.4%; removing adversarial loss then increases it to 59.0%.The perceptual loss itself supplies semantic representations through a supervised ImageNet classifier, so it is excluded thereafter.Source: §4.1 · p. 4
- 63.4% linear-probe accuracy follows from replacing the generation-oriented noise schedule, which allocates more capacity to cleaner images, though FID worsens to 93.2.The original schedule spends many steps on very noisy images; the revised schedule improves representation quality but harms generation quality.Source: §4.1 · p. 4
Deconstructing the Tokenizer
- Across convolutional, patch-wise, AE, and PCA tokenizers, low latent dimensions perform best, with d=16 or 32 despite 768 dimensions per 16×16×3 patch.Patch-based tokenizers consistently outperform the convolutional VAE, and KL regularization is unnecessary because AE and PCA variants work well.Source: Latent dimension of the tokenizer is… · p. 5
- A pixel-based tokenizer confirms that latent dimensionality matters: d=48 is optimal, whereas d=768 yields only 23.6% linear-probe accuracy.The result supports the conclusion that applying additive Gaussian noise directly in pixel space performs poorly.Source: High-resolution, pixel-based DDMs are… · p. 6
5. Analysis and Comparisons
The analysis finds that l-DAE learns useful representations from latent-space denoising, with performance competitive with masking-based methods but below contrastive learning under the reported protocol.
Denoising results
- Heavy latent noise creates a challenging pretext task that can require higher-level, holistic semantics to infer objects and scenes.The denoising visualizations show reasonable predictions despite heavy noise, while the paper notes image restoration is an established field.Source: §5 · p. 7, Figure 8 · p. 8
Data augmentation
- 64.5% versus 65.0% with center crop and random crop, respectively, shows mild augmentation gives l-DAE only a slight improvement.The authors conclude representation learning is largely independent of augmentation, unlike contrastive learning methods.Source: §5 · p. 7
Training epochs
- The study examines training duration alongside prior MAE and MoCo v3 references, using 400 epochs by default and considering 800 and 1600 epochs.The supplied passages introduce the schedule but do not report the resulting l-DAE values.Source: §5 · p. 8
Model size
- 10.6% gain from ViT-B to ViT-L indicates strong scaling with model size, compared with MAE’s 7.8% gain.The comparison supports a similar scaling trend across the two methods.Source: §5 · p. 8
Comparison with previous baselines
- l-DAE trails MAE by 1.4% with ViT-B and 0.8% with ViT-L under matched 1600-epoch, random-crop training.The authors nevertheless characterize the accuracy gap between MAE and the DAE-driven method as largely closed.Source: §5 · p. 8, Table 4 · p. 8
- Autoencoder-based methods still fall short of contrastive learning methods under this protocol, especially when the model is small.The authors identify this as an outstanding gap for further research.Source: §5 · p. 8
6. Conclusion
The paper concludes that l-DAE, a model closely resembling the classical DAE, can perform competitively for self-supervised learning. Its critical ingredient is a low-dimensional latent space where noise is added, motivating renewed study of denoising-based methods.
Conclusion
- l-DAE largely resembles the classical DAE while performing competitively in self-supervised learning.The method is presented as the final outcome of the deconstructive study.Source: §6 · p. 8
- A low-dimensional latent space on which noise is added is identified as the critical component for representation learning.The conclusion motivates renewed interest in denoising-based methods.Source: §6 · p. 8
A. Implementation Details
The implementation uses a DiT-based Transformer with specified optimization and probing protocols, while transfer learning fixes the conditioning timestep and explores encoder depth.
DiT architecture
- The DiT implementation conditions each Transformer block through a two-layer MLP whose output supplies adaptive LayerNorm scale and bias parameters.The MLP hidden dimension is reduced to one quarter of the original without reported accuracy cost.Source: A. Implementation Details · p. 9
Training
- Training uses batch size 2048, 100-epoch linear warmup, half-cycle cosine decay, base learning rate 1e-4, no weight decay, and 400 epochs.The actual learning rate follows linear scaling with batch size.Source: A. Implementation Details · p. 9
Linear probing
- Linear probing uses globally averaged ViT features, parameter-free BatchNorm, a linear classifier, 90 epochs, and reports ImageNet top-1 accuracy.Training uses random resized crops and flips, while testing uses a single center crop.Source: A. Implementation Details · p. 9
- Fixing t during transfer learning removes the need for timestep conditioning in the probe and yields a model parameter count similar to standard ViT.The reported sweep evaluates clean and noisy inputs across several fixed timestep values.Source: A. Implementation Details · p. 9
Encoder and decoder
- The DiT-L model has 24 blocks, with the first 12 designated as encoder and the remainder as decoder, although this split is described as artificial.The study probes different encoder block counts using the same pretrained model.Source: A. Implementation Details · p. 9
- Matching encoder and decoder depth achieves optimal accuracy, unlike the asymmetric design used by MAE.This comparison concerns the reported architecture-depth behavior.Source: A. Implementation Details · p. 9
B. Fine-tuning Results
End-to-end fine-tuning shows both autoencoder-based methods outperform MoCo v3, while l-DAE remains slightly behind MAE at the larger model size.
Fine-tuning Results
- 83.7 and 84.7: l-DAE achieves higher end-to-end fine-tuning accuracy than MoCo v3 at both ViT-B and ViT-L sizes.MoCo v3 scores 83.2 and 84.1, compared with l-DAE scores of 83.7 and 84.7.Source: B. Fine-tuning Results · p. 9
- 83.6 and 85.9: MAE also outperforms MoCo v3 at ViT-B and ViT-L, respectively.The reported MAE scores exceed MoCo v3 for both model sizes.Source: B. Fine-tuning Results · p. 9
- l-DAE performs similarly to MAE with ViT-B but falls short of MAE with ViT-L.The comparison is based on end-to-end fine-tuning results using the reported protocol.Source: B. Fine-tuning Results · p. 9