PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Spectral Representations in Generative Diffusion Models

Yuehao Wang, Peihao Wang, Hanwen Jiang, Ziyi Yang, Qixing Huang, Zhangyang Wang

arXiv:2609.08253v1cs.LG

TL;DR

The paper studies how self-supervised spectral representation learning can align with diffusion modeling through shared perturbation kernels, addressing an insufficiently understood connection. It introduces a spectral regularizer whose optimization is equivalent to diffusion score distillation in representation space, yielding consistent gains across image and 3D point-cloud generation.

  • Problem

    The connection between self-supervised spectral representation learning and diffusion generative modeling, including how representation alignment benefits diffusion training, remains insufficiently understood.

  • Method

    The paper unifies diffusion models and spectral representation learning through perturbation kernels, then integrates a self-supervised spectral representation alignment regularizer into diffusion training.

  • Results

    Consistent gains were achieved in image generation across four datasets and strong performance over the baseline in point-cloud generation.

  • Takeaways & Limitations

    The findings provide a practical, principled path for exploring the synergy between diffusion modeling and representation learning without relying on pretrained external encoders.

  • Takeaways & Limitations

    Validation is limited to relatively small-scale datasets and model sizes, and the method does not outperform external-teacher approaches such as REPA in large-scale latent image generation.

Abstract

from arXiv · show

Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality. Code is released at https://github.com/yuehaowang/spectral-reg-diffusion.

1. Introduction

The paper studies the synergy between self-supervised spectral representation learning and diffusion generation through their shared use of perturbation kernels. It proposes spectral alignment for diffusion training and reports consistent gains across image and point-cloud generation.

  • Diffusion models reverse Gaussian perturbation processes by learning time-dependent score functions on noisy data.
  • Prior work links diffusion models with self-supervised representation learning, including approaches that reuse diffusion models as representation learners and external-teacher alignment methods.
  • The paper unifies diffusion models and spectral representation learning through perturbation kernels and aligns intermediate diffusion representations with time-varying kernel eigenfunctions.
  • The proposed self-supervised spectral alignment yields consistent image-generation gains across four datasets and strong point-cloud performance where pretrained encoders are unavailable.

2. Related Work

Related work connects diffusion models with representation learning through internal feature analysis, external alignment, and self-supervised objectives. The paper is positioned against methods that depend on pretrained encoders or additional supervision.

  • REPA aligns diffusion denoiser features with pretrained vision encoders, while extensions target U-Nets, VAEs, video, semantic tokens, and other settings.
  • External-teacher alignment depends on strong foundation encoders, an assumption that can fail in resource-constrained domains such as 3D shapes and proteins.
  • Studies of self-supervised diffusion models find that hidden activations across time steps encode semantically meaningful information useful for image editing and analysis.
  • Self-supervised representation learning includes negative-based contrastive methods such as SimCLR and MoCo and positive-only methods such as BYOL and SimSiam.

3. Preliminary

The preliminary section formulates diffusion and spectral representation learning through perturbation kernels. Diffusion learns scores for reversing noise, while spectral methods learn kernel eigenfunctions from related and unrelated perturbed samples.

  • 3.1. Diffusion Models from Perturbation Kernels: Diffusion transports data toward a standard Gaussian distribution by gradually adding Gaussian noise through a time-dependent perturbation kernel.
  • 3.1. Diffusion Models from Perturbation Kernels: The perturbation kernel determines the forward SDE through its scaling coefficient s(t), noise scale σ(t), drift, and diffusion coefficient.
  • 3.1. Diffusion Models from Perturbation Kernels: Sampling reverses the diffusion process using a score function estimated by a neural network to match the perturbed data distribution.
  • 3.1. Diffusion Models from Perturbation Kernels: Perturbation-kernel formulations provide analytic conditional scores and simulation-free sampling of noisy views, including xt = s(t)x0 + s(t)σ(t)ϵ.
  • 3.2. Spectral Representation from Perturbation Kernels: Spectral representation learning models perturbations through kernels whose eigenfunctions preserve relationships among data views.
  • 3.2. Spectral Representation from Perturbation Kernels: Spectral contrastive learning uses same-instance perturbed views as positive pairs and independently perturbed views from different samples as unrelated examples.
  • 3.2. Spectral Representation from Perturbation Kernels: Neural Eigenmap trains neural networks to approximate principal eigenfunctions of a kernel integral operator by solving an eigenvalue problem.

4. Bridging Spectral Representations and Diffusion Models

The paper bridges diffusion and spectral learning by reusing diffusion perturbation kernels to construct time-varying spectral representations. It adds a self-supervised spectral regularizer whose optimization has a score-distillation interpretation in representation space.

  • 4.1. Spectral Geometry in Diffusion Process: The method constructs spectral representations from the same perturbation kernel used by the diffusion model.
  • 4.1. Spectral Geometry in Diffusion Process: The resulting time-dependent eigenfunctions represent local geometry on a latent, time-evolving data manifold.
  • 4.1. Spectral Geometry in Diffusion Process: Unlike predefined affinity kernels, the paper derives diffusion distance from the joint probability between two points.
  • 4.2. Spectral Representation Alignment: The self-supervised alignment regularizer avoids external pretrained encoders such as DINO and CLIP, unlike REPA's teacher-based alignment.
  • 4.2. Spectral Representation Alignment: The spectral objective uses two independent noisy views of one clean sample and can jointly reuse a perturbed input with the diffusion objective.
  • 4.2. Spectral Representation Alignment: The regularizer yields multi-scale representations whose embedding distances become less discriminative as noise increases.
  • 4.3. Spectral Representation Learning as Diffusion Score Distillation: The spectral objective is dimension-contrastive and has a dual sample-contrastive form equivalent to diffusion score distillation in representation space.
  • 4.3. Spectral Representation Learning as Diffusion Score Distillation: Its optimization moves anchor representations to reconcile positive and negative score fields, tightening similar-sample clusters and separating dissimilar ones.

5. Experiments

Experiments evaluate spectral regularization on 2D patterns, images, and 3D point clouds, showing improved generation quality and faster convergence over matched diffusion baselines. The method consistently improves image FID and achieves stronger early and final point-cloud results, while adding training overhead and remaining below REPA in large-scale latent generation.

  • Evaluation Setup: Experiments cover 2D pattern fitting, four image datasets, and three ShapeNet point-cloud categories.The evaluation includes settings without external pretrained encoders, including low-resolution images and 3D point clouds.
  • 2D Pattern Fitting: Spectral regularization produces cleaner, more compact 2D samples with fewer out-of-distribution points and finer spiral geometry than vanilla diffusion.The comparison uses 1K training points and visualizes 3K generated samples.
  • Image Generation: The image experiments use DiT and compare matched vanilla baselines using FID primarily, with sFID, IS, precision, and recall as secondary metrics.Image-generation results are reported in Table 1 across adapted resolutions and model sizes.
  • Image Generation: FID decreases by 1.5 on ImageNet-64, 0.2 on ImageNet-256, 2.8 on CIFAR-10, 3.1 on CelebA, and 0.7 on FFHQ versus matched baselines.These correspond to relative improvements of 15%, 8%, 25%, 11%, and 5%, respectively; REPA remains strongest on latent ImageNet-256.
  • Point Cloud Generation: At 5K iterations, Chair 1-NNA falls from 0.850/0.875 to 0.583/0.627, while COV rises from 0.295/0.221 to 0.488/0.493 for CD/EMD.These are relative improvements of 31% and 28% for 1-NNA, and 65% and 123% for COV; longer training gives the best overall results across metrics.
  • Limitations: The method adds an additional perturbed view and projection head, increasing training time by approximately 10% on ImageNet-64, 70% on ImageNet-256, and 20% on Airplane point clouds.Validation is limited to relatively small-scale datasets and model sizes, and REPA remains stronger in large-scale latent image generation.

6. Conclusion

The paper connects self-supervised spectral representation learning and diffusion models through perturbation kernels, then uses that connection to formulate and interpret spectral alignment for diffusion training. The resulting regularizer improves image and 3D point-cloud generation, while establishing an equivalence to representation-space diffusion score distillation.

  • Conclusion: The method unifies diffusion models and spectral representation learning through a shared perturbation-kernel perspective.It also provides a geometric interpretation of joint spectral learning and establishes an equivalent representation-space score-distillation form.
  • Conclusion: Integrating the spectral regularizer into standard diffusion objectives yields consistent gains on image and 3D point-cloud generation.The conclusion presents this alignment as a practical, principled path for studying diffusion and representation-learning synergy.

Impact Statement

The paper presents a spectral alignment approach for improving diffusion training under limited or specialized data conditions, alongside a perturbation-kernel formulation of rectified flow. Its stated impact includes potentially reducing training time and computational cost, while the technical derivation relates the perturbation kernel to forward and reverse dynamics.

  • Impact: The proposed approach targets generative modeling settings with limited training data or specialized structure.The impact statement frames the intended benefit as more effective diffusion training under these constraints.
  • Impact: The stated potential impact is reduced training time and computational cost, with a possible contribution to energy efficiency.These are presented as potential benefits rather than experimentally established outcomes in the impact statement.
  • Technical Connection: The perturbation kernel p0t(xt|x0) = N(xt; (1−t)x0, t^2I) rewrites rectified flow’s linear interpolation as an SDE-compatible stochastic process.This kernel defines the noisy state conditional on clean data and exposes the forward-process formulation.
  • Technical Connection: The perturbation kernel determines the forward SDE coefficients, which then yield corresponding reverse SDE and marginal-preserving ODE formulations.The resulting ODE provides the velocity field adopted by rectified flow through its relation to the score function.

B. Geometric Interpretation of Representations in Eigenspace

The diffusion-induced perturbation kernel defines a compact, self-adjoint, positive operator whose eigendecomposition provides a geometric representation of perturbed data. This structure supports an eigenspace expansion of diffusion distance.

  • The diffusion perturbation kernel κt is a valid symmetric positive semidefinite kernel.
  • Under the stated condition, its integral operator is compact, self-adjoint, and positive on L2(pt).
  • The operator therefore admits an orthonormal eigendecomposition with time-dependent eigenvalues and eigenfunctions.
  • The kernel has a spectral expansion in the eigenspace of its associated kernel integral operator.
  • Diffusion distance can be expressed as an expansion over eigenspace coordinates.

C. Duality of Spectral Representation Learning and Closed-form Diffusion Score Distillation

The paper shows that spectral representation alignment has a diffusion-score interpretation in representation space. Its gradients resemble closed-form diffusion scores, and the spectral regularizer is formally equivalent to score distillation.

  • With normalized embeddings, the spectral regularization objective can be rewritten using an equivalent contrastive formulation.
  • When negative-sample embeddings are detached, the spectral gradients take a batch-wise closed-form diffusion-score form in representation space.
  • For rectified flow, the correspondence identifies the temperature τ with 2t^2 and maps representation samples to scaled data samples.
  • The spectral gradient combines displacement vectors toward other representations with distance-dependent softmax weights.
  • The positive term pushes representations toward positive samples, while the negative term pushes them away from positive-sample representations.
  • The paper models spectral embeddings as a generator and interprets optimizing the regularizer as score distillation toward a target distribution.
  • The proof concludes that optimizing the spectral representation regularizer performs diffusion score distillation.

D.1. Synthetic Distributions

On synthetic 2D distributions, spectral regularization produces tighter samples earlier in training and cleaner hidden-state separation in low-noise conditions, supporting better geometric fitting.

  • Training progress: Our method captures the geometric structure of four 2D distributions more tightly at earlier training stages than the baseline.The comparison covers circles, 8-gaussians, pinwheel, and swissroll.
  • Hidden-state visualization: In low-noise regimes at t = 0.97 and 0.98, our method yields cleaner and more clearly separated hidden representations than the baseline.At t = 0.5 and 0.7, the two methods produce broadly similar hidden representations.
Loading 2609.08253v1…