PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion

Mang Ning, Mingxiao Li, Le Zhang, Lanmiao Liu, Matthew B. Blaschko, Albert Ali Salah, Itir Onal Ertugrul

arXiv:2603.14645v1cs.CV

TL;DR

Latent diffusion quality depends on latent diffusability, not reconstruction fidelity alone. The paper proposes Spectrum Matching through ESM and DSM, improving generation quality across VAE and REPA settings, with REPA-DoG reaching a best gFID of 4.98.

  • Problem

    Reconstruction improvements for VAEs do not consistently improve latent-diffusion generation, motivating better understanding and optimization of latent diffusability.

  • Method

    Spectrum Matching uses flattened power-law latent spectra and frequency-consistent decoding through ESM PSD matching and DSM shared spectral masking.

  • Results

    Spectrum Matching improves generation quality over SD-VAE and prior methods, while REPA-DoG achieves the best ImageNet gFID of 4.98 at 400k steps.

  • Takeaways & Limitations

    The spectral perspective unifies latent-diffusion regularization and identifies directional spectral energy as a useful target-representation property for REPA.

  • Takeaways & Limitations

    The study removes the Gaussian KL term because ESM or DSM is assumed to provide a similar Gaussian regularization effect in latent space.

Abstract

from arXiv · show

In this paper, we study the diffusability (learnability) of variational autoencoders (VAE) in latent diffusion. First, we show that pixel-space diffusion trained with an MSE objective is inherently biased toward learning low and mid spatial frequencies, and that the power-law power spectral density (PSD) of natural images makes this bias perceptually beneficial. Motivated by this result, we propose the \emph{Spectrum Matching Hypothesis}: latents with superior diffusability should (i) follow a flattened power-law PSD (\emph{Encoding Spectrum Matching}, ESM) and (ii) preserve frequency-to-frequency semantic correspondence through the decoder (\emph{Decoding Spectrum Matching}, DSM). In practice, we apply ESM by matching the PSD between images and latents, and DSM via shared spectral masking with frequency-aligned reconstruction. Importantly, Spectrum Matching provides a unified view that clarifies prior observations of over-noisy or over-smoothed latents, and interprets several recent methods as special cases (e.g., VA-VAE, EQ-VAE). Experiments suggest that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets, and outperforms prior approaches. Finally, we extend the spectral view to representation alignment (REPA): we show that the directional spectral energy of the target representation is crucial for REPA, and propose a DoG-based method to further improve the performance of REPA. Our code is available https://github.com/forever208/SpectrumMatching.

1 Introduction

The paper reframes latent diffusion around latent diffusability rather than reconstruction fidelity, proposing Spectrum Matching as a unified spectral perspective. It further extends this perspective to representation alignment by linking directional spectral energy to effective target representations and introducing a DoG-based method.

  • Motivation: Better VAE reconstructions do not necessarily improve downstream diffusion generation, motivating optimization for latent diffusability or learnability.Reconstruction-focused VAE improvements can produce limited or inconsistent gains in diffusion quality.
  • Spectral perspective: Pixel-space diffusion trained with MSE has an implicit low- and mid-frequency learning bias, which matches the perceptual structure of natural images’ power-law PSD.The paper theoretically argues that this bias is beneficial for modeling image perceptual semantics.
  • Spectrum Matching: The Spectrum Matching Hypothesis proposes that diffusible latents should have flattened power-law PSDs and preserve frequency-to-frequency semantic correspondence through decoding.The paper instantiates these requirements as Encoding Spectrum Matching and Decoding Spectrum Matching.
  • Representation alignment: For REPA, the spectral view identifies directional spectral energy as a key property of effective target representations and motivates a Difference-of-Gaussians method.The paper relates iREPA’s RMS Spatial Contrast metric to directional spectral energy.
  • Spectrum Matching: Encoding Spectrum Matching uses PSD matching, while Decoding Spectrum Matching uses shared spectral masking and frequency-aligned reconstruction to regulate latent representations.These mechanisms provide the paper’s practical implementation of the two Spectrum Matching requirements.

2 Related Work

Prior work separates VAE reconstruction fidelity from latent diffusability, motivating spectral and representation-alignment approaches for improving diffusion performance. Related efforts include increasing latent capacity, shaping latent spectra, and aligning latents with foundation-model representations.

  • VAE reconstruction: Two-stage latent diffusion models use VAEs for first-stage compression, with prior work improving reconstruction through larger batches, EMA updates, and increased latent channel capacity.These efforts include SDXL, SD3-VAE, and Flux-VAE.
  • Latent diffusability: Strong VAE reconstruction fidelity does not necessarily improve downstream diffusion, motivating research on latent diffusability, spectral structure, and alignment with foundation-model representations.One study favors biased, non-uniform latent spectra, while another line aligns VAE latents with foundation-model representations.

3 Spectrum Matching

Spectrum Matching argues that latent diffusion benefits when latent spectra flatten natural-image power laws and preserve frequency-aligned semantics through decoding. It operationalizes these conditions with ESM and DSM regularization, enabling diffusion to inherit low-frequency bias and coarse-to-fine generation.

  • Motivation: Pixel-space MSE diffusion is inherently biased toward low and mid spatial frequencies because its learnable signal decays rapidly with frequency under natural-image power-law PSDs.This bias aligns diffusion optimization with perceptually dominant image structure.
  • Spectrum Matching Hypothesis: The Spectrum Matching Hypothesis requires Encoding Spectrum Matching, which flattens the latent PSD relative to the natural-image spectrum to improve diffusability.ESM targets Sz(ω) ∝ |ω|^−(α−δ) with δ > 0 when images follow Sx(ω) ∝ |ω|^−α.
  • Spectrum Matching Hypothesis: Decoding Spectrum Matching requires frequency-aligned decoding so latent frequency bands reconstruct corresponding image bands, preserving their semantic meaning.DSM trains reconstruction from identically masked image and latent representations using an ℓ1 objective.
  • Practical regularization: Together, ESM and DSM let latent diffusion inherit perceptually useful spectral bias and coarse-to-fine generation, modeling low-frequency structure before refining higher-frequency details.The method removes the Gaussian KL term because ESM or DSM can provide a similar Gaussian regularization effect, while ESM adds negligible computational cost.
  • Implications: Both ESM and DSM achieve improved diffusion results compared with standard VAEs, while Spectrum Matching also offers a unified lens for interpreting prior latent-space methods.The framework is presented as explaining observations of over-noisy or over-smoothed latents and related VAE approaches.

4 Experiments

Experiments evaluate Spectrum Matching autoencoders and DoG-based REPA across CelebA and ImageNet. ESM-AE and DSM-AE improve latent spectra and CelebA generation quality over SD-VAE, while DSM is further benchmarked on ImageNet and REPA-DoG is evaluated against REPA variants.

  • Spectrum Matching evaluation: The study evaluates reconstruction with rFID, PSNR, and SSIM, and generation with gFID, using SD-VAE and Scale Equivariance as baselines.The Spectrum Matching autoencoders retain the SD-VAE U-Net architecture and compare f8d4 and f16d16 latent configurations.
  • Spectrum Matching evaluation: On CelebA, ESM-AE and DSM-AE consistently outperform SD-VAE in gFID while requiring faster diffusion-model training.Models are trained from scratch with shared architecture, batch size 48, and 500,000 steps before unconditional U-ViT latent-distribution modeling.
  • Latent-space analysis: ESM-AE and DSM-AE produce smoother latents than SD-VAE while preserving semantic structure, and both induce flatter power-law PSDs than RGB images.The spectral behavior is reported in Figures 3–5; Scale Equivariance also shows a power-law-like spectrum.
  • ImageNet evaluation: Because DSM achieves better diffusability than ESM while also flattening the power-law PSD, the authors advocate DSM as a simple Spectrum Matching solution and benchmark it on ImageNet.The ImageNet experiment trains f16d16 SD-VAE and DSM-AE for 600,000 steps with batch size 128, then models latents with SiT and accelerates training using REPA.
  • REPA evaluation: The REPA-DoG experiment tests whether preprocessing target representations with a DoG filter improves REPA using REPA and iREPA as comparison methods.All methods are trained on ImageNet 256×256 for 400,000 steps with SiT-B/2 and the same recommended configuration, and results are reported using gFID-50k.

5 Conclusion

Spectrum Matching offers a unified perspective on latent diffusability through the Spectrum Matching Hypothesis, instantiated by ESM and DSM. Experiments improve generation quality over SD-VAE and the prior method, while a spectral extension to REPA uses directional spectral energy and DoG-based preprocessing.

  • Spectrum Matching: Spectrum Matching unifies latent diffusability through the Spectrum Matching Hypothesis and its practical mechanisms, ESM and DSM.The paper formalizes the perspective with the Spectrum Matching Hypothesis and instantiates it using Encoding Spectrum Matching and Decoding Spectrum Matching.
  • Experimental findings: Experiments on CelebA and ImageNet demonstrate improved generation quality over SD-VAE and the prior method.The conclusion reports this improvement across both datasets and both comparison baselines.
  • REPA extension: The spectral perspective extends to REPA by linking RMSC to directional spectral energy and introducing DoG-based band-pass preprocessing.The conclusion states that the DoG-based preprocessing yields additional gains for REPA.

A Appendix · A.1 Proof of Proposition 3.1

The proof decomposes the spatial-domain diffusion MSE into independent frequency-wise terms, then characterizes each term’s maximum reducible error as learnable signal power. Under a local Gaussian/LMMSE approximation, this power depends on the frequency spectrum and timestep SNR, yielding the proposition’s expression.

  • A.1 Proof of Proposition 3.1: Parseval’s theorem converts the spatial-domain MSE into a sum of per-frequency MSE terms.The argument begins with Fourier coefficients of a random natural image and uses the unitary DFT.
  • A.1 Proof of Proposition 3.1: Because the diffusion noise variance is frequency-independent, each Fourier coefficient can be analyzed as a scalar diffusion transition.The proof defines Y0 and Yt for an arbitrary frequency ω before analyzing estimation error.
  • A.1 Proof of Proposition 3.1: For each frequency, the Bayes-optimal estimator is the conditional mean, and its minimum squared-error risk determines the maximum achievable MSE reduction.The proof denotes the signal power by S(ω) = E|Y0|².
  • A.1 Proof of Proposition 3.1: The maximal per-frequency MSE reduction is interpreted as the learnable signal power at frequency ω and timestep t.This interpretation connects the estimation-theoretic reduction directly to diffusability analysis.
  • A.1 Proof of Proposition 3.1: Under the local Gaussian/LMMSE approximation, the conditional mean is linear, with covariance and variance determined by S(ω) and the timestep noise schedule.Specifically, Cov(Y0, Yt) = √ᾱ_t S(ω) and Var(Yt) = ᾱ_tS(ω) + (1 − ᾱ_t).
  • A.1 Proof of Proposition 3.1: The resulting learnable signal power is Δ(ω, t) = S(ω) · SNR_t(ω) / (1 + SNR_t(ω)).This follows from substituting the scalar diffusion covariance and variance into the conditional-mean reduction.
  • A.1 Proof of Proposition 3.1: Thus the maximum reduction of per-frequency MSE is proportional to G_t(ω) = S(ω) SNR_t(ω) / (1 + SNR_t(ω)), completing the proposition’s proof.The expression combines the frequency-dependent signal power with the timestep-dependent SNR factor.

A.2 Encoding Spectrum Matching (ESM) from an Information Theory Perspective

Under a finite latent-capacity budget, maximizing carried information favors a spectrally flat latent distribution. Thus, relative to natural-image spectra, the encoder should suppress low-frequency redundancy and relatively enhance high-frequency energy.

  • Information-theoretic motivation: Finite latent capacity makes entropy-maximizing information allocation favor a flat latent power spectrum.This is described as a whitening or spectral-flattening tendency.
  • Implication for VAEs: Because natural-image PSDs decay with frequency, the encoder should produce latents whose PSD decays more slowly than the input spectrum.This reflects flattening relative to the natural-image spectrum under finite latent dimensionality.
  • Gaussian maximum-entropy bound: Under the Gaussian reference model, differential entropy is maximized when the latent spectrum is constant across the frequency domain.The Gaussian model supplies a maximum-entropy upper-bound argument for stationary fields with fixed PSD.
  • Gaussian maximum-entropy bound: Equality in the entropy bound holds if and only if the latent PSD is constant almost everywhere.The proof uses strict concavity of log and Jensen’s inequality under the fixed energy budget.
  • Implication for VAEs: The encoder therefore suppresses low-frequency redundancy while relatively increasing high-frequency energy.This is the operational interpretation of spectral flattening in the encoder.

A.3 Spectrum Analysis of UAE and VA-VAE

On ImageNet 256×256, VA-VAE latents closely match the target power-law spectrum with δ = 1.0, suggesting that DINOv2 feature alignment implicitly regularizes Encoding Spectrum Matching.

  • VA-VAE Spectrum: DINOv2 feature alignment acts as an implicit regularization of Encoding Spectrum Matching in VA-VAE.

A.4 Scale Equivariance and EQ-VAE are special cases of DSM

Scale Equivariance and EQ-VAE can be unified as spatial-downsampling methods whose decoder regularization removes high-frequency spectral components. This connects them to DSM, where frequency masking generalizes the specific downsampling strategy.

  • Spatial Downsampling as DSM: Scale Equivariance and EQ-VAE are unified as spatial-downsampling methods because their gains mainly come from scaling transformations.Scale Equivariance downsamples latents by 2× or 4×, while EQ-VAE uses rotation and scaling to regularize the decoder.
  • Spatial Downsampling as DSM: Spatial downsampling corresponds in the frequency domain to removing high-frequency components from the spectrum.In the 2D-DCT, the top-left quarter contains the information needed to reconstruct an image downsampled by 2×.
  • Spatial Downsampling as DSM: DSM generalizes these methods by allowing the frequency mask M to take any shape, instantiated here with triangular masks.Figure 9 contrasts spatial downsampling in DCT space with the triangular-mask construction used for DSM.

A.5 Proof of Proposition 3.2 … A.8 Ablation Studies of ESM and DSM

The appendix proves Proposition 3.2 using orthonormal DCT coefficients and Parseval’s identity, specifies training setups, defines triangular DCT masks for progressive filtering, and reports ESM and DSM ablations on CelebA.

  • A.5 Proof of Proposition 3.2: Because ∥u_t∥2 = 1 for every t, the first expanded term equals 1.The proof then identifies the DC coefficient with the mean and derives the remaining energy expression.
  • A.5 Proof of Proposition 3.2: The proof applies an orthonormal DCT along the temporal dimension and uses Parseval’s identity to relate coefficient energy to the original representation.The DCT matrix satisfies C⊤C = I_T, and coefficient vectors U_k are formed for each feature dimension.
  • A.5 Proof of Proposition 3.2: The proof derives 2 = T − ∥U_0∥2^2 = T − T∥ū∥2^2 = T RMSC(x)^2 and completes the result by dividing by T.The final equality uses equation (11).
  • A.6 Training Parameters: SD-VAE, ESM-AE, and DSM-AE are trained on CelebA 256×256 with 4 A100 GPUs and on ImageNet 256×256 with 8 A100 GPUs.The appendix provides the complete autoencoder training parameters in Table 5.
  • A.6 Training Parameters: REPA, iREPA, and REPA-DoG are trained on ImageNet 256×256 using 4 A100 GPUs.Their training parameters are listed in Table 6.
  • A.7 Design of Frequency Mask M: The DCT frequency mask set uses triangular shapes, with the number of diagonal rows n uniquely defining each mask under the 8×8 JPEG DCT convention.Low frequencies occupy the top-left of zigzag-ordered DCT space, while high frequencies occupy the bottom-right.
  • A.8 Ablation Studies of ESM and DSM: For ESM, δ = 1.0 and β = 0.01 provide the best diffusability on CelebA 256×256.The optimal δ = 1.0 is consistent with the analyzed DINOv2 feature spectrum.
  • A.8 Ablation Studies of ESM and DSM: For DSM, the mask-set design controls diffusability, and the best result comes from a proper element density while retaining n = 0 for original-image training.The study tests mask sets including n = {8, 12}, where the two samples remove 8 and 12 diagonal frequency rows.
Loading 2603.14645v1…