PaperlayerView exact revision on arXiv

Source-linked AI summary

On the spectral properties of generative denoiser Jacobians

Alexandros Graikos, Nebojsa Jojic, Dimitris Samaras

arXiv:2609.36210v1cs.LGcs.CV

TL;DR

Generation-quality metrics offer limited insight into how denoisers differ, so this paper analyzes denoiser Jacobian spectra and controls them through input perturbations during training. Better generative performance is associated with larger eigenvalues, and ImageNet spectral regularization improves metrics through principal-response strengthening, noisy-response suppression, or both.

  • Problem

    Generation-quality metrics describe synthesized outputs but provide limited insight into how the underlying learned denoising functions differ.

    Source: §1 · p. 1

  • Method

    The paper analyzes denoiser Jacobian spectra and regularizes them by perturbing denoiser inputs during training to suppress or amplify responses.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Better generative performance is associated with larger Jacobian eigenvalues, while ImageNet spectral regularization improves generative metrics through principal-response strengthening or noisy-response suppression, with cumulative gains when combined.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Denoiser Jacobian spectra provide a useful tool for identifying model differences, and the findings suggest effective denoisers benefit from both principal-direction strengthening and suppression of noisy, data-irrelevant responses.

    Source: §1 · p. 2

    Residual regularization requires a second denoiser pass, considerably slows training, and its increased sensitivity weakens classifier-free guidance unless combined with random perturbation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative denoising models, such as diffusion and flow-matching, learn to sample from complex distributions by training a deep neural network denoiser to recover clean data from noise-corrupted samples. While such models are typically compared on the quality of their synthesized samples, these metrics provide limited insight into how the underlying denoiser, which drives generation, differs. In this work, we propose to analyze the spectrum of the denoiser Jacobian as a tool to characterize these differences. Across pre-trained denoising models, we observe that better generative performance is associated with larger Jacobian eigenvalues. Motivated by this, we introduce a regularization scheme that controls the Jacobian spectrum by training the denoiser on perturbed inputs, with perturbations suppressing or amplifying Jacobian responses. On ImageNet, we test whether directly modifying the Jacobian spectral properties leads to improved generations. Our findings suggest that denoisers benefit from both strengthening responses along data-relevant principal eigen-directions and suppressing the noisy, data-irrelevant ones. This establishes the denoiser Jacobian as a useful tool for identifying differences between generative denoising models.

1 INTRODUCTION

Generation-quality metrics describe outputs but offer limited insight into how learned denoisers differ. This work probes denoiser Jacobian spectra, finds larger eigenvalues associated with better generative performance, and tests perturbation-based spectral regularization on ImageNet.

1 INTRODUCTION

  • Generation-quality metrics characterize model outputs but provide limited insight into how the underlying denoising functions differ.The paper therefore examines denoiser properties to understand why one model synthesizes better samples than another.Source: §1 · p. 1
  • Better-performing pre-trained denoisers exhibit larger Jacobian eigenvalues, linking spectral properties to generative performance.The Jacobian describes how small changes in noisy inputs affect denoised outputs and captures learned dependencies.Source: §1 · p. 1
  • The authors propose controlling the Jacobian spectrum by perturbing denoiser inputs during training, without explicitly computing an eigendecomposition.Appropriate perturbations can suppress spurious spectral directions or amplify responses along principal eigenvectors.Source: §1 · p. 2
  • On ImageNet, spectral regularization improves generative metrics by strengthening principal-direction responses or suppressing noisy, data-irrelevant responses, with cumulative improvements when combined.The authors suggest effective denoisers benefit from both properties.Source: §1 · p. 2
  • The denoiser Jacobian is presented as a tool for identifying model differences and understanding how training shapes the learned denoising function.The introduction positions this framework as relevant to understanding existing methods and designing future objectives.Source: §1 · p. 2

2 RELATED WORK

Related work develops denoising models and tools for examining denoiser Jacobians, while prior regularizers often suppress Jacobian responses. This paper compares spectra across denoisers and explicitly targets increased responses.

Denoising generative models

  • Recent denoising-model advances combine algorithmic and architectural improvements, but all such models rely on a learned denoiser.The related-work discussion includes diffusion and flow-matching approaches to generating physical signals.Source: §2 · p. 2

Denoiser Jacobians and spectral analysis

  • Prior Jacobian studies examined learned data statistics or individual models, whereas this work compares denoisers to relate spectra to generation quality.Related analyses connect posterior uncertainty to Jacobians or show that generative-denoiser eigenvectors form a data-adaptive harmonic basis.Source: §2 · p. 2

Training regularization

  • Prior input perturbations address training–sampling mismatch or impose Jacobian nuclear-norm regularization; this work also explicitly regularizes for increased responses.Earlier work connected denoising and Jacobian regularization to local data geometry.Source: §2 · p. 3

3 SPECTRAL PROPERTIES OF DENOISER JACOBIANS

The paper studies denoiser Jacobians as local input–output maps and analyzes their spectra across pre-trained models. Better generative models show larger eigenvalues and richer top-component variability.

3.1 GENERATIVE DENOISING MODELS

  • Diffusion and flow-matching models train a denoiser to recover clean data from noisy samples, then generate by solving an ODE backward from Gaussian noise.The denoiser may use different schedules, prediction targets, and timestep weights; the paper focuses on fθ(xt, t).Source: §3.1 · p. 3

3.2 DENOISER JACOBIAN PROPERTIES

  • The denoiser Jacobian maps local changes in noisy input to changes in clean output and captures learned covariances at each noise level.The posterior-covariance identity motivates spectral analysis, though it applies only to the ideal denoiser.Source: §3.2 · p. 3
  • The analysis treats eigenvectors as principal components and eigenvalues as expansion along eigen-directions, while noting learned Jacobians are not perfectly symmetric or positive semidefinite.For image denoisers, eigenvectors represent maximal output changes obtained by perturbing the noisy input.Source: §3.2 · p. 3
  • The paper estimates top Jacobian eigen-directions with simultaneous power iteration and finite differences, using forward passes that can be parallelized across eigenvectors.The finite-difference approximation is presented as a fast and accurate alternative to autodifferentiation.Source: §3.2 · p. 4

3.3 SPECTRAL ANALYSIS OF PRE-TRAINED DENOISER JACOBIANS

  • Eigenvalue statistics follow the generative-performance ordering SiT-S < SiT-XL < SiT-XL+REPA, while denoising error is inconclusive.The larger eigenvalues appear notably during initial and middle denoising steps; Figure 1 reports that classifier-free guidance accentuates the model differences.Source: Figure 1 · p. 4, §3.3 · p. 5
  • Visualized top eigen-directions show less output variation for SiT-S than for larger models, whose perturbations produce more meaningful changes.Figure 2 shows the top ten eigenvectors for the same image at t = 0.4; the caption describes sharper, richer variability in better generative models.Source: §3.3 · p. 5, Figure 2 · p. 5

4 JACOBIAN REGULARIZATION IN TRAINING

The paper controls denoiser Jacobian responses by adding a perturbed-input denoising term, with the perturbation direction determining which responses are constrained or amplified. Residual perturbations increase responses along principal directions, while random or axis-aligned perturbations provide other forms of control.

Perturbed-input objective

  • A second denoising term on a slightly perturbed input makes the Jacobian explicit in training, with perturbation direction v determining the regularization.The first-order approximation describes how input perturbations change the denoiser output.Source: §4 · p. 5

Stochastic perturbation

  • For normally distributed random perturbations, the regularizer penalizes Jacobian expansion isotropically through the Frobenius norm.This choice constrains expansion across directions rather than targeting selected eigen-directions.Source: §4 · p. 6

Axis-aligned perturbation

  • Axis-aligned perturbations minimize variation along horizontal and vertical axes, producing square-shaped fits to the Gaussian components.Figure 3 compares this model with the unregularized baseline and residual perturbation.Source: §4 · p. 6, Figure 3 · p. 6

Residual perturbation

  • Residual regularization pushes the Jacobian response toward Jθr ≈ 1/δ r, using the residual as an eigenvector proxy to strengthen principal directions.Increasing response along the residual increases expansion disproportionately along principal eigenvectors.Source: §4 · pp. 6–7
  • In the 2D Gaussian experiment, residual regularization produces larger Jacobian eigenvalues and fewer samples between modes.The plotted grayscale encodes the maximum Jacobian eigenvalue at t = 0.5.Source: §4 · p. 6, Figure 3 · p. 6

5 IMAGENET EXPERIMENTS

ImageNet SiT experiments test whether input-perturbation regularizers change Jacobian spectra and generation quality. The combined regularizers achieve the best overall FID, while spectral analyses show that their effects depend on perturbation strength.

5.1 TRAINING SETUP

  • ImageNet experiments use SiT-S and an adaptive residual gain based on the Rayleigh quotient; residual masking retains the top-5% pixels.The adaptive gain addresses eigenvalue variation across timesteps, while masking avoids collapse of the perturbed input.Source: §5.1 · p. 7
  • The stochastic setting uses unit-variance Normal perturbations with τ = 1 or τ = 0.1, and the combined model adds both regularization terms.The combination uses the best-performing hyperparameters from the separate experiments.Source: §5.1 · p. 7

5.2 JACOBIAN REGULARIZATION CONTROLS SPECTRAL PROPERTIES

  • Residual gain increases mean, standard deviation, and maximum eigenvalues, with diminishing returns from 1/δ = 5 to 1/δ = 10.The increase is more pronounced at earlier timesteps and with classifier-free guidance; Figure 4 analyzes these spectra.Source: §5.2 · p. 8, Figure 4 · p. 8
  • With τ = 0.1, stochastic regularization suppresses random, data-irrelevant responses without changing principal components; τ = 1 over-constrains the spectrum.Figure 5 also shows the combined model has spectra similar to residual-only regularization.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 JACOBIAN REGULARIZATION IMPROVES GENERATIVE PERFORMANCE

  • The +vres model benefits less from classifier-free guidance, and this difference is reduced with 20 sampling steps.The evaluation varies sampling steps and guidance scales; fidelity and diversity metrics show small variations among models.Source: §5.3 · p. 9, Table 1 · p. 9
  • The combined regularizers achieve the best overall FID; residual regularization improves FID, while overly strong stochastic regularization worsens it.The moderate stochastic setting improves FID between the baseline and residual-regularized model.Source: §5.3 · p. 9
  • The authors interpret the combined model's result as complementary effects: strengthening data-relevant directions while attenuating weak, spurious variations.They report the combined model achieves the best overall FID.Source: §5.3 · p. 9

6 LIMITATIONS

Residual regularization has training-cost and guidance trade-offs, and the authors identify timestep-dependent gain as a possible alternative.

Limitations

  • Computing the residual requires a second denoiser forward pass, considerably slowing training relative to methods using one pass.This added pass is required because the residual is constructed from the initial model prediction.Source: §6 · p. 10
  • Residual-induced sensitivity weakens classifier-free guidance, though combining it with random perturbation resolves this issue.Source: §6 · p. 10
  • A timestep-dependent regularizer is proposed as a possible improvement, increasing eigenvalues only at timesteps where beneficial.Source: §6 · p. 10

7 CONCLUSION

The conclusion presents denoiser Jacobian spectra as a way to distinguish generative models and inform training. The appendix relates the Jacobian of an MMSE-optimal denoiser to posterior covariance.

7 CONCLUSION

  • The denoiser Jacobian provides a tool for identifying model differences, and its spectral properties can inform training objectives.The paper reports that better-performing models have larger principal Jacobian eigenvalues and that imposing desired denoiser properties during training can improve model efficiency and performance.Source: §7 · p. 10

A JACOBIAN AND COVARIANCE

  • For an MMSE-optimal denoiser, the Jacobian connects to the posterior covariance learned by the model.The appendix frames this relationship under an additive Gaussian noise model.Source: A JACOBIAN AND COVARIANCE · p. 13
  • For an ideal denoiser, the Jacobian is expected to be symmetric, with real, nonnegative eigenvalues.The passage grounds this expectation in the covariance matrix being positive semidefinite.Source: A JACOBIAN AND COVARIANCE · p. 13

B.1 SPECTRAL ANALYSIS

The appendix extends spectral analysis and residual regularization across model scales and architectures, and compares residual and stochastic perturbations. It also explains how masking can strengthen the spectral effect of a residual perturbation.

B.1 SPECTRAL ANALYSIS

  • With n = 20 and K = 20, the eigenvalue ordering still correlates with expected performance, while values are lower and the spectrum decays exponentially.The SiT-S residual-regularized model's increase is concentrated at the higher end of the top-100 spectrum across noise levels.Source: §B.1 · p. 14, Figure 7 · p. 14, Figure 8 · p. 14

B.2 EXTENSION TO OTHER MODELS

  • Residual regularization improves fidelity across evaluated models; SiT-B FID improves by ∼5 points from baseline to the best regularized model.The SiT-B model with 1/δ = 5 performs worse than with 1/δ = 3, indicating that the best gain target depends on the model.Source: §B.2 · pp. 14–15, Table 2 · p. 16

B.3 COMPARING RESIDUAL AND STOCHASTIC REGULARIZATION

  • Stochastic regularization with τ = 0.1 improves over baseline without a different measured Jacobian spectrum; residual perturbations produce larger corrections and may over-correct images.The authors suggest over-correction may explain smaller improvements with classifier-free guidance.Source: §B.3 · p. 15, Figure 11 · p. 16

C.1 EFFECT OF δ

  • Masking the residual preserves the same regularization while increasing the denoiser Jacobian’s top eigenvalues; it also multiplies target gain 1/δ by η > 1, which increases with masking.Source: §C.1 · pp. 17–18

C.2 TRAINING WITHOUT MASKING

Full-residual regularization performs similarly to masked regularization at 1/δ = 3 on SiT-S. The appendix also contrasts their spectra and Jacobian-residual products, and shows the poor generative outcome of maximizing all eigenvalues directly.

C.2 TRAINING WITHOUT MASKING

  • At the same 1/δ = 5, the masked model has larger eigenvalues, especially at lower timesteps, than the full-residual model.The full-residual model's mean eigenvalues are similar to those of a masked model using 1/δ = 3.Source: §C.2 · p. 18
  • Full-residual regularization performs similarly in FID to masked residual regularization with 1/δ = 3.The authors caution that eigenvalue statistics alone do not determine generative performance.Source: §C.2 · p. 18
  • The masked SiT-S Jacobian-residual product is qualitatively closer to the structured SiT-XL + REPA product than the blurrier SiT-S variants.The authors present this as support for examining denoiser properties beyond the spectrum.Source: §C.2 · p. 18
  • Directly maximizing all eigenvalues on a 2D Gaussian mixture makes the model overly sensitive and unable to learn the target distribution.The figure contrasts this outcome with residual perturbation, which pushes the maximum eigenvalue higher.Source: Figure 15 · p. 19

D.1 TRAINING

The appendix compares direct Jacobian regularization with the proposed regression-based approach, then contrasts expansion along random and residual directions. Direct regularization is unstable in the reported SiT-S experiment, while random-direction expansion increases all eigenvalues simultaneously.

D.1 TRAINING

  • Direct Jacobian regularization gives SiT-S an FID of ∼65, versus 59 for the baseline; larger tested τ values made the loss diverge.The authors hypothesize that direct regularization is more prone to training noise and instability.Source: §D.1 · p. 19

D.2 RESIDUAL VS. NORMAL EXPANSION

  • Maximizing expansion along a random Normal direction increases all eigenvalues, whereas the residual direction focuses on the largest eigenvalues.The authors hypothesize that focusing on principal directions helps avoid uncontrolled noise; the proposed perturbed-input regularization does not naturally yield the isotropic objective.Source: §D.2 · p. 20
  • The isotropic objective requires direct Jacobian probing, which the authors report performs subpar; an example shows the resulting model is generatively poor.The figure illustrates failure to learn a 2D Gaussian-mixture target under simultaneous eigenvalue maximization.Source: §D.2 · p. 20

E TOY EXPERIMENT

The toy experiment uses a small MLP denoiser and compares square and residual regularization settings. The largest Jacobian response is visualized through a 2×2 Jacobian and SVD.

E TOY EXPERIMENT

  • The toy denoiser is a three-layer, 128-dimensional SiLU MLP trained for 10,000 iterations to predict 2D x0.It uses 64-dimensional sinusoidal timestep embeddings, a zero-initialized output layer, learning rate 0.002, and batch size 512.Source: E TOY EXPERIMENT · p. 20
  • Square regularization samples four specified 2D directions, while residual regularization uses 1/δ = 10; both settings specify τ.The square setting uses δ = 1 and τ = 0.1; residual regularization uses τ = 1.Source: E TOY EXPERIMENT · p. 20
  • The largest Jacobian response is visualized by constructing a 2×2 Jacobian and applying SVD, with singular values treated as eigenvalues under a symmetry assumption.The experiments assume a symmetric Jacobian and use singular value and eigenvalue terms interchangeably.Source: E TOY EXPERIMENT · p. 20

F IMAGENET TRAINING

The ImageNet training section describes perturbation-based Jacobian regularization and its spectrum estimates, then visualizes changes in eigenvectors and denoiser features. The visualizations report sharper eigenvectors and earlier emergence of image structure in the regularized model.

F IMAGENET TRAINING

  • The Rayleigh quotient targets expansion parallel to the residual, unlike raw gain, which can overestimate it through orthogonal changes; it was more stable experimentally.For numerical stability, both measures are set to 1 for t > 0.85.Source: F IMAGENET TRAINING · p. 20, Figure 16 · p. 21
  • Increasing τ had limited effect for residual regularization, so τ = 1 was used; for stochastic perturbations, τ is the main control signal.Residual perturbations require tuning δ: larger values make regression too simple by including more of the target, while smaller values approach the base denoising task.Source: F IMAGENET TRAINING · p. 20
  • ImageNet training estimates a Rayleigh quotient from residual-direction responses and uses it to set the gain target; residual masks retain the top 5% of pixels.The training algorithm adds the regularization loss to the denoising loss and updates the spectrum estimate with EMA decay α.Source: F IMAGENET TRAINING · pp. 20–21, Figure 16 · p. 21
  • The stochastic-regularization response remains stationary throughout training rather than changing as denoising improves.Source: F IMAGENET TRAINING · p. 21

G VISUALIZING EIGENVECTORS

  • During SiT-B training, top eigenvectors become sharper and more varied, while residual regularization makes larger changes in top components than the baseline.The eigenvectors are decoded to image space after scaling latent vectors by 100 and removing the decoder’s zero-bias.Source: G VISUALIZING EIGENVECTORS · pp. 21–22, Figure 17 · p. 22
  • At t = 0.4, classifier-free guidance with w = 4.0 yields larger eigenvectors, while variations at t = 0.2 capture larger-scale image structures.Figure 18 also reports amplified eigenvector differences and larger eigenvalues with increased guidance.Source: G VISUALIZING EIGENVECTORS · p. 22, Figure 18 · p. 23

H DENOISER FEATURES

  • At t = 0.3, global image structures appear around block 3 in Jacobian-regularized SiT-B, versus block 5 in the baseline.The comparison uses L2-normalized block features projected to RGB with PCA.Source: H DENOISER FEATURES · p. 22, Figure 19 · p. 24

Related papers

Loading 2609.36210v1…