Source-linked AI summary
Denoising Diffusion Generative Models Secretly Calculate Attentions
Farzan Haddadi, Leila Monfared, Ebrahim Rezaii, Mohammadreza Malek-Mohammadi, Pejman Zakalvand, Narges Mokhtari
TL;DR
Diffusion denoising is analyzed as an attention-like operation, motivating a simpler generator based on local interpolation among dataset signals. The approach removes the inner denoising autoencoder, achieves an 8.5× FFHQ generation speedup, and reports competitive dataset-dependent FID results.
Problem
Diffusion systems require lengthy training and iterative generation, while their relationship to attention and autoencoders is not fully used to simplify computation.
Method
The paper derives fixed-σ diffusion solutions as attention combinations, relates autoencoders to attention, and generates images through local interpolation of nearby latent dataset signals.
Results
0.02 second versus 0.17 second on FFHQ yields an 8.5× speedup; FID is better on MNIST, comparable on Fashion-MNIST and CelebA, and lower on FFHQ.
Takeaways & Limitations
Image generation can use attention-based local interpolation without training an inner denoising autoencoder, with substantially lower computation.
Abstract
from arXiv · showhide
Denoising diffusion models are the dominant architecture for image generation, whereas most natural language generation and modeling are primarily handled by well-known transformer architectures employing attention mechanism. Here, we show that diffusion models also inherently use an attention mechanism very similar to that of transformers. Therefore, attention emerges as a universal machine learning principle, based on a general training objective. We also show similarities in basic functional principle of auto-encoders and attention-based models. These equivalences allows us to interchange these designs based on practical requirements. As an example, we can reformulate the diffusion framework to reduce the lengthy training process and computation-intensive image generation. Using this approach, a simplified algorithm is proposed for image generation which is based on attention mechanism. Results show that the attention-based implementation achieves comparable performance with significantly less effort and computational resources.
A. Background
Diffusion trains a denoising network by progressively adding Gaussian noise to dataset images and learning to recover the originals. Prior finite-dataset analyses derive score-based reverse generation and extend it to CNN denoisers.
- Forward diffusion adds Gaussian noise to the true image x0 over hundreds or thousands of steps.
- At the final forward stage, the signal becomes noise-only, while the network is trained to recover x0 from each noisy image xt.
- Finite-dataset analysis models Gaussian noise as a mixture of Gaussian distributions centered on dataset points.
- Prior work solves a flow-matching continuous-time model backward in time for signal generation and defines its score function from the mixture density.
- The prior analysis extends to CNN denoisers and develops a theory of combinatorial creativity for CNN diffusion systems.
C. Diffusion is Attention
Under finite-dataset training, the paper formulates denoising as convex optimization and shows that its fixed-noise optimum is an attention vector. The derivation also connects attention with softmax, while each diffusion step uses a different noise level.
- The analysis starts from finite-dataset training and seeks relations among diffusion models, transformers, and autoencoders.
- Training minimizes MSE between the estimated signal and each dataset signal for noisy inputs y = si + n.
- With infinitely many updates for fixed si and y, the network learns to map y to si; across signals, the fixed-y problem becomes an average loss.
- For fixed y, the average-loss formulation is a Quadratic Programming convex optimization problem with nonnegative coefficients.
- For sufficiently small σ, nearby signals dominate the convex combination, under an approximately constant signal-norm condition.
- The fixed-σ optimum is the attention vector, providing a mathematical basis for attention as the optimum recovery of signals in Gaussian noise.
- This attention result describes one diffusion step, whereas forward-process steps use distinct increasing σ values.
- With σ = 1 and one-hot signal vectors, the exponential form becomes softmax, establishing equivalence between softmax and attention layers.
D. Convergence
Reverse diffusion can be understood as sequential attention with decreasing σ: it first approaches the mean and then converges toward a nearby dense dataset region. This convergence motivates bypassing the iterative trajectory.
- In reverse generation, a shared denoising network applies attention-like updates while σ decreases across steps.
- When σ is large, distance differences have little effect and the output sequence approaches the mean of nearby vectors.
- When σ becomes small, the process quickly converges to the nearest signal with contributions from nearby signals, producing a sparse representation.
- The simulation decreases σt from 1 to 0.37 using σt = 0.96 σt+1, while practical network limitations prevent σ from reaching zero.
- Overall, diffusion initially approaches the distribution mean and later reaches the vicinity of a dense dataset region.
- This convergence enables generation by directly using a convex combination of nearby dataset signals, avoiding the full convergence process.
E. Finite Training
Finite training limits the denoiser’s coverage and optimization accuracy, introducing effective output noise. The resulting higher effective σ prevents fully sparse solutions and leaves outputs as combinations of many signals.
- The analysis assumes infinite training to cover every y and reach the ground-truth closed-form optimum.
- In practice, limited silicon, time, and energy usually restrict training to an acceptable-performance point.
- With finite training, the network sees only finite adjacent inputs and is expected to interpolate its learned behavior when the training set is sufficiently dense.
- Finite training errors in the denoiser and autoencoder reconstruction can be modeled as additive output noise.
- The effective higher σ increases intermittent signals, prevents a fully sparse solution, and leaves the final output as a convex combination of many signals.
III. Auto Encoders
The paper presents auto-encoders, attention systems, and diffusion systems as closely related structures with interchangeable computational roles. This equivalence motivates replacing computationally intensive training with attention-based alternatives.
- III. Auto Encoders: Auto-encoder and attention structures differ mainly in computational-load placement: training for auto-encoders versus test-time product calculations for attention.This trade-off motivates simpler attention mechanisms as alternatives to complex diffusion and auto-encoder training.
- III. Auto Encoders: Auto-encoders, attention systems, and diffusion systems share a strong functional resemblance.The paper frames this as a unity among three important AI system types.
- III. Auto Encoders: Under Gaussian-noise augmentation and perfect training, an auto-encoder’s optimization has the same solution as an attention system with fixed σ.The attention parameter σ can be tuned to match structural parameters of the auto-encoder.
- III. Auto Encoders: The equivalence is two-sided: auto-encoders can in principle replace attention systems, including compressed-state linear attention.The paper relates causal linear attention to recursive auto-encoder structure.
B. Auto-encoders and Manifolds
The paper interprets a trained auto-encoder as estimating a low-dimensional data manifold. Locally, code-preserving directions form a null space, while the auto-encoder projects off-manifold inputs toward the learned manifold.
- B. Auto-encoders and Manifolds: An auto-encoder with m-dimensional input/output and a k-dimensional code layer estimates a k-dimensional manifold representing dataset signals and meaningful augmentations.This follows the manifold hypothesis and assumes the auto-encoder is sufficiently trained.
- B. Auto-encoders and Manifolds: Because the code space is k-dimensional, any k + 1 code changes are linearly dependent and yield an input direction with no output effect.The resulting direction defines a local null-space structure.
- B. Auto-encoders and Manifolds: The local null space N(s) contains nearby inputs whose encoded representation equals E(s), within a δ-neighborhood.The null space is local and effective only near s.
- B. Auto-encoders and Manifolds: Displacements along the local null space leave the code and output unchanged, whereas manifold directions alter both.The auto-encoder tunes the null space to fit neighboring data points.
- B. Auto-encoders and Manifolds: Assuming a sufficiently dense dataset, an auto-encoder follows the data manifold and moves an off-manifold input along local null spaces before reaching it orthogonally.The construction assumes no specific weight-sharing structure, although convolutional networks can better estimate certain manifolds through embedded invariances.
C. Attention and manifold
The paper argues that attention also projects nearby inputs onto a low-dimensional affine structure formed by neighboring data signals. This provides a geometric basis for its claimed resemblance to auto-encoders and diffusion systems.
- C. Attention and manifold: Attention projects data onto a low-dimensional manifold-like structure, supporting its proposed similarity to auto-encoders and diffusion systems.The paper presents this as part of a shared geometric interpretation.
- C. Attention and manifold: When y lies near the data manifold and σ is sufficiently small, only the k nearest signals contribute effectively to the attention sum.The attention coefficients are then determined locally by those neighboring signals.
- C. Attention and manifold: The nearby signals form a k − 1-dimensional affine subspace, onto which y is orthogonally projected at h.The projection geometry is illustrated in Fig. 5.
- C. Attention and manifold: The attention output is a convex combination of neighboring signals, remains on their affine subspace, and is unchanged along a local orthogonal null space.This gives attention a local projection behavior analogous to the auto-encoder construction.
D. Auto-encoders resemble attention
The paper connects diffusion, auto-encoder, and attention computations through their shared projection behavior. It then proposes using attention over signal latents to avoid training an inner denoising auto-encoder and shorten generation.
- D. Auto-encoders resemble attention: Attention acting on a point near the data manifold projects it onto the manifold and maps it to a convex-hull point of nearby signals.The paper identifies this projection behavior as shared with auto-encoders.
- D. Auto-encoders resemble attention: The equivalence between diffusion and attention extends to both the outer and inner auto-encoders used in conventional diffusion systems.The paper therefore treats the three structures as alternatives depending on practical requirements.
- D. Auto-encoders resemble attention: Attention requires no training but concentrates computation at test time through product calculations over a long context of vectors.Auto-encoders make the opposite trade-off by concentrating computation in training.
IV. Tree Cross Attention
The paper addresses the quadratic cost of attention over large image datasets by organizing similar data in a tree, reducing neighbor search to logarithmic depth.
- Computational challenge: Attention requires comparing the query with every dataset image, while nearby signals could suffice when the attention parameter σ is small.Finding those neighbors directly still requires distances to all images, equivalent to inner-product computation.
- Computational challenge: A data structure reduces attention computation by storing reusable information instead of recomputing all query–image dot products.The direct calculation has O(|D|) complexity.
- Tree construction: The proposed tree orders images by similarity, routing each new image through progressively more similar branches.Each node is an image, and branching decisions compare similarity at each tree level.
- Tree cross attention: A balanced tree clusters nearby images into branches, allowing neighbor search with O(log |D|) inner-product calculations.The method is called tree cross attention when base attention vectors are constant.
V. Proposed Generation Algorithm
The proposed generator replaces iterative reverse diffusion with attention-based interpolation among nearby latent representations, followed by decoding and limited autoencoder refinement.
- Latent-space generation: The method exploits the approximately sparse diffusion solution by starting synthesis from a dataset image and attending to nearby latent signals.Attention is treated as approximately constant within a neighborhood of the image latent.
- Manifold refinement: The generated latent is decoded and the resulting image can be passed through the outer autoencoder a few times to better fit the data manifold.This refinement addresses slight displacement from the manifold.
- Complexity reduction: The algorithm eliminates training a denoiser autoencoder and performs only local neighbor attention during generation.It retains training of an outer autoencoder that projects input images into a low-dimensional latent space.
- Latent-space generation: A random latent query is matched to K nearest neighbors, whose attention-weighted interpolation forms a new latent representation.The interpolated latent is decoded into an image, with nearby latent representations providing semantically related intermediate representations.
- Spatial attention: Spatial attention assigns pixel-level contributions by comparing corresponding query and neighbor depth vectors.The resulting spatial attention map controls each pixel’s contribution during latent-space sampling.
VI. Simulation Results
Experiments evaluate the attention-based generator on MNIST, Fashion-MNIST, FFHQ, and CelebA using outer-VAE training and qualitative image comparisons. The method generates FFHQ images 8.5× faster, with better MNIST FID, comparable Fashion-MNIST and CelebA FID, and lower FFHQ FID than the baseline.
- Experimental setup: The experiments use MNIST, Fashion-MNIST, FFHQ, and CelebA at resolutions from 28 × 28 to 128 × 128.Training uses a P100 GPU, Adam optimization, and an outer VAE loss combining L1, perceptual, and adversarial components.
- Generation speed: 0.17 second versus 0.02 second per FFHQ image yields an 8.5× generation speedup over diffusion.The proposed method uses local interpolation instead of iterative reverse diffusion.
- Qualitative results: Qualitative comparisons on FFHQ, MNIST, and Fashion-MNIST show generated images that retain query similarity while incorporating features from nearest neighbors.Figure 8 displays the query, generated image, and three retrieved neighbors in each row.
- Qualitative results: Generated samples are reported as natural and good quality across examples, including 128×128 CelebA samples.Additional samples are shown for FFHQ and CelebA.
- FID comparison: The proposed model uses fewer parameters, achieves better MNIST FID, comparable Fashion-MNIST and CelebA FID, and lower FFHQ FID than baseline diffusion.Table III reports the cross-dataset FID comparison.