PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Sample Quality of Diffusion Models Using Self-Attention Guidance

Susung Hong, Gyuseong Lee, Wooseok Jang, Seungryong Kim

arXiv:2210.00939v6cs.CVcs.AIcs.LG

TL;DR

Traditional diffusion guidance improves image quality but depends on external conditions and, in some cases, additional training complexity. This paper introduces generalized internal-information guidance, including blur guidance and SAG, which uses self-attention to blur attended regions during sampling. SAG improves quality across several diffusion models and further improves results when combined with conventional guidance, while adding cost relative to no guidance.

  • Problem

    Traditional diffusion guidance improves samples but depends on external class or text conditions and can require additional training procedures.

  • Method

    The paper generalizes guidance to internal intermediate information, proposing blur guidance and SAG, which adversarially blurs self-attended regions during reverse diffusion.

  • Results

    SAG improves sample quality across ADM, IDDPM, Stable Diffusion, and DiT, and combines with existing guidance methods for further improvement.

  • Takeaways & Limitations

    Condition- and training-free SAG provides a broadly applicable guidance method that is orthogonal to existing guidance methods.

  • Takeaways & Limitations

    SAG has nearly the same memory and time consumption as classifier-free guidance but costs more than sampling without guidance because it adds an extra step.

Abstract

from arXiv · show

Denoising diffusion models (DDMs) have attracted attention for their exceptional generation quality and diversity. This success is largely attributed to the use of class- or text-conditional diffusion guidance methods, such as classifier and classifier-free guidance. In this paper, we present a more comprehensive perspective that goes beyond the traditional guidance methods. From this generalized perspective, we introduce novel condition- and training-free strategies to enhance the quality of generated images. As a simple solution, blur guidance improves the suitability of intermediate samples for their fine-scale information and structures, enabling diffusion models to generate higher quality samples with a moderate guidance scale. Improving upon this, Self-Attention Guidance (SAG) uses the intermediate self-attention maps of diffusion models to enhance their stability and efficacy. Specifically, SAG adversarially blurs only the regions that diffusion models attend to at each iteration and guides them accordingly. Our experimental results show that our SAG improves the performance of various diffusion models, including ADM, IDDPM, Stable Diffusion, and DiT. Moreover, combining SAG with conventional guidance methods leads to further improvement.

1. Introduction

Diffusion guidance improves image synthesis but traditional methods depend on external conditions and added training complexity. The paper generalizes guidance using intermediate internal information and proposes SAG, which improves quality across diffusion models and combines with existing guidance.

  • Motivation: Traditional classifier and classifier-free guidance improve samples but require external class or text conditions, with classifier guidance also requiring an additional classifier.Classifier-free guidance adds label-dropping complexity during training.
  • Generalized guidance: The paper generalizes diffusion guidance to use information within intermediate samples, supporting guidance with or without external conditions.This formulation broadens guidance beyond traditional condition-dependent approaches.
  • Blur guidance: Blur guidance uses information removed by Gaussian blur to improve sample quality at moderate guidance scales.At large scales, global blur can create structural ambiguity across entire regions and noisy results.
  • Self-Attention Guidance: SAG adversarially blurs salient regions identified by self-attention maps and guides the diffusion model with the resulting residual information.It uses internal attention during reverse diffusion without external information or additional training.
  • Experiments: Experiments evaluate SAG with ADM, IDDPM, Stable Diffusion, and DiT, showing broad applicability and further gains when combined with classifier or classifier-free guidance.The authors also report ablation studies supporting their design choices.
  • Contributions: The contributions include a condition-free guidance formulation, SAG, flexible combination with existing methods, and extensive ablation studies.These contributions target improved sample quality without external conditions or additional fine-tuning.

2. Related Work

Related work covers diffusion sampling, conventional class- and text-based guidance, self-attention in generative models, and internal diffusion representations. SAG instead uses diffusion models’ self-attention information to provide training- and condition-free guidance.

  • Denoising diffusion models: Diffusion models generate images through iterative denoising and are studied for their high sampling quality and diversity.DDPM established an iterative process that progressively denoises toward an image.
  • Sampling guidance: Classifier guidance uses a trained classifier, while classifier-free guidance avoids an additional classifier but relies on conditional training procedures.Related work also extends these ideas to text-to-image generation with CLIP guidance and classifier-free guidance.
  • Sampling guidance: Existing guidance approaches are limited because they do not apply to unlabeled datasets and require additional training procedures.These limitations motivate guidance based on internal diffusion information.
  • Self-attention in generative models: Self-attention is used in generative models because it can encode global context, motivating its incorporation into computer vision and generation.The related work discusses self-attention in Transformer-based models and GANs.
  • Internal representations: Prior work also uses diffusion-model representations for tasks such as semantic segmentation, including label-efficient strategies based on U-Net representations.These studies treat diffusion representations as useful beyond image generation.
  • Self-Attention Guidance: SAG compares with classifier-free guidance by extracting internal self-attention information rather than using external class information, making it training- and condition-free.The comparison emphasizes the different information sources used for guidance.

3. Preliminaries

DDPMs generate images through iterative forward and reverse diffusion processes, while classifier and classifier-free guidance modify denoising to improve conditional generation. Self-attention maps provide an internal representation used by diffusion models during generation.

  • Denoising diffusion probabilistic models: DDPMs recover images from white noise through an iterative denoising process governed by forward and reverse transitions.The forward process adds noise using a variance schedule, while the reverse process samples progressively less-noisy states.
  • Denoising diffusion probabilistic models: The model predicts an intermediate reconstruction of the original image x_0 from the noisy state x_t and its predicted noise.This reconstruction uses the cumulative noise schedule and the network prediction ϵ_θ(x_t,t).
  • Classifier guidance and classifier-free guidance: Classifier guidance adds a classifier-gradient term to the conditional diffusion prediction, scaled by sσ_t, to steer sampling toward a class label.The classifier estimates p(c|x_t), and its gradient modifies the guided output.
  • Classifier guidance and classifier-free guidance: Classifier-free guidance combines conditional and unconditional noise predictions, avoiding an additional classifier while retaining externally conditioned guidance.Its formulation uses the difference between conditional and unconditional predictions, scaled by the guidance parameter.
  • Self-attention in diffusion models: Diffusion models use self-attention at intermediate layers of U-Net or Transformer architectures to process feature maps across spatial positions.For feature maps with spatial dimensions H×W and channel dimension C, attention is computed across multiple heads.

4. Generalizing Diffusion Guidance

The paper generalizes diffusion guidance by allowing conditions to come from internal sample information, external inputs, or both. This formulation supports guidance through an imaginary regressor and extends guidance to unconditional models.

  • Generalized diffusion guidance: The generalized framework treats the condition h_t as internal information, an external condition, or both, alongside a perturbed sample x̄_t.An imaginary regressor p_im(h_t|x̄_t) is assumed to predict the generalized condition from the perturbed sample.
  • Generalized diffusion guidance: Guidance modifies the conditional diffusion prediction using the gradient of an imaginary regressor with respect to the perturbed sample.The gradient ∇_{x̄_t} log p_im(h_t|x̄_t) steers generated samples toward compatibility with h_t.
  • Generalized diffusion guidance: At guidance scale s=5.0, blur guidance produces noisier images than SAG because global blur introduces structural ambiguity across entire regions.The authors motivate a more adaptive approach that captures finer and more relevant information during reverse diffusion.
  • Generalized diffusion guidance: Bayes’ rule expresses the imaginary-regressor score through the difference between conditional and unconditional true scores.The resulting score is substituted into the generalized guidance equation to obtain the final guidance form.
  • Generalized diffusion guidance: Classifier-free guidance is a special case of the generalized formulation when x̄_t=x_t and h_t is the class condition c.Under this setting, the imaginary regressor reduces to the implicit classifier used in classifier-free guidance.
  • Generalized diffusion guidance: The framework enables guidance for unconditional models by making them self-conditional on visual information from intermediate reverse-process samples.This extends guidance beyond models that require external labels or captions.

5. Utilizing the Self-Attention Map to Improve Sample Quality

The paper develops blur guidance and then Self-Attention Guidance (SAG), which selectively conceals salient, self-attended regions and uses the concealed information to guide diffusion. SAG preserves intact regions to reduce global-blur ambiguity and improves sample quality across diffusion settings.

  • 5.1. Blur Guidance for Diffusion Models: Blur guidance uses Gaussian-blurred intermediate reconstructions to guide diffusion toward information removed from fine-scale details.The original prediction is compared against a re-noised blurred prediction, with their difference supplying guidance.
  • 5.2. Self-Attention Guidance for Diffusion Models: Experiments report improved quality for ADM and IDDPM, fewer artifacts in ADM samples, and more favorable human judgments for Stable Diffusion samples.The evaluations span ImageNet-trained models and Stable Diffusion comparisons using matched random seeds or paired samples.
  • 5.1. Blur Guidance for Diffusion Models: Large guidance scales make global blur produce noisy results because blurring entire regions creates structural ambiguity.This motivates an adaptive method that captures finer, more relevant information during the reverse process.
  • 5.2. Self-Attention Guidance for Diffusion Models: Self-attention masks overlap with high-frequency regions, linking the method’s selected patches to details diffusion models should elaborate during generation.The frequency masks are computed after generation, whereas self-attention masks are accumulated throughout the reverse process.
  • 5.2. Self-Attention Guidance for Diffusion Models: SAG adversarially blurs patches selected by self-attention maps, then uses the concealed information to guide the reverse process.The method targets salient patches rather than applying blur globally.
  • 5.2. Self-Attention Guidance for Diffusion Models: SAG retains intact input regions, preventing the structural ambiguity of global blur and helping maintain outputs closer to the original input.This design supports stability at larger guidance scales.

6. Experiments

Experiments across unconditional, conditional, and text-to-image settings show that SAG improves sample quality across several diffusion models and can complement conventional guidance. Ablations identify effective masking and scale choices, while revealing additional computational cost and a fidelity–diversity trade-off.

  • Experimental Results: SAG improves unconditional ADM results on ImageNet 256×256, LSUN Cat, and LSUN Horse for FID, sFID, and IS, while lowering recall.The authors associate lower recall with a possible fidelity–diversity trade-off.
  • Experimental Results: SAG improves FID for unconditional IDDPM and produces more visually favorable or realistic Stable Diffusion samples in human evaluation.The Stable Diffusion comparison used 500 image pairs with identical random seeds and empty prompts.
  • Compatibility with Existing Guidance: Combining SAG with CG improves FID and precision, while SAG alone achieves the best sFID in the reported ADM comparison.Results on DiT also show that samples guided by CFG benefit from SAG.
  • Ablation Studies: The self-attention masking strategy outperforms alternative masking schemes, whereas global masking and DINO self-attention masking perform worse on reported FID and IS metrics.The comparison used 10k samples on ADM with other schemes masking an equivalent 40% of image pixels.
  • Ablation Studies: On ADM, s = 0.1 gives the best FID, sFID, and Inception Score, while s = 0.3 gives the best precision; negative or overly large scales harm sample quality.The tested scales were −0.1, 0.1, 0.2, 0.3, and 0.4.
  • Computational Cost: SAG has nearly the same memory and time consumption as CFG but costs more than using no guidance because it adds an additional sampling step.The reported overhead from blurring and masking is negligible relative to CFG.

7. Conclusion

The conclusion presents SAG as a general guidance formulation that uses internal diffusion-model information rather than external conditions. It describes the method as condition- and training-free, applicable across several diffusion models, and complementary to existing guidance.

  • 7. Conclusion: The paper formulates diffusion guidance using internal information within diffusion models to synthesize higher-quality images.The conclusion frames this as a novel and general formulation of guidance.
  • 7. Conclusion: SAG is condition- and training-free and applies to ADM, IDDPM, Stable Diffusion, and DiT.The conclusion reports quality improvement and artifact reduction through self-conditioning.
  • 7. Conclusion: The experiments support SAG’s effectiveness and its orthogonality to existing guidance methods.The supplied conclusion does not specify a numerical result.

B. Additional Implementation Details

The implementation details describe sampling SAG from pretrained diffusion models, including an efficient selective-blurring procedure and an efficient formulation for combining SAG with CFG. Additional analyses examine attention-map structure and frequency content.

  • Implementation: Experiments sample pretrained ADM, IDDPM, Stable Diffusion v1.4, and DiT models using two servers equipped with eight RTX 3090 GPUs each.The implementation uses publicly available model weights and PyTorch implementations.
  • SAG Sampling: Selective blurring estimates x̂0, applies complementary masks to the original and blurred reconstructions, aggregates them, and re-noises the result using predicted noise.This procedure produces the same bxt as the main-paper formulation.
  • Combining SAG with CFG: When combining SAG with CFG, the guided noise prediction uses separate scales for CFG and SAG, with c denoting the text prompt.The efficient formulation avoids the four feedforward steps required by a naive combination.
  • Hyperparameters: The supplementary implementation reports hyperparameter settings and notes that the blur parameter σ depends on input resolution.These settings are used as constants when other parameters are ablated.
  • Attention Analysis: Intermediate self-attention maps in ADM capture generated-image structure, and high-attention patches contain more high-frequency details.The analyses visualize attention at 8×8, 16×16, and 32×32 resolutions and compare attention with frequency and semantic structure.

C.2. Additional ablation studies

Additional ablations examine the self-attention masking threshold and extraction layer, while supplementary visualizations compare attention masks with generated objects and frequency structure. The reported best threshold is 1.0, and final-layer attention gives the strongest FID and IS results.

  • Masking Threshold: The masking-threshold ablation tests ψ values of 0.7, 1.0, and 1.3 on 10k ADM samples, with the highest metrics at ψ = 1.0.The experiment uses ADM trained on ImageNet 128×128.
  • Attention-Map Layer: Attention-map extraction from the evaluated layers consistently improves over baseline, while the final layer yields the best FID and IS results.The tested locations include final self-attention layers at each resolution and the bottleneck layer.

C.3. Qualitative results

The section presents random SAG samples from ADM across ImageNet and LSUN datasets, alongside an example of the human-evaluation question format.

  • C.3. Qualitative results: Random SAG samples are shown from ADM trained on ImageNet at 128×128 resolution.
  • C.3. Qualitative results: Additional random SAG samples are shown for ADM trained on LSUN Cats.
  • C.3. Qualitative results: The evaluation example does not reveal which row was sampled with the proposed method.

D. Human Evaluation Protocol

The human evaluation compares paired Stable Diffusion samples generated with identical seeds, asking participants to choose the higher-quality group without cherry-picking.

  • D. Human Evaluation Protocol: The study generated 500 empty-prompt pairs with and without SAG, using a SAG scale of 1.0 for SAG samples.
  • D. Human Evaluation Protocol: Each pair used the same seed for the SAG and non-SAG samples, enabling direct comparison.
  • D. Human Evaluation Protocol: Fifty participants compared two groups of four samples and selected the group with higher image quality.
  • D. Human Evaluation Protocol: The sample pairs were neither cherry-picked nor filtered.

E. Limitations & Future Work

The discussion identifies diversity, novelty, computational cost, and model-type suitability as boundaries and directions for SAG. It also notes that guidance strength can moderate its effects.

  • E. Limitations & Future Work: Higher self-conditioning can make images more appealing to humans while potentially reducing diversity and novelty.
  • E. Limitations & Future Work: Controlling the guidance scale can moderate SAG’s impact, supporting applications where its effects need adjustment.
  • E. Limitations & Future Work: SAG requires twice as many feedforward steps, creating a computational-cost challenge shared with classifier-free guidance.
  • E. Limitations & Future Work: Distilling guidance into diffusion models is proposed as a possible way to reduce SAG and classifier-free guidance costs without sacrificing quality.
  • E. Limitations & Future Work: Self-attention-based guidance may be more suitable for discrete diffusion models that directly model token probabilities.
Loading 2210.00939v6…