PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, Seungryong Kim

arXiv:2403.17377v2cs.CVcs.AIcs.LG

TL;DR

CG and CFG can be unavailable for unconditional generation and downstream tasks, while also requiring added training or external modules. PAG creates structurally degraded predictions by perturbing self-attention and guides sampling away from them; the paper reports improved quality across conditional, unconditional, and downstream settings.

  • Problem

    CG and CFG require additional training or external modules, can reduce sample diversity, and are unavailable in unconditional generation and some downstream tasks.

    Source: §1 · p. 2

  • Method

    PAG perturbs diffusion U-Net self-attention to create degraded predictions and guides denoising away from them without additional training or external modules.

    Source: Perturbation guidance · p. 6, Perturbed self-attention · p. 9, §6 · p. 14

  • Results

    PAG improves sample quality in conditional and unconditional settings and enhances downstream tasks, including inverse problems and ControlNet with empty prompts.

    Source: §1 · pp. 2–3, §6 · pp. 14–15

  • Takeaways & Limitations

    PAG offers a guidance approach for diffusion generation and restoration tasks where text prompts or conventional guidance are unavailable.

    Source: §1 · p. 3, §6 · p. 15

    At high guidance scales, PAG can produce over-saturation, and each generation step requires two forward paths.

    Source: F Limitation and Future Works · p. 57

Abstract

from arXiv · show

Recent studies have demonstrated that diffusion models are capable of generating high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG is designed to progressively enhance the structure of samples throughout the denoising process. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, by considering the self-attention mechanisms' ability to capture structural information, and guiding the denoising process away from these degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios. Moreover, PAG significantly improves the baseline performance in various downstream tasks where existing guidances such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and image restoration such as inpainting and deblurring.

1 Introduction

Classifier guidance (CG) and classifier-free guidance (CFG) improve diffusion sample quality but require additional training or external modules, can reduce diversity, and are unavailable for unconditional generation. PAG uses perturbed self-attention predictions to guide sampling, improving quality in conditional and unconditional settings and downstream tasks.

1 Introduction

  • CG and CFG improve sample quality but require additional training or external modules, can reduce diversity, and are unavailable in unconditional generation.When these methods are unavailable, unconditional generation can be sub-optimal despite its usefulness for avoiding costly annotations and providing general priors for inverse problems.Source: §1 · p. 2
  • PAG guides denoising away from degraded samples created by replacing diffusion U-Net self-attention maps with an identity matrix.The method uses self-attention's capacity to capture structural information and requires neither additional training nor external modules.Source: §1 · pp. 2–3
  • PAG improves sample quality in ADM and Stable Diffusion across conditional and unconditional settings, and further improves results when combined with CFG.The paper also reports gains on inverse problems and ControlNet with empty prompts, where CFG is unusable because conditions are absent.Source: §1 · p. 3

2 Related Work

Prior work spans diffusion-model variants and sampling guidance methods. The section reviews DDIM and latent diffusion models alongside classifier, classifier-free, self-attention, and conditioning-focused guidance.

Related Work

  • DDIM speeds sampling through a non-Markovian process, while latent diffusion models operate in compressed latent space to balance efficiency and synthesis quality.Diffusion models are described as strong in image generation and distribution estimation.Source: §2 · p. 3
  • Classifier guidance increases fidelity at the expense of diversity, while classifier-free guidance models an implicit classifier to achieve similar effects.Both are presented as central advances in sampling guidance.Source: §2 · p. 3
  • Self-attention guidance improves unconditional sample quality using adversarial blurring, while other methods focus on conditioning or image editing.SAG guides sampling using noise predicted from blurred and original samples.Source: §2 · p. 3

3 Preliminaries

The preliminaries describe diffusion as forward noise addition followed by iterative denoising, then introduce classifier guidance and classifier-free guidance as ways to steer conditional sampling. CFG combines conditional and unconditional noise estimates through a guidance signal.

Diffusion models

  • Forward diffusion adds Gaussian noise to an image to produce a noisy sample, and the denoising network is trained to predict that noise.The objective is a squared error between the sampled noise and the network's prediction.Source: §3 · pp. 3–4
  • Sampling iteratively estimates each reverse step from the current noisy sample and the model's noise prediction to produce a clean image.The text relates the noise estimate to the score of the noisy-sample distribution.Source: §3 · p. 4

Classifier-free guidance

  • CFG combines conditional and unconditional noise estimates, using their difference as a guidance signal that steers samples toward condition c.The null token ϕ supplies the unconditional estimate; Fig. 2 visualizes the signal and the corresponding difference in generated structure.Source: Classifier-free guidance · pp. 4–5, Figure 2 · p. 5

4 PAG: Perturbed-Attention Guidance

PAG guides denoising by contrasting the pretrained model’s prediction with a perturbed prediction that represents undesirable structure. It perturbs self-attention to degrade structure while preserving appearance, then uses the resulting difference to strengthen structural features during sampling.

4.1 Self-rectifying sampling with implicit discriminator

  • The implicit discriminator compares desirable and undesirable noisy-sample distributions, and its generator-loss gradient yields the guidance used in sampling.The paper defines the labels as imaginary labels for desirable and undesirable samples.Source: Perturbation guidance · p. 6
  • PAG combines the original and perturbed noise predictions, steering denoising away from predictions associated with undesirable samples.The framework treats the pretrained score as the desirable prediction and a perturbed forward pass as the undesirable one.Source: Perturbation guidance · pp. 5–6
  • PAG extends perturbed forward-pass guidance beyond CFG’s dropped-label implementation, and self-attention perturbation is intended to limit deviation from the original sample.Directly perturbing the input or condition can cause out-of-distribution guidance; the paper also warns that perturbing appearance features may distort samples.Source: Connections to CFG · p. 6, §4.2 · p. 7, Perturbed self-attention · p. 8

4.2 Perturbing self-attention of U-Net diffusion model

  • Perturbed self-attention replaces a selected self-attention map with an identity matrix, removing structural information while retaining appearance features.The paper distinguishes query-key similarities as structural and value features as appearance information.Source: Perturbed self-attention · pp. 8–9

4.3 Analysis on PAG

  • The guidance difference highlights missing features in perturbed predictions, and adding it to the original prediction strengthens sample structure.The visualization identifies eyes, nose, and tongue as features missed when global structure is degraded.Source: Figure 3 · p. 7, §4.3 · p. 9

5 Experiments

Experiments evaluate PAG with pretrained diffusion models across pixel-level and latent generation, prompt-guided synthesis, image restoration, and spatially conditioned generation. The reported results include improved ADM metrics, qualitative gains for Stable Diffusion, and improved restoration and ControlNet samples.

5.1 Experimental and Implementation Details

  • Experiments use pretrained ADM, Stable Diffusion 1.5, and SDXL models, with the same evaluation metrics as ADM.Additional experimental settings are provided in Appendix A.Source: §5.1 · p. 10

5.2 Pixel-Level Diffusion Models

  • ADM with PAG outperforms ADM and SAG by a large margin in FID and IS on ImageNet 256×256 conditional and unconditional generation.The study evaluates 50K samples; the reported Improved Recall and Precision patterns reflect a fidelity–diversity trade-off.Source: §5.2 · p. 10

5.3 Latent Diffusion Models

5.4 Downstream Tasks

  • For PSLD restoration on FFHQ, PAG outperforms the baseline on FID across box inpainting, ×8 super-resolution, Gaussian deblur, and motion deblur, and mostly on LPIPS.Fig. 8 qualitatively shows fewer artifacts and less blurriness in restorations with PAG.Source: Inverse problems · p. 13, Table 4 · p. 13, Figure 8 · p. 13
  • PAG improves ControlNet samples conditioned only on pose or depth, making them more realistic, less artifact-prone, and semantically coherent.The paper notes that this setting can support tailored training datasets and imaginative work without detailed prompts.Source: ControlNet · pp. 13–14, Figure 9 · p. 14

5.5 Ablation Studies

  • The authors report ablations of self-attention perturbation and guidance scales in Appendix D.PAG parallelizes its two denoising passes by batching duplicated U-Net inputs, giving computational cost nearly identical to CFG.Source: §5.5 · p. 14

6 Conclusion and Discussion

The paper introduces perturbation guidance, with PAG steering sampling away from a perturbed forward pass to improve image generation without additional training or external modules. It reports benefits across conditional and unconditional settings and downstream image restoration.

6 Conclusion and Discussion

  • PAG improves sample quality in both conditional and unconditional settings without additional training or external modules.It replaces the diffusion U-Net’s self-attention map with an identity matrix and guides generation away from structural degradation.Source: §6 · pp. 14–15
  • The conclusion presents PAG as a versatile guidance approach and argues that unconditional diffusion can reduce reliance on text prompts and CFG.The authors state that their exploration enriches understanding of sampling guidance and diffusion models.Source: §6 · p. 15

Appendix

  • The appendix documents implementations for ADM, Stable Diffusion, and other applications, plus qualitative results, ablations, CFG/PAG dynamics, and limitations.The appendix also covers human evaluations, downstream tasks, and computational details.Source: Appendix · p. 20

A.1 Experiments on ADM

  • ADM experiments replace Softmax(QtKT) with an identity matrix in selected self-attention layers, using DDPM sampling for 250 steps.The selected layers differ between unconditional and conditional models.Source: Quantitative results · p. 20

A.2 Experiments on Stable Diffusion

  • Stable Diffusion experiments apply perturbed self-attention in the U-Net middle layer, with guidance scales and sampling steps varying by experiment.Quantitative experiments use DDIM 50-step sampling; qualitative experiments include 25-, 50-, and 200-step settings.Source: §A.2 · p. 21, Qualitative results · p. 21
  • When combining CFG and PAG, the denoising estimate adds their separately weighted guidance deltas to the conditional estimate.The method computes conditional, unconditional, and perturbed conditional estimates simultaneously in the denoising U-Net.Source: Combination of CFG and PAG · p. 21

A.3 Experiments with PSLD

This block documents PSLD and ControlNet experiments, ablation settings, and Stable Diffusion sampling costs. The reported cost measurements show nearly identical memory use and similar denoising speeds for CFG and PAG.

A.3 Experiments with PSLD

  • PSLD experiments use Stable Diffusion v1.5 and apply perturbed self-attention to the same input-block layer on FFHQ and ImageNet.For gradient updates, the experiments use unconditional rather than guided scores when predicting z0.Source: §A.3 · p. 22, Table 5 · p. 22

A.4 Experiments with ControlNet

  • ControlNet pose and depth experiments use PAG guidance scales of 2.5 and 1.0, respectively, with 50-step DDIM sampling.Perturbed self-attention is applied in the U-Net middle layer.Source: §A.4 · p. 22

A.5 Ablation Study

  • The guidance-scale and perturbation-strategy ablations use 5,000 ADM ImageNet unconditional samples with 25-step DDIM sampling.The scale ablation consistently uses identity-matrix replacement.Source: §A.5 · pp. 22–23

A.6 Computational Cost

  • PAG and CFG have nearly identical Stable Diffusion GPU memory use, while both denoising speeds are slightly slower than sampling without guidance.The measurements used one NVIDIA GeForce RTX 3090 GPU and one batch.Source: §A.6 · p. 23

B.1 ADM Results

The figures provide uncurated ADM ImageNet 256 samples for unconditional and conditional generation, with and without PAG. Each image set places unguided samples above PAG-guided samples.

B.1 ADM Results

  • The unconditional ADM figures compare unguided top-row samples with PAG-guided bottom-row samples.The captions specify guidance scale s = 3.0 and perturbed layers i13, i14, i16, and m1.Source: Figure 10 · p. 24, Figure 11 · p. 25
  • Across these figures, compare the uncurated sample sets with and without PAG under the stated ADM settings.The captions identify the row ordering but do not state a comparative outcome.Source: Figure 10 · p. 24, Figure 12 · p. 26
  • The conditional ADM figures use the same top-row versus bottom-row comparison of samples without guidance and with PAG.The captions specify guidance scale s = 3.0 and perturbed layers i13, i14, i16, and m1.Source: Figure 12 · p. 26, Figure 13 · p. 27

B.2 Stable Diffusion Results

The additional results cover PAG with PSLD and DPS restoration, prompt-free Stable Diffusion image tasks, and text-to-3D generation. Reported findings include better restoration quality and added detail in text-to-3D comparisons.

B.2 Stable Diffusion Results

B.3 PSLD Results

C.1 Diffusion Posterior Sampling

  • PAG outperforms the DPS baseline on FID except in ×8 super-resolution, where FID is comparable.Table 8 reports quantitative results; the authors say the hard-degradation super-resolution task may emphasize FID over LPIPS.Source: §C.1 · p. 36, Table 8 · p. 36

C.2 Stable Diffusion Super-Resolution and Inpainting

  • PAG produces sharper and more realistic Stable Diffusion super-resolution and inpainting results without requiring text prompts.The super-resolution figure describes clearer edges and finer details, including an empty-prompt example; the inpainting figure notes improved realism and fewer artifacts.Source: §C.2 · p. 39, Figure 30 · p. 40, Figure 31 · p. 41

C.3 Text-to-3D

  • Combining PAG with CFG yields text-to-3D results with enhanced details and textures compared with CFG alone.The experiments use the Dreamfusion implementation provided by Threestudio.Source: §C.3 · p. 42, Figure 32 · p. 42

C.4 Human Evaluation

Human evaluation favors PAG in unconditional generation and text-to-image synthesis; ablations examine guidance scale and self-attention perturbations, with identity replacement producing the strongest reported perturbation results.

C.4 Human Evaluation

  • Most user-study questions preferred PAG for unconditional samples, while samples using CFG plus PAG were frequently rated higher than CFG-only samples.The study included 60 participants, with 10 questions per task across three tasks.Source: §C.4 · p. 43

D.1 Guidance scale

  • PAG achieved its best FID at guidance scale 1.0 and its best IS at scale 2.0 in ADM unconditional sampling.As scale increases, Stable Diffusion samples also show more natural structure and fewer artifacts, though excessive scale can smooth textures and slightly saturate images.Source: §D.1 · p. 44, Figure 35 · p. 44

D.2 Perturbation on Self-Attention Maps

  • 32.34 FID for identity-matrix replacement compared with 40.20 for random masking and 39.49 for off-diagonal random masking.Qualitative comparisons report that identity replacement yields plausible structures and semantics, whereas other perturbations can produce over-smoothed textures or artifacts.Source: Figure 36 · p. 45, §D.2 · p. 45, Table 9 · p. 45

D.3 Layer Selection

Layer choice affects PAG outcomes across diffusion models, while identity attention is motivated as a way to degrade structure while preserving appearance; the section also contrasts PAG with CFG and discusses optional training for stability.

D.3 Layer Selection

  • Perturbing deeper ADM layers, particularly near the U-Net bottleneck, generally performs better than perturbing shallower layers, while the Stable Diffusion mid-block gives its best reported FID.Some layer combinations improve ADM but not Stable Diffusion, and fixed layers across timesteps outperform random layer selection.Source: §D.3 · p. 47, Figure 38 · p. 47, Figure 39 · p. 48

E.1 Theoretical Insights on Using Identity Matrix as Perturbation

  • The self-attention forward pass is framed as pattern retrieval, with learned query-key relationships and value contents contributing contextual relationships and content patterns.Identity replacement removes relationships while retaining learned value features, producing a structurally undesirable but in-domain distribution that preserves local texture.Source: §E.1 · pp. 50–51
  • Identity replacement is effective, but the authors leave better perturbations for different tasks and models as future work.The proposed perturbation selectively preserves appearance information while degrading structure.Source: §E.1 · p. 51

E.2 Further Analysis on CFG and PAG

  • CFG implemented with separately trained conditional and unconditional models does not work properly, whereas PAG can generate perturbed predictions by manipulating U-Net self-attention without an external model or additional training.The authors suggest that the separate-model CFG predictions do not align, making the guidance signal ineffective.Source: §E.2 · p. 51, CFG with separately trained models · p. 52
  • Adding training to PSA could further improve its handling of out-of-distribution samples and robustness to hyperparameter settings.The proposed fine-tuning approach uses a switching input for self-attention and is described as less computationally costly than training a new unconditional CFG model.Source: Additional training for stability · pp. 52–53

E.3 Complementarity between CFG and PAG

PAG and CFG affect different stages of denoising: PAG’s signal strengthens later, while CFG is strongest early; their combination is reported to improve outcomes. The section also compares PAG’s formulation and operating characteristics with SAG.

E.3 Complementarity between CFG and PAG

  • PAG’s predicted-epsilon difference grows as self-attention strengthens during sampling, while CFG’s difference is strongest early and diminishes later.The text associates CFG’s early signal with prompt reliance and PAG’s later signal with the image taking shape.Source: §E.3 · pp. 53–54, Figure 43 · p. 53, Figure 44 · p. 54
  • Applying CFG for the first 15 of 25 sampling steps yields results comparable to applying it throughout all 25 steps.PAG continues to provide guidance into the middle and later stages of sampling.Source: §E.3 · pp. 54–55, Figure 45 · p. 54
  • Combining CFG and PAG is reported to outperform CFG alone in qualitative comparisons, Stable Diffusion quantitative results, and human evaluation.The authors attribute the combination’s coverage across the sampling process to using both self-attention and cross-attention.Source: §E.3 · p. 55

E.4 Comparison with SAG

  • PAG’s framework permits perturbing internal representations, and the authors describe SAG as one instance of the broader perturbation framework.SAG’s adversarial blurring uses a blurred sample and an imaginary regressor; the framework allows other perturbation processes.Source: Generalizability · p. 55, §E.4 · p. 55
  • SAG requires multiple blur and mask settings, while PAG requires selecting the layer for perturbed self-attention; SAG also shows artifacts at high guidance scales, unlike PAG’s plausible shapes and details.The SAG comparison reports visible adversarial-mask boundaries and high-frequency artifacts as scale increases.Source: Hyperparameter count and sensitivity… · pp. 55, 57, Figure 46 · p. 56
  • PAG can process perturbed and regular self-attention within one batch and one denoising-network evaluation, so it could theoretically be more than twice as fast as SAG with swift concurrent GPU computation.Source: Inference speed · p. 57

F Limitation and Future Works

The authors identify high-scale over-saturation and two forward paths per generation step as limitations; the changelog records additions in the camera-ready update.

F Limitation and Future Works

  • At high guidance scales, PAG can over-saturate results, and each generation step requires two forward paths.The authors call for careful scale calibration and identify reducing computational overhead as future work.Source: F Limitation and Future Works · p. 57

G Changelog

  • The July 6, 2025 camera-ready update added SDXL examples, theoretical motivation for identity-matrix perturbation, and discussion of subsequent works.Source: Jul 06, 2025 · p. 57

Related papers

Loading 2403.17377v2…