PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward the Frontiers of Reliable Diffusion Sampling via Adversarial Sinkhorn Attention Guidance

Kwanyoung Kim

arXiv:2511.07499v1cs.CVcs.AI

TL;DR

Existing guidance methods often rely on heuristic perturbations, while the theoretically grounded choice of a perturbation for weakening outputs remains an open question. ASAG uses optimal transport and Sinkhorn to adversarially disrupt self-attention; the authors report improved generation quality across unconditional and conditional sampling and compatibility with ControlNet and IP-Adapter without additional training.

  • Problem

    Existing output-weakening guidance methods rely on heuristic perturbations, and what constitutes an optimal theoretically grounded perturbation remains an open question.

    Source: Introduction · p. 2

  • Method

    ASAG defines an attention cost from interactions between pixel embeddings and uses Sinkhorn within an optimal-transport framing to disrupt those interactions.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Results

    ASAG reportedly improves generation quality in unconditional and conditional sampling and achieves significant improvements over existing guidance methods with ControlNet and IP-Adapter.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    ASAG is compatible with ControlNet and IP-Adapter and improves over existing guidance methods without additional training.

    Source: Conclusion · p. 7

    The stated theorem setup specifies timestep-t query and key matrices Q_t,K_t ∈ R^{n×d}.

    Source: Proof of Theorem 1 · p. 10

Abstract

from arXiv · show

Diffusion models have demonstrated strong generative performance when using guidance methods such as classifier-free guidance (CFG), which enhance output quality by modifying the sampling trajectory. These methods typically improve a target output by intentionally degrading another, often the unconditional output, using heuristic perturbation functions such as identity mixing or blurred conditions. However, these approaches lack a principled foundation and rely on manually designed distortions. In this work, we propose Adversarial Sinkhorn Attention Guidance (ASAG), a novel method that reinterprets attention scores in diffusion models through the lens of optimal transport and intentionally disrupt the transport cost via Sinkhorn algorithm. Instead of naively corrupting the attention mechanism, ASAG injects an adversarial cost within self-attention layers to reduce pixel-wise similarity between queries and keys. This deliberate degradation weakens misleading attention alignments and leads to improved conditional and unconditional sample quality. ASAG shows consistent improvements in text-to-image diffusion, and enhances controllability and fidelity in downstream applications such as IP-Adapter and ControlNet. The method is lightweight, plug-and-play, and improves reliability without requiring any model retraining.

Introduction

Existing guidance methods often weaken outputs with heuristic perturbations, without a clear theoretical basis for choosing them. ASAG uses an optimal-transport perspective to adversarially perturb self-attention and is reported to improve generation quality across sampling settings and downstream frameworks.

Introduction

  • Earlier guidance methods used heuristic perturbations such as identity masking or Gaussian blur, leaving the optimal theoretically grounded perturbation as an open question.The introduction notes that these approaches lack a clear justification for why their perturbations consistently improve sample quality.Source: Introduction · p. 2
  • ASAG defines an attention cost from pixel-embedding interactions and uses Sinkhorn to disrupt those interactions rather than relying on heuristic perturbations.Its self-attention scores are reinterpreted through optimal transport; the resulting attention map is described as entropy-maximizing.Source: Introduction · p. 2
  • ASAG improves generation quality in unconditional and conditional sampling and reportedly outperforms other guidance approaches with ControlNet and IP-Adapter.The introduction describes its applicability across these downstream frameworks as broad and generalizable.Source: Introduction · p. 2
  • The paper presents an OT-based theoretical analysis and identifies its adversarial reinterpretation of OT for perturbing attention scores as a first, to the authors’ knowledge.The stated aim is to provide a rigorous foundation for the guidance method.Source: Introduction · p. 2

Preliminary

The preliminary section introduces diffusion sampling and guidance, then reviews optimal transport and its connection to self-attention. It motivates ASAG by noting that existing guidance perturbations lack a clear theoretical justification.

Diffusion Models

  • Diffusion models generate samples by reversing a forward noise process, with sampling proceeding sequentially from Gaussian noise.The reverse process is parameterized by a denoising model.Source: Diffusion Models · pp. 2–3

Guidance Sampling in Diffusion Models

  • Guidance uses conditional and unconditional or otherwise weakened score estimates to steer sampling, but the construction of undesirable paths has limited theoretical justification.The weaker estimate may be formed by dropping the condition in CFG or injecting an identity condition in PAG.Source: Guidance Sampling in Diffusion Models · p. 3

Optimal Transport and Sinkhorn-Knopp

  • Entropy-regularized optimal transport can be solved efficiently with Sinkhorn iterations, unlike direct OT, which incurs O(n^3 log n) computational cost.The transport polytope contains nonnegative matrices with prescribed row and column sums.Source: Optimal Transport and Sinkhorn-Knopp · p. 3

Connecting Self-Attention with Sinkhorn

  • Sinkhorn attention treats 1 − QK⊤ as a cost and optimizes a doubly stochastic plan favoring similar query-key pairs.The first Sinkhorn iteration corresponds exactly to SoftMax; the cited work reports more structured maps and empirical performance improvements.Source: Connecting Self-Attention with Sinkhorn · p. 3
  • Unlike heuristic identity or Gaussian-blur perturbations, ASAG minimizes a pixel-interaction-based attention cost to disrupt semantic interactions.The comparison describes ASAG as using the Sinkhorn algorithm to break semantic interactions.Source: Figure 2 · p. 4

Methods

The methods use Sinkhorn attention selectively in reverse to construct a degraded attention path for guidance. The adversarial plan has an entropy-maximizing limit, while finite regularization and few iterations are used for controlled, stable degradation.

Self- and Sinkhorn Attention in Diffusion Models

Adversarial Sinkhorn Attention Guidance

  • Adversarial Sinkhorn Attention minimizes interaction between noisy queries and keys, producing collapsed attention and disrupted interactions for the undesirable score estimate.Only a few Sinkhorn iterations are needed because degradation does not require precise convergence, resulting in minimal computational overhead.Source: Adversarial Sinkhorn Attention Guidance · p. 4
  • ASAG frames its perturbation as a constrained entropy-maximizing transport plan that disrupts semantic alignment, rather than an arbitrary direction in score space.The guidance energy is described as anchored to original attention while diverging from the entropy-maximizing adversarial trajectory.Source: Adversarial Sinkhorn Attention Guidance · pp. 4–5
  • As λ approaches zero, the adversarial Sinkhorn solution converges to a maximum-entropy plan; with uniform marginals, that plan is the uniform coupling.This limit diminishes semantic preferences and yields an increasingly unstructured attention map.Source: Adversarial Sinkhorn Attention Guidance · p. 5
  • The method uses a small but finite λ rather than the uniform-plan extreme, increasing entropy while preserving doubly stochastic structure and attention stability.Applying the theoretical extreme directly can reduce generation diversity and produce unstable behavior.Source: Adversarial Sinkhorn Attention Guidance · p. 5

Experiments

Experiments evaluate ASAG on unconditional and conditional diffusion generation, then test it with ControlNet and IP-Adapter. ASAG is reported to improve generation quality and preserve structure across these settings.

Experiments

  • Experiments compare guidance methods using SDXL and SD3, evaluate visual quality, diversity, text-image alignment, and human preference, and generate 30K MS-COCO samples.PAG and SEG use guidance scale 3.0; ASAG uses scale 1.5 and 25 sampling steps.Source: Experiments · p. 5, Results on Diffusion Generation · p. 5

Results on Diffusion Generation

  • ASAG outperforms other guidance methods in unconditional generation across evaluation metrics, with improved visual fidelity and diversity.The comparison uses 30K MS-COCO samples; qualitative outputs are also described as more aligned with vanilla results.Source: Table 1 · p. 5, Results on Diffusion Generation · p. 6
  • With CFG, ASAG achieves superior visual quality and text-image alignment, and state-of-the-art performance across human-preference metrics.The results include SD3 compatibility and qualitative preservation of the original CFG output's structural intent.Source: Table 1 · p. 5, Table 2 · p. 5, Results on Diffusion Generation · p. 6

Results on Downstream Tasks

Ablation Study

Ablations examine the transport cost, computation, and guidance scale. The similarity-minimizing Sinkhorn formulation balances semantic disruption with diversity and stability, using only a small reported inference-time increase.

Ablation Study

  • ASAG's cost explicitly minimizes query-key similarity, disrupting semantic alignment, unlike the similarity-maximizing cost variant.The similarity-maximizing variant also improves over vanilla sampling but incurs higher inference time due to more complex Sinkhorn optimization.Source: Ablation Study · p. 7, Table 3 · p. 7
  • The Sinkhorn formulation achieves semantic disruption comparable to a uniform transport plan while preserving better diversity and stability.The uniform plan is described as a theoretical upper bound on semantic disruption; it improves over baseline but reduces sample diversity.Source: Ablation Study · p. 7, Table 3 · p. 7
  • Two Sinkhorn iterations increase inference time by only +0.35 seconds while improving generation quality.Early stopping uses a convergence threshold, and two iterations suffice in most cases for stable transport plans.Source: Ablation Study · p. 7, Table 4 · p. 7
  • ASAG performance peaks at guidance scale s = 1.5, while overly large scales cause slight degradation.The scale is varied and performance measured across multiple metrics.Source: Ablation Study · p. 7, Figure 6 · p. 7

Conclusion

ASAG frames pixel-embedding interactions as optimal transport costs and disrupts them with the Sinkhorn algorithm. The paper reports state-of-the-art unconditional and conditional generation and improvements when ASAG is combined with ControlNet and IP-Adapter.

  • ASAG defines an attention cost from pixel-embedding interactions and disrupts those interactions with the Sinkhorn algorithm through an optimal-transport framing.Source: Conclusion · p. 7
  • The method demonstrates state-of-the-art performance across unconditional and conditional generation.Source: Conclusion · p. 7
  • With ControlNet and IP-Adapter, ASAG significantly improves over existing guidance methods without additional training.Source: Conclusion · p. 7
  • The authors suggest ASAG may generalize to a wide range of generative tasks and inform future attention-based guidance research.Source: Conclusion · p. 7

Toward the Frontiers of Reliable Diffusion Sampling

The supplementary document outlines additional implementation and evaluation details, theoretical proofs, qualitative results, and ablation studies.

Metrics and Implementation Detail

The implementation details specify sampling setups for unconditional and conditional evaluation, guidance-scale robustness, and additional prompt-based experiments.

Proof of Theorem 1

The theorem setup names the query and key matrices and an adversarial cost matrix, then invokes a closed-form Sinkhorn solution with scaling vectors chosen to enforce marginals.

Proof of Lemma 1

The proof shows that the uniform transport plan uniquely maximizes Shannon entropy under uniform marginals.

Proof of Lemma 1

  • Under uniform marginals, the uniform transport plan uniquely maximizes Shannon entropy over the transport polytope.Strict concavity makes the stationary point the unique global maximizer.Source: Proof of Lemma 1 · pp. 10–11
  • The stationarity condition factorizes the plan as P_ij = u_i v_j, and enforcing the marginals gives P_ij = 1/n^2.The proof defines u_i = e^(α_i−1) and v_j = e^β_j.Source: Proof of Lemma 1 · p. 11

Additional Qualitative Results

Across unconditional and conditional qualitative comparisons, ASAG improves visual quality while preserving structure and semantic intent; the results also examine guidance scales and use with or without CFG.

Comparison of Guidance Sampling with Our Method

Effect of Guidance Scale in ASAG

Comparison with and without CFG

  • Under ControlNet pose conditioning, ASAG produces high-quality generations without CFG and further boosts quality when CFG is applied, unlike PAG without CFG.The comparison reports that PAG fails to deliver meaningful improvements without CFG.Source: Additional Qualitative Results · p. 11, Figure 12 · p. 17
Loading 2511.07499v1…