Attention Distillation: A Unified Approach to Visual Characteristics Transfer

1. Introduction

• Texture synthesis and style transfer remain challenging because local-patch methods miss global structure, while global feature matching can miss local detail.Traditional texture methods copy repeated local patterns, whereas newer deep-feature approaches match global distributions or nearest-neighbor patches.Source: §1 · pp. 1–2
• Diffusion self-attention keys and values can represent exemplar appearance, but domain gaps, accumulated matching errors, and residual-branch limits can weaken transfer.The paper describes unreliable target-query/source-key-value similarity across dissimilar images, errors propagating across layers, and bounded influence on target latents.Source: §1 · p. 2
• Attention Distillation defines the ideal stylization as attention between target queries and source keys and values, then minimizes its L1 distance from current attention output.The synthesized image is optimized through backpropagation in latent space, under the assumption that denoising-network self-attention features capture visual appearance.Source: §1 · p. 2
• The method compares differences across multiple layers to avoid error accumulation and gradually reduce target-query/source-key-value disparity.The paper states that this improves similarity calculations and final stylization.Source: §1 · p. 2
• The paper proposes attention distillation and guided sampling, reporting superior transfer results, faster synthesis, and broad visual-characteristic transfer applications.The loss is integrated into denoising as improved Classifier Guidance and can be combined with Classifier-Free Guidance and ControlNet.Source: §1 · p. 2

Texture synthesis

• Texture synthesis ranges from parametric and patch-sampling methods to deep feature, distribution-loss, and GAN approaches, with trade-offs in complex textures and large-scale structure.Self-Rectification uses pretrained diffusion networks and self-attention to refine non-stationary textures.Source: §2 · pp. 2–3

Neural style transfer

• Neural style transfer progresses from Gram-loss optimization to semantic feature matching, transformers, and diffusion-based style methods.The cited approaches include StyTR2, InST, StyleDiffusion, and StyleID.Source: §2 · p. 3

Appearance transfer

• Appearance transfer targets corresponding semantic regions, using trained GANs, DINO-ViT features, or diffusion cross-image attention.The diffusion approach realizes zero-shot appearance transfer by implicitly establishing semantic correspondences.Source: §2 · p. 3

Customized image generation

• Personalized text-to-image methods learn styles from reference images through fine-tuning or style-content separation, but fine-tuning can overfit or leak content.B-LoRA uses LoRA to implicitly separate style and content, while Pair Customization learns stylistic differences from one image pair.Source: §2 · p. 3
• Encoder-based customization can miss fine-grained reference textures, while plug-and-play methods preserve style consistency through self-attention feature sharing or swapping.The cited methods include StyleAligned, Visual Style Prompt, and RB-Modulation.Source: §2 · p. 3

3.1. Preliminaries

• KV-injection transfers exemplar appearance by replacing target-branch keys and values, but residual connections can limit reproduction of reference details.The denoising network's self-attention operates within a residual branch, so its injected information is subsequently influenced by the identity connection.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.2. Attention Distillation Loss

• Attention Distillation (AD) loss is the L1 distance between target self-attention output and an ideal output formed by reaggregating reference features with target queries.Backpropagation optimization lets information flow through both residual self-attention modules and the identity connection while updating the latent.Source: §3.2 · p. 4

3.3. Content-preserving Optimization

• Combining AD loss with a self-attention content loss enables style or appearance transfer while preserving the target image's content.The content loss is the L1 distance between target and reference queries; the optimized latent is decoded with the pretrained VAE.Source: §3.3 · p. 5

3.4. Attention Distillation Guided Sampling

• AD-guided sampling incorporates the loss into denoising updates, using Adam to manage guidance strength and allowing content loss to be added for structure preservation.The method approximates the AD-loss gradient at timestep t with the gradient at t−1, following an initial DDIM sampling step.Source: §3.4 · p. 5

3.5. Improved VAE Decoding

• For detail-intensive tasks, the method optionally fine-tunes the VAE decoder on the example image using L1 loss to improve reconstruction and novel-image decoding.The paper describes the pretrained VAE as perceptually lossy, particularly in high-frequency local details.Source: §3.5 · p. 5, Figure 5 · p. 6

4.1. Applications and Comparisons

• Style transfer preserves content structure while capturing coherent styles; appearance transfer avoids the oversaturation seen with Cross-Image Attention.The style comparison includes diffusion-based, traditional, and transformer-based methods.Source: §4.1 · p. 6, Figure 6 · p. 7
• In style-specific text-to-image generation, textual semantics align comparably to alternatives while reference-style coherence is notably better.The comparisons use the same prompts—“A deer,” “A rocket,” and “A piano”—for each style reference.Source: §4.1 · p. 6, Figure 7 · p. 7
• For controlled texture synthesis, results have comparable texture detail to GCD with smoother object edges and avoid its color-aliasing artifacts.Mask guidance restricts attention to corresponding regions; ControlNet conditioning is also used for text-to-image generation with modalities such as depth and canny edges.Source: §4.1 · p. 6, Figure 8 · p. 7
• Against Self-Rectification, the method adheres more closely to the original texture examples, while Self-Rectification produces smoother texture transitions.Source: §4.1 · p. 6, Figure 9 · p. 7
• Attention distillation guided sampling with MultiDiffusion expands 512×512 textures to 512×1536 and shows significant advantages over GCD and GPDM.Source: §4.1 · p. 6, Figure 10 · p. 8

4.2. Ablation Studies

• Varying content-loss weight λ changes abstraction in style transfer and produces a smooth identity transition in appearance transfer.The examples show differing levels of abstraction and appearance transition as λ varies.Source: §4.2 · pp. 7–8, Figure 11 · p. 8
• Adam-managed guidance better preserves exemplar features than manually set guidance strength; two optimization iterations per sampling timestep balance quality and computation time.Increasing iterations generally enhances quality but adds computation time.Source: §4.2 · p. 8, Figure 12 · p. 8

4.3. User Preference Study

• Across three transfer tasks, the method consistently outperforms selected alternatives in user preference, based on 1,500 responses from 50 participants.The study used 30 questions, with five questions for each of six competitors.Source: §4.3 · p. 8, Figure 13 · p. 8

• The approach covers style and appearance transfer, style-specific image generation, and texture synthesis.Source: §5 · p. 9
• Its key component is an attention distillation loss that measures the difference between ideal and current stylization and gradually modifies the synthesis.Source: §5 · p. 9
• Experiments validate the method's superiority across the presented tasks.Source: §5 · p. 9

• The content-preserving optimization algorithm uses style transfer as its example and applies attention distillation loss.Source: A. Algorithm · p. 12
• The guided-sampling algorithm uses style-specific text-to-image generation as its example, with Sampling(·) denoting a diffusion step from z_t to z_{t−1}.Source: A. Algorithm · p. 12
• The section denotes the VAE encoder and decoder as E(·) and D(·), the denoising network as ϵ_θ(·), and uses AdaIN to modulate feature variance and mean.Source: A. Algorithm · p. 12

• Experiments use SDXL for style-specific text-to-image generation and Stable Diffusion v1.5 for other tasks, extracting attention features from the last six U-Net self-attention layers.All experiments run on a single NVIDIA RTX 6000 Ada GPU.Source: B. Implementation Details · p. 12
• Style and appearance transfer use 512×512 images, 200 latent-optimization iterations, and content-loss weights of 0.25 and 0.2, respectively.Synthesis takes about 30 seconds per image.Source: B. Implementation Details · p. 12
• Style-specific text-to-image generation uses SDXL at 1024×1024 with 50 DDIM steps and two latent-optimization iterations per step.The process takes no more than 30 seconds.Source: B. Implementation Details · p. 12
• Controlled texture synthesis uses content loss to improve spatial alignment because U-Net features have low spatial resolution; texture expansion instead uses MultiDiffusion with three optimization iterations per sampling step.The texture-expansion setup uses 50 DDIM sampling steps.Source: B. Implementation Details · p. 12

Time efficiency

• The timing comparison excludes model loading, compilation, and image encoding or decoding; sampling uses 50 DDIM steps without classifier-free guidance.The cited table caption identifies Stable Diffusion v1.5, but the supplied passages do not provide its timing values.Source: C. Additional Experiments · p. 12, Table 1 · p. 12

Timestep choices

• Features from larger timesteps focus on coarse structures, while smaller-timestep features focus on fine details, supporting a linearly decreasing timestep during optimization.The experiment used the same Gaussian noise, Adam at 0.05, and 200 iterations.Source: C. Additional Experiments · p. 14

Optimization and sampling comparison

• Guided sampling significantly improves texture synthesis quality and speed over optimization on noisy latents, which produces noisier results and converges more slowly.The comparison uses the same Gaussian noise, 50 steps, and an Adam learning rate of 0.05.Source: C. Additional Experiments · pp. 14–15

Style-specific text-to-image settings

• In style-specific text-to-image generation, low learning rates or few iterations under-stylize results, while high values can weaken text-derived structure.The authors set two optimization iterations and a learning rate of 0.015 to balance image quality, text alignment, and time.Source: C. Additional Experiments · p. 15

Study procedure

• For each task, users compare the method against one of two selected competitors in randomized side-by-side results and choose the better one.References or prompts appear on the left, and evaluation criteria appear above the comparison.Source: D. Details of User Study · p. 15

Evaluation criteria

• Style-transfer judgments consider structural similarity to the content image and stylistic similarity to the style image.Appearance-transfer judgments consider structural similarity to the structure image and appearance similarity to the appearance image.Source: D. Details of User Study · p. 15
• Style-specific text-to-image judgments consider semantic alignment with the prompt and stylistic similarity to the style reference.Source: D. Details of User Study · p. 15

Texture expansion

• Texture expansion can occasionally produce oversaturated colors because the attention distillation loss does not explicitly constrain data-distribution consistency.The approach instead relies on the model’s understanding of the reference to reassemble visual elements.Source: E. Limitation and Discussion · p. 15
• When generated resolution exceeds the model’s training scope, aggregation may produce suboptimal results.Source: E. Limitation and Discussion · pp. 15–16

Style and appearance transfer

• In style and appearance transfer, substantially different image content can lead the model to incorrect semantic matches and negatively affect the output.Source: E. Limitation and Discussion · p. 16

Text-guided generation

• Additional figures present creative text-guided generation with style-specific guidance.Source: F. Additional Results · p. 16

Style transfer

• Additional style-transfer results use diverse content and style examples.Source: F. Additional Results · p. 16

Unconditioned texture synthesis

• The unconditioned texture-synthesis comparison includes optimization-based and sampling-based approaches against Self-Rectification [70], GCD, GPDM [16], and SWD [24].Source: F. Additional Results · p. 16

Texture synthesis and expansion

• Additional results show stationary and non-stationary texture synthesis and expansion using guided sampling.Source: F. Additional Results · p. 16

Texture expansion

• A 4096×4096 texture expansion is generated using a 512×512 example.Source: F. Additional Results · p. 16