PaperlayerView exact revision on arXiv

Source-linked AI summary

What the DAAM: Interpreting Stable Diffusion Using Cross Attention

Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, Ferhan Ture

arXiv:2210.04885v5cs.CVcs.CL

TL;DR

Large diffusion models generate high-quality images but remain poorly understood. The paper introduces DAAM, which aggregates cross-attention to attribute image pixels to words, and finds plausible attribution quality alongside syntactic interaction patterns and feature-entanglement phenomena.

  • Problem

    Large-scale diffusion models lack interpretability analyses, leaving their word-to-image synthesis dynamics undercharacterized.

  • Method

    DAAM upscales and aggregates cross-attention scores in Stable Diffusion to create pixel-level attribution maps for prompt words.

  • Results

    DAAM achieves 58.9–64.8 mIoU for noun segmentation, 3.4–4.2 MOS for interpretable words, and reveals directional syntactic interactions plus entanglement involving cohyponyms and adjectives.

  • Takeaways & Limitations

    DAAM provides a visuolinguistic lens for interpreting diffusion models through word–pixel attribution, syntax-to-pixel interactions, and semantic failure phenomena.

  • Takeaways & Limitations

    Adjectival entanglement is not necessarily undesirable because its significance depends on the model’s use case.

Abstract

from arXiv · show

Large-scale diffusion neural networks represent a substantial milestone in text-to-image generation, but they remain poorly understood, lacking interpretability analyses. In this paper, we perform a text-image attribution analysis on Stable Diffusion, a recently open-sourced model. To produce pixel-level attribution maps, we upscale and aggregate cross-attention word-pixel scores in the denoising subnetwork, naming our method DAAM. We evaluate its correctness by testing its semantic segmentation ability on nouns, as well as its generalized attribution quality on all parts of speech, rated by humans. We then apply DAAM to study the role of syntax in the pixel space, characterizing head--dependent heat map interaction patterns for ten common dependency relations. Finally, we study several semantic phenomena using DAAM, with a focus on feature entanglement, where we find that cohyponyms worsen generation quality and descriptive adjectives attend too broadly. To our knowledge, we are the first to interpret large diffusion models from a visuolinguistic perspective, which enables future lines of research. Our code is at https://github.com/castorini/daam.

1 Introduction

The paper introduces DAAM to interpret how words influence pixels in Stable Diffusion, then evaluates attribution quality and uses it to analyze syntax and feature entanglement.

  • 1 Introduction: Stable Diffusion enables white-box analysis of large diffusion models, whose image-synthesis dynamics were previously undercharacterized.The model was open-sourced as a 1.1-billion-parameter latent diffusion model pretrained and fine-tuned on LAION 5-billion.
  • 1 Introduction: DAAM combines cross-attention maps to produce two-dimensional attribution maps showing which image regions each prompt word influences.The method is named diffusion attentive attribution maps and illustrated for words such as “monkey,” “hat,” and “walking.”
  • 1 Introduction: 58.9–64.8 mIoU measures noun-attribution alignment with annotated image segments, while human ratings reach 3.4–4.2 MOS for interpretable words.These evaluations cover semantic segmentation and generalized attribution across parts of speech.
  • 1 Introduction: Across ten syntactic relations, dependent and head maps show directional subsumption or substantial overlap, with coreferent words indicating identity.The paper conjectures that verb maps contain subject maps because verbs contextualize subjects and their surroundings.
  • 1 Introduction: Cohyponyms can produce only one of two prompted objects, while descriptive adjectives attend broadly and alter the entire image, indicating feature entanglement.The paper links greater heat-map overlap with cohyponym status and generation incorrectness.
  • 1 Introduction: The paper contributes an evaluated attribution method, a mapping of syntax to pixels, and analyses of entangled features associated with adjectives and cohyponyms.These contributions target interpretation of diffusion models from a visuolinguistic perspective.

2 Our Approach

The approach interprets Stable Diffusion by aggregating cross-attention scores between prompt words and image locations into pixel-level attribution maps. DAAM aligns multiscale denoising-network attention spatially, aggregates it across model dimensions, and optionally thresholds the resulting maps.

  • Diffusion preliminaries: Stable Diffusion combines a language model, VAE, and time-conditional U-Net to iteratively denoise random latent vectors into images conditioned on text.The U-Net performs the denoising process, while the VAE decodes the final denoised latent into an image.
  • Cross-attention attribution: Cross-attention scores connect each prompt word to intermediate image coordinates, which locally correspond to affected patches in the final image.Different layers produce heat maps at different spatial scales because deeper representations are coarser.
  • Diffusion Attentive Attribution Maps: DAAM upsamples intermediate attention arrays with bicubic interpolation, then sums them across heads, layers, and time steps into a single heat map.Aggregating across all time steps and layers is reported as necessary for the method.
  • Diffusion Attentive Attribution Maps: The resulting positive, scale-normalized values form soft attribution maps in which larger values indicate greater word-to-pixel attribution.Because the scores are positive and scale normalized, summation preserves their linear scale for visualization.
  • Diffusion Attentive Attribution Maps: DAAM can convert soft attribution maps into binary influence maps by thresholding pixel values with τ ∈[0, 1].The thresholding operation uses an indicator function to mark pixels as influenced or not.

3 Attribution Analyses

DAAM is evaluated as an unsupervised word-attribution method by comparing its maps with segmentation masks and human judgments across interpretable parts of speech. The analyses find strong segmentation performance, generally plausible attribution maps, and a caveat involving overlapping explanations between words.

  • Object Attribution: DAAM segmentation masks are compared with annotated noun segments using mIoU, alongside supervised and unsupervised baselines.The evaluation uses COCO captions, noun-swapped prompts, and thresholds τ ∈{0.3, 0.4, 0.5}.
  • Object Attribution: DAAM outperforms COCO-supervised models by 22–28 points in mIoU∞ but underperforms them by 20 points in mIoU80.CLIPSeg achieves the highest mIoU∞ overall, while DAAM largely outperforms the unsupervised baselines by 4.4–29 points.
  • Object Attribution: 57.9–64.8 mIoU80 makes DAAM a strong segmentation baseline, with τ = 0.4 performing best and layer and timestep contributions supporting the result.Threshold variation changes mIoU by 3–6 points, indicating limited sensitivity to τ.
  • Generalized Attribution: Human raters judge DAAM maps for adjectives, verbs, nouns, proper nouns, numerals, and adverbs using a five-point attribution-quality scale.The study samples 2,800 word–prompt pairs, presents generated images with heat maps, and uses three raters per example.
  • Generalized Attribution: Mean opinion scores are close to or slightly above good for adjectives, verbs, nouns, and proper nouns, while numerals and adverbs are closer to fair.Fair–excellent ratings exceed 80% for numerals and adverbs and 90% for the other interpretable parts of speech.
  • Generalized Attribution: A qualitative limitation is that different heat maps may plausibly explain the same word, making comparisons less meaningful.The example “quickly” could conceivably explain “running” as well.

4 Visuosyntactic Analysis

DAAM reveals systematic relationships between syntactic structure and spatial attribution in generated images, including directional dominance, overlap, and feature entanglement.

  • Setup: The study analyzes 8,000 head–dependent pairs from 1,000 COCO prompts across the ten most common dependency relations.Maps were binarized to quantify head–dependent overlap and dominance.
  • Measures: mIoU measures map similarity, while the difference between mIoD and mIoH measures which word’s map dominates.If mIoD exceeds mIoH, the head contains the dependent more; the reverse indicates dependent dominance.
  • Non-dominant relations: Noun compounds show no dominance, while punctuation and determiners attend broadly and also lack dominance.The broad attention of punctuation and articles is associated with their limited semantic meaning.
  • Coordination: Noun pairs linked by “and” have lower overlap at 38.7 mIoU, yet retain substantial overlap attributed partly to feature entanglement.The reduced overlap is associated with visual separation between objects such as cats and dogs.
  • Dominant relations: Verbs dominate subject and object maps by 12–29 points, whereas descriptive adjectives dominate modified nouns by approximately 15 points.Coreferent pairs instead have the highest overlap at 66.6 mIoU, indicating attention to the same referent.

5 Visuosemantic Analyses

DAAM is used to investigate semantic entanglement, showing that cohyponyms reduce two-object generation quality and that adjectives can influence image regions beyond their modified nouns.

  • 5.1 Cohyponym Entanglement: Cohyp​onym prompts are tested using WordNet’s hierarchy over COCO’s visual-object categories, with correctness requiring both prompted nouns to appear.The setup includes 28 categories with at least one cohyponym across 16 hypernym groups.
  • 5.1 Cohyponym Entanglement: 61% accuracy for non-cohyponyms versus 52% for cohyponyms supports the hypothesis that semantically similar nouns worsen generation quality.The difference is statistically significant at the 99% level.
  • 5.1 Cohyponym Entanglement: Cohyponym pairs have 46.7 mIoU overlap versus 22.9 for non-cohyponyms, while incorrectness and cohyponymy independently increase overlap.Generation accuracy is 71.7–77.5% for low-overlap pairs but 9.8–36% for high-overlap pairs.
  • 5.2 Adjectival Entanglement: Adjective maps can attend broadly across the image, with adjective changes altering backgrounds as well as the modified objects.Rusty, metallic, and wooden shovels change the shed, while bumpy, smooth, and spiky balls alter the surrounding ground or grass.
  • 5.2 Adjectival Entanglement: Color-adjective prompts also show background changes, suggesting quantifiable adjectival entanglement through measurable hue outside the cropped object.The authors caution that such entanglement is not necessarily unwanted and depends on the model’s use case.

6 Related Work and Future Directions

The paper positions DAAM as a visuolinguistic attribution analysis for generative diffusion models and identifies future applications in parsing, disentanglement, and segmentation.

  • Related work: Unlike prior linguistic analyses using textual perturbation or attention visualization, this work attributes generated diffusion-model pixels to words.It frames the contribution as analysis of how syntax and semantics relate to generative image output.
  • Future directions: A proposed future direction is testing Stable Diffusion’s unsupervised parsing ability with syntactic–geometric probes.The proposed probes are analogous to methods used to study syntactic structure in BERT.
  • Future directions: The paper connects DAAM to efforts that disentangle style and spelling in generative models, proposing disentanglement of cohyponym and adjectival concepts.This future work is intended to improve generative quality.
  • Future directions: DAAM could potentially provide encoder features for a semantic-segmentation pipeline.The paper presents this as a possible extension related to unsupervised segmentation.

7 Conclusions

The paper interprets diffusion models through word–pixel cross-attention, evaluates DAAM’s attribution behavior, and uses it to study syntax and feature entanglement.

  • Conclusions: DAAM interprets diffusion models by analyzing word–pixel cross-attention maps and is evaluated through quantitative segmentation and qualitative generalized attribution studies.The method is then applied to syntactic visual interactions and feature-entanglement hypotheses.
  • Conclusions: The analysis finds that some head maps inappropriately subsume their dependents’ maps.This conclusion links syntactic relations to spatial interactions in generated images.

A.1 Object Attribution

The paper details the generation and segmentation setups used to evaluate noun attribution, then tests whether aggregating across diffusion time steps and layers is necessary.

  • Generation setup: Images used Stable Diffusion 2.0 at 512 by 512 pixels with 30 inference steps, default 7.5 classifier guidance, and the DPM solver.Offensive images were automatically filtered, and offensive prompts were avoided.
  • Segmentation process: Ground-truth masks marked each area corresponding to a present noun using Photoshop selection and brush tools.Two trained annotators created masks for 200 image–prompt pairs.
  • Layer and time step ablation: The ablation studies restricted DAAM’s summation over time steps and layer resolution from either side of the full ranges.These studies tested whether summing across all time steps and layers, as in Eqn. 6, was necessary.
  • Layer and time step ablation: Figure 11 compares using the first or last n time steps and the equivalent layer-resolution restrictions.
  • Layer and time step ablation: All time steps and layers contribute positively to segmentation quality.

A.2 Generalized Attribution

Generalized attribution was evaluated through crowdsourced judgments of how relevant a highlighted image area was to a red word, with safeguards for response quality.

  • Annotation process: Three unique master-level Amazon MTurk annotators rated each red word’s relevance to the highlighted image area.The interface also allowed annotators to mark poor images or missing words.
  • Annotation quality: Workers who failed punctuation checks were removed, while fewer than 2% of tasks were rejected overall.Punctuation tokens served as attention checks because they were too abstract to be represented in the image.

B Supplements for Syntactic Analyses

The syntactic analyses use overlap measures to characterize interactions between dependent and head DAAM maps, with figures documenting annotation, response, and overlap statistics.

  • Measures of overlap: mIoU, mIoD, and mIoH measure overlap between binarized dependent and head maps.mIoU captures overall overlap, while mIoD and mIoH assess intersection relative to the dependent and head maps.
  • Measures of overlap: High mIoU indicates substantial map overlap, whereas contrasting mIoD and mIoH values indicate which map occupies more of the other.
  • Annotation: Figure 12 presents the annotation UI for generalized attribution.
  • Annotation: Figure 13 reports response statistics by part of speech.
  • Overlap calculations: The overlap calculations use logical-and for intersections and logical-or for unions of map pixels.The intersection is denoted INT in the measure definitions.
  • Results visualization: Figure 14 visualizes mIoD, mIoH, and multiple forms of overlap.

C Supplements for Semantic Analyses

The semantic analyses supplement DAAM with a relation ontology and cohyponym annotation procedure designed to examine generated-object entanglement.

  • Semantic relation ontology: The semantic analysis presents a relation ontology for organizing semantic categories.
  • Semantic relation ontology: The ontology includes example cohyponym groups such as backpack, handbag, and suitcase under FOOD.
  • Semantic relation ontology: Other listed groups pair hot dog, pizza, and sandwich with FRUIT, and apple, banana, and orange with ELECTRICAL DEVICE.
  • Semantic relation ontology: Additional groups include oven, refrigerator, and toaster under MONITOR DEVICE, and cell phone, laptop, and tv under FURNITURE.
  • Semantic relation ontology: The ontology also lists bench, chair, and couch under KITCHENWARE, plus fork, knife, and spoon under VESSEL.
  • Cohyponym annotation process: Cohyponym examples were annotated through an MTurk interface asking workers to identify the generated objects.Three unique master-level annotators completed each task, responses were manually verified, and workers with consistently poor quality were removed.
  • Semantic relation ontology: Bear, elephant, giraffe, and zebra are grouped under SPORTS, while skateboard, snowboard, and surfboard are grouped under VEHICLE.
Loading 2210.04885v5…