PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models

Binglei Li, Mengping Yang, Zhiyu Tan, Wenxu Qian, Junping Zhang, Hao Li

arXiv:2610.13337v1cs.CVcs.CL

TL;DR

Attention sinks are well studied in language models, but their behavior and functional relevance in text-to-image Transformers remain comparatively unexplored. This paper analyzes sinks across diffusion and autoregressive models and tests their effects through targeted interventions and register tokens. The models show architecture- and stage-dependent sink patterns, with perturbations revealing differing functional effects.

  • Problem

    Attention-sink behavior and functional relevance remain largely unexplored in text-to-image Transformers, despite extensive study in large language models.

    Source: §1 · p. 2

  • Method

    The paper systematically characterizes attention sinks across diffusion and autoregressive T2I models, applies targeted interventions, and tests register tokens for attention redistribution.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    Sink patterns differ by architecture and generation stage: the evaluated autoregressive model favors the first token, while diffusion models favor modality boundaries and spatial corners; sink-related head perturbations produce larger average changes than random or low-sink comparisons.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Attention sinks offer a lens for interpreting and potentially controlling generative attention, while the results distinguish where attention accumulates from where generation is functionally sensitive.

    Source: §1 · p. 2, §7 · p. 9

    The study covers three T2I Transformers and inference-time behavior, leaving broader architectures, scales, training recipes, and sink emergence during training for future validation.

    Source: §7 · p. 9

Abstract

from arXiv · show

Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest. Despite extensive study in large language models, their behavior and functional relevance in text-to-image (T2I) Transformers remain largely unexplored. To fill this gap and better understand how text-to-image Transformers allocate and route attention, we explore this phenomenon of text-to-image transformers across bidirectional diffusion models and causal autoregressive models. Specifically, we explore whether attention sinks exist, where they emerge, and how they functionally influence generation. Through systematic attention analysis and targeted interventions, we identify that attention sinks emerge in both bidirectional diffusion and autoregressive models but exhibit distinct patterns. The autoregressive model concentrates attention on the first and global-scale tokens, whereas the bidirectional models develop sinks near modality boundaries, within text regions, and at spatial corners. Moreover, these patterns vary across generation stages and are localized to a sparse subset of layers and heads. To assess their functional relevance, we perturb sink-related heads and token positions. It turns out that perturbing sink-ranked heads produces larger average changes than non-sink comparisons, while some sink-position perturbations have limited effects on bidirectional model scores. We further introduce register tokens to redistribute attention sinks and find that several configurations preserve or modestly improve generation performance. Overall, attention sinks are architecture-, inference-stage-, and component-dependent phenomena, providing a basis for mechanistic analysis and principled attention design in text-to-image transformers.

1. Introduction

Attention sinks are well studied in language models but remain underexplored in text-to-image Transformers, whose multimodal layouts and generation processes may produce different attention patterns. This paper systematically analyzes sinks across diffusion and autoregressive models and tests their functional effects.

1. Introduction

  • T2I Transformers combine text and image tokens and generate through either iterative diffusion denoising or autoregressive image-token prediction.Their spatial or multiscale visual-token structures and attention directionality may shape sink locations beyond the conventional first-token pattern.Source: §1 · p. 2
  • The study asks whether attention sinks exist in T2I Transformers, where they occur, and whether they affect generation.It examines FLUX.1 and Z-Image diffusion models and the Infinity-2B autoregressive model.Source: §1 · p. 2
  • The analysis characterizes sink strength, inequality, localization, positional preference, prompt sensitivity, and generation-time dynamics on GenEval and DPG-Bench.Targeted interventions test attention routing, value contributions, sink-related heads, and candidate positions; register tokens probe redistribution while preserving synthesis quality.Source: §1 · p. 2
  • Attention sinks are structured in both model paradigms, but their locations differ: autoregression concentrates on the first token, while diffusion favors modality boundaries, text regions, and spatial corners.Sink behavior is localized to a sparse subset of layers and heads and changes across denoising or coarse-to-fine generation stages.Source: §1 · p. 2
  • Perturbing sink-ranked heads produces larger average changes than random or low-sink comparisons, while some sink-position perturbations have limited effects in bidirectional models.Several register-token configurations redistribute attention while preserving or modestly improving generation performance.Source: §1 · p. 2

2. Preliminaries

The preliminaries define attention sinks as positions with consistently high cumulative attention and introduce the Transformer attention framework used in the study. They distinguish bidirectional diffusion models from causal autoregressive models by their token organization and generation process.

2. Preliminaries

  • An attention sink is a token position that consistently receives disproportionately high cumulative attention across queries and heads.The text introduces a set of sink tokens, with a concentration threshold τ used in its formulation.Source: §2 · p. 3
  • A Transformer forms queries, keys, and values through linear projections of its input and computes attention from them.The head dimension is d_k = D/H, where D is the feature dimension and H is the total number of heads.Source: §2 · p. 3
  • MMDiTs concatenate image features and text embeddings and jointly model them with bidirectional attention to denoise the latent across transformer blocks.Autoregressive models factorize the joint distribution and predict the next token with causal attention or the next scale in a coarse-to-fine hierarchy.Source: §2 · p. 3

3. Analytical Framework

The framework operationalizes attention-sink metrics, tests sink-related interventions, and evaluates diffusion and autoregressive text-to-image models using complementary benchmarks.

3.1. Metrics for Attention Sinks

  • The framework ranks heads by sink concentration, compares candidate positions by their prevalence among top-k sink sets, and treats values above 1/N as genuine concentration.The uniform-attention null is approximately 1/N across bidirectional, causal, and variable-availability masks.Source: §3.1 · p. 3
  • Generation-time metrics track sink shifts across early, middle, and late stages, positional stability across steps, and prompt-distribution stability.Diffusion uses the image-region share of top-k sinks; Infinity uses a fine-scale score, while prompt stability uses profile correlation and normalized sink-position entropy.Source: §3.1 · p. 3

3.2. Intervention Strategy

  • Three suppressive interventions test sink function: penalizing attention logits, zeroing sink-position values, and pruning selected high-concentration heads.Each intervention is applied separately; value substitution preserves attention routing, while head pruning removes the selected heads’ output contribution.Source: §3.2 · pp. 3–4

3.3. Experimental Setups

  • The evaluation covers FLUX.1-dev, Z-Image, and Infinity-2B at 1024×1024, using GenEval and DPG-Bench plus auxiliary preference and perceptual-quality metrics.Diffusion models use 28-step EDM sampling; Infinity-2B uses a 13-scale bitwise schedule. The benchmarks together contain 1,618 prompts.Source: §3.3 · p. 4
  • Table 1 reports first-token concentration in Infinity and concentration near modality boundaries, text regions, and spatial corners in diffusion models.Figure 2 shows attention-mass peaks by token position across FLUX.1, Z-Image, and Infinity-2B, marking modality or scale boundaries and magnifying local peaks.Source: Table 1 · p. 4, Figure 2 · p. 4

4. Systematic Analysis

Across the four analyses, sink locations vary with model architecture, generation stage, and prompt distribution. Sink-dominant heads are sparse, while sink positions can remain locally stable yet shift over longer generation stages or across prompts.

4.1. Effects of Token Position on Attention Sinks

  • Diffusion models rarely select the first position as a sink (ρ1(0) < 1%), unlike Infinity-2B (ρ1(0) = 37.52%); diffusion sinks favor text, modality boundaries, and image corners.Image-side peaks include FLUX.1’s bottom-left corner and Z-Image’s top-left corner; Infinity-2B’s analogous position is its global-scale token.Source: §4.1 · pp. 4–5

4.2. Effects of Layer-Head on Attention Sinks

  • Only about 3%–5% of heads are sink-dominant, with model-specific depth patterns rather than a universal placement.FLUX.1 peaks at L39–L47, Z-Image is dominated by L02H25, and Infinity-2B concentrates sinks around L27–L30.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3. Effects of Timestep on Attention Sinks

  • Image-directed ratios rise through generation in all three models, while sink positions remain relatively stable across adjacent steps but shift across full stages.The reported image-directed ratios rise from 41.21% to 50.47% for FLUX.1, 7.46% to 12.73% for Z-Image, and 81.24% to 89.95% for Infinity-2B.Source: §4.3 · pp. 5–6, Table 2 · p. 5, Figure 4 · p. 6

4.4. Effects of Prompt on Attention Sinks

  • Sink-head concentration maps correlate above 0.98 across prompt datasets, but sink-position variation increases on DPG-Bench for both diffusion models.The high-entropy ratio rises from 6.8% to 92.2% for FLUX.1 and from 2.7% to 49.4% for Z-Image; Infinity-2B remains nearly unchanged. The datasets differ in prompt length, semantics, and complexity, so the change is not attributed to a single prompt factor.Source: §4.4 · p. 6, Table 3 · p. 6

5. Intervention Results

Interventions probe the sensitivity of sink-related pathways, though selected sinks may also coincide with structural tokens, so score changes are not isolated causal effects of sink behavior. The experiments show model- and stage-dependent effects, with head pruning most disruptive and register tokens partially redistributing sink dominance.

5. Intervention Results

  • The interventions measure pathway sensitivity rather than isolated causal effects of sink behavior, because selected sinks may coincide with structural tokens.The study perturbs sink positions, values, and heads relative to each model’s baseline.Source: §5 · p. 6

5.1. Impact of Intervention Strategies.

  • Across FLUX.1, Z-Image, and Infinity-2B, logit penalties change scores by −0.19%, −0.36%, and −2.79%; value substitution and head pruning are more disruptive.Value substitution changes scores by −0.71%, −0.58%, and −5.45%, respectively; head pruning changes them by −4.19%, −2.36%, and −6.56%. As more sink tokens are perturbed, Infinity-2B degrades more steeply than FLUX.1 and Z-Image, with sensitivity growing as more model-specific sink structure is covered.Source: §5.1 · pp. 6–7, Table 4 · p. 7, Figure 5 · p. 7

5.2. Sensitivity to Structural Sink-Token Removal.

  • Removing Infinity-2B’s position-0 token at s1 drops GenEval from 64.56 to 11.37 and ImageReward from 0.9761 to −0.8055; later removal recovers GenEval.Starting removal at s4 or s8 recovers GenEval to 62.90 or 63.35; FLUX.1 remains robust across tested positions, with GenEval between 64.81 and 65.45 and DPG and ImageReward near or slightly above baseline.Source: Table 5 · p. 7, §5.2 · p. 7

5.3. Interventions on Sink Heads.

  • On Z-Image, pruning high-concentration heads increases degradation from −1.12% for Top-20 to −5.84% for Top-100; Top-50 is more damaging than Random-50 or Bottom-50.The comparison supports a greater role for selected sink heads than other heads in generation, where they may provide semantic or structural information.Source: Table 6 · p. 7, §5.3 · p. 7

5.4. Probing Sink Attention with Register Tokens.

  • Adding a register improves FLUX.1 GenEval from 67.14 to 68.85 and DPG from 83.67 to 84.01 while only partially weakening native sink positions.The top-1 sink ratios decrease from 12.95% to 12.44% at position 7 and from 3.04% to 2.72% at position 4544; this measures sink dominance, not total attention mass.Source: Table 7 · p. 8, §5.4 · p. 8

6. Related Work

Text-to-image Transformers use diffusion and autoregressive generation paradigms, but attention-sink research has focused largely on language models. Prior findings span several model domains, while unified analyses of recent T2I architectures remain limited.

6.1. Text-to-Image Transformer

  • Diffusion T2I models progressed from DiT [27] to joint text-image attention in SD3 [9] and models including FLUX.1, while autoregressive models generate image-token sequences left-to-right.The related-work discussion also notes autoregressive approaches including DALL-E [28], CogView [8], Parti [43], LlamaGen [33], VAR [34], and Infinity [15] [24].Source: §6.1 · p. 8

6.2. Attention Sinks in Transformers

  • Attention sinks are extremely high-attention tokens; prior work attributes them to softmax’s sum-to-one constraint forcing attention onto uninformative tokens as a numerical buffer [2] [32].Autoregressive LLM studies report initial-token dominance under causal masking [41].Source: §6.2 · p. 8
  • Prior sink findings span language, vision, multimodal, and diffusion models, but unified comparisons across layers, heads, positions, and generation stages in recent T2I architectures remain limited.For example, vision studies locate sinks on low-information background patches [2], while diffusion studies report moving sinks [29] and their possible role in protecting against over-mixing [46].Source: §6.2 · p. 8

7. Conclusions

The study finds architecture-dependent attention-sink patterns in the evaluated T2I models and reports that selected sink-related heads and positions can affect performance. Its evidence is limited to three models and inference-time behavior.

7. Conclusions

  • The evaluated autoregressive model exhibits strong first-token attraction, whereas evaluated bidirectional models concentrate sink mass near modality boundaries and spatial corners.The models do not all reproduce the canonical language-model first-token pattern; sink patterns vary with multimodal token layouts and generation stages.Source: §7 · pp. 8–9
  • Interventions show performance sensitivity to selected sink-related heads and positions, motivating further study of dedicated non-semantic positions.The authors describe attention sinks as a useful lens for interpreting and potentially controlling generative attention in T2I models.Source: §7 · p. 9
  • The study covers three T2I Transformers and inference-time behavior; validation across broader architectures, scales, and training recipes remains future work.Understanding how attention sinks emerge during training is also identified as an important next step.Source: §7 · p. 9
Loading 2610.13337v1…