PaperlayerView exact revision on arXiv

Source-linked AI summary

RefAM: Attention Magnets for Zero-Shot Referral Segmentation

Anna Kukleva, Enis Simsar, Alessio Tonioni, Muhammad Ferjad Naeem, Federico Tombari, Jan Eric Lenssen, Bernt Schiele

arXiv:2509.22650v3cs.CV

TL;DR

Referring segmentation motivates using diffusion-transformer cross-attention, which encodes spatial and semantic information without task-specific training. RefAM redirects surplus attention with stop-word magnets and reports state-of-the-art training-free results across image and video benchmarks, while leaving which-frame temporal expressions unresolved.

  • Problem

    Diffusion-transformer cross-attention encodes spatial and semantic information without task-specific training, motivating its use for zero-shot referring segmentation.

    Source: §1 · p. 1

  • Method

    RefAM appends stop-word magnets, filters their attention maps, and aggregates the remaining diffusion-transformer cross-attention for grounding.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    RefAM is state of the art among training-free methods, including a +4.45 mIoU gain over HybridGL on RefCOCO testB, and is best among fully training-free methods on Ref-DAVIS17, Ref-YouTube-VOS, and MeViS.

    Source: §5 · p. 11

  • Takeaways & Limitations

    Diffusion attention can serve as a general foundation for grounding in images and videos without task-specific training or architectural changes.

    Source: §1 · p. 3, §5 · p. 11

    For video referring segmentation, RefAM ignores temporal aspects of expressions and does not explicitly resolve which-frame references.

    Source: §1 · p. 2, D Limitations · p. 21

Abstract

from arXiv · show

Most existing approaches to referring segmentation achieve strong performance only through fine-tuning or by composing multiple pre-trained models, often at the cost of additional training and architectural modifications. Meanwhile, large-scale generative diffusion models encode rich semantic information, making them attractive as general-purpose feature extractors. In this work, we introduce a new method that directly exploits features (attention scores) from diffusion transformers for downstream tasks, requiring neither architectural modifications nor additional training. To systematically evaluate these features, we extend benchmarks with vision-language grounding tasks spanning both images and videos. Our key insight is that stop words act as attention magnets: they accumulate surplus attention and can be filtered to reduce noise. Moreover, we identify global attention sinks (GAS) emerging in deeper layers and show that they can be safely suppressed or redirected onto auxiliary tokens, leading to sharper and more accurate grounding maps. We further propose an attention redistribution strategy, where appended stop words partition background activations into smaller clusters, yielding sharper and more localized heatmaps. Building on these findings, we develop RefAM, a simple training-free grounding framework that combines cross-attention maps, GAS handling, and redistribution. Across zero-shot referring image and video segmentation benchmarks, our approach achieves strong performance and surpasses prior methods on most datasets, establishing a new state of the art among training-free methods, without task-specific training or architectural modification. For video, we localize the referent from the spatio-temporal features of a video diffusion transformer via a first-frame query that is then propagated; we therefore do not explicitly resolve temporal (which-frame) expressions.

1 Introduction

The introduction identifies stop words as attention magnets and global attention sinks in diffusion transformers, then presents RefAM as a training-free grounding framework for images and videos. It reports strong zero-shot benchmark performance without fine-tuning or architectural modification.

1 Introduction

  • Stop words can form global attention sinks that spread nearly uniform attention across text and image tokens, while other stop words attract attention to irrelevant regions.Filtering global-sink channels does not harm downstream performance; appended stop words can redistribute background attention into cleaner heatmaps.Source: §1 · p. 2
  • For video, RefAM derives a referring query from a video diffusion transformer's spatio-temporal features in the first frame and propagates the mask with SAM2, without resolving which-frame expressions.Source: §1 · p. 2
  • RefAM augments referring expressions with stop-word magnets, filters their attention maps, and aggregates the remaining cross-attention for grounding.The pipeline identifies an argmax location and applies SAM to produce the final segmentation mask.Source: §1 · p. 2, Figure 2 · p. 3
  • RefAM achieves state-of-the-art or competitive zero-shot referring segmentation results on image and video benchmarks, outperforming prior training-free methods without fine-tuning or architectural modification.Source: §1 · p. 3

2 Related Work

Related work covers high-attention tokens across transformer architectures and zero-shot referring segmentation methods. RefAM draws on naturally emerging diffusion-transformer sinks and stop-word magnets rather than learned registers or iterative refinement.

2 Related Work

  • High-attention tokens appear across language, vision, and vision-language models under several names, and are often described as lacking semantic relevance.The paper identifies global attention sinks as tokens spanning language and visual streams under one query and suppressing useful signals in both modalities.Source: §2 · p. 3
  • Proposed explanations for attention sinks include softmax normalization, positional encodings, learned bias terms, and repeated matrix multiplication that amplifies dominant feature directions.Source: §2 · p. 3
  • Unlike approaches that retrain models with learnable registers, RefAM uses naturally occurring stop words as training-free magnets that filter noise rather than store information.Source: §2 · p. 3
  • Zero-shot referring image methods include proposal scoring and iterative Grad-CAM refinement, while some text-supervised methods still train a dedicated grounding model.RefAM instead uses diffusion-transformer attention without an iterative refinement loop or task-specific training.Source: §2 · pp. 3–4
  • Prior video methods include supervised temporal approaches and zero-shot frame-wise grounding with pre-trained models; RefAM unifies image and video segmentation using diffusion-derived features.Source: §2 · p. 4

3 Semantic Features in DiTs

RefAM uses diffusion-transformer cross-attention and appended stop words to redistribute surplus attention for training-free referring segmentation. Its design filters attention sinks and background activations while retaining useful semantic alignment.

3 Semantic Features in DiTs

  • Diffusion-transformer cross-attention maps provide spatial grounding signals, which RefAM aggregates after identifying and filtering sinks and redistributing surplus attention through added tokens.The framework uses no additional training or model modifications.Source: §3 · p. 4, §3.1 · p. 5

3.1 Feature Extraction from DiTs

  • Intermediate DiT activations encode semantic information, and cross-attention maps between text and image tokens provide spatial grounding signals.A separate text branch encodes the referring expression for feature extraction without affecting the denoising trajectory.Source: §3.1 · p. 5

3.2 Referring Segmentation

  • Appending stop words absorbs surplus background activations, and filtering their maps leaves sharper heatmaps that yield a predicted location for segmentation.For video, the query point comes from the first frame and SAM2 propagates segmentation across the sequence.Source: §3.2 · p. 5

3.3 Emergence of Semantic Information in DiT

  • 60% of DiT blocks contain no structured information; semantic alignment emerges with depth, while early blocks can be filtered without changing performance.The table reports that performance starts degrading only after filtering more than 60% of blocks.Source: §3.3 · p. 5, Table 1 · p. 6, Figure 4 · p. 6

3.4 Interpretation of Global Attention Sinks

  • Global attention sinks emerge in later layers after semantic structure, but are uninformative themselves and can be removed without harming performance.In sampled expressions, 77% of identified sinks were stop words, while about 10% were color tokens and another 10% other content words.Source: §3.4 · p. 6

3.5 Redistribution strategy with attention magnets

  • Auxiliary stop words and color words redirect about 89% of color-sink attention, while multiple appended stop words partition background into smaller clusters for cleaner heatmaps.The combined redistribution is training-free and uses behavior already learned during pretraining.Source: §3.5 · pp. 6–7

4 Results

RefAM is evaluated on standard referring-image and referring-video segmentation benchmarks against training-free baselines. It achieves new state-of-the-art results on four of six datasets, with ablations and backbone analysis examining its components and generality.

4.1 Quantitative Analysis

  • RefAM achieves state-of-the-art results on 4 of 6 image and video datasets, outperforming prior training-free methods and remaining competitive on RefCOCO+ and RefCOCOg.The image and video benchmark comparisons are reported in Tables 2 and 3.Source: §4.1 · p. 8, Table 2 · p. 8, Table 3 · p. 9
  • On RefCOCO testB, RefAM gains 4.45 mIoU over HybridGL, the strongest prior zero-shot method, while comparisons remain within the training-free, no-external-grounding-model setting.RefAM uses frozen generative features and SAM; the authors do not claim to outperform grounding-model pipelines.Source: §4.1 · p. 9

4.2 Ablations

  • Appending and filtering attention magnets raises predicted point accuracy from 59.9 to 68.9 and improves RVOS J &F by 3.2 points.Combining noun-phrase encoding and spatial bias with attention magnets gives the best performance across the reported benchmarks.Source: §4.2 · pp. 9–10, Table 4 · p. 10
  • Across five sampled stop-word sets results are consistent, while random-vector replacements perform slightly worse; color tokens improve metrics by roughly 1%.The authors interpret this as evidence that background redistribution matters and frequently encountered real stop words are effective magnets.Source: §4.2 · p. 10, Table 5 · p. 10

4.3 Generalization & Backbone Analysis

  • On SD3.5 RefCOCO TestA, T5 alone outperforms CLIP, and removing stop words reduces T5 performance by 5.8 mIoU and 5.2 oIoU.CLIP changes from 46.3 to 44.8 mIoU on TestA when stop words are removed.Source: §4.3 · p. 11, Table 6 · p. 11

5 Conclusion

RefAM uses flow-matching diffusion-transformer cross-attention for training-free referring segmentation, with attention-magnet redistribution to sharpen localization. It performs strongly among training-free methods on image and video benchmarks, while relying on large foundation models and remaining intended for research rather than high-risk decisions.

5 Conclusion

  • RefAM is best among fully training-free methods on Ref-DAVIS17, Ref-YouTube-VOS, and MeViS, and exceeds prior zero-shot approaches on RefCOCO by +4.45 mIoU over HybridGL on testB.It is competitive on RefCOCO+/g; the comparison excludes pipelines using external grounding models.Source: §5 · p. 11
  • The framework uses diffusion-transformer cross-attention and attention-magnet redistribution to sharpen localization without retraining or architectural changes.The paper identifies stop words as attention magnets and global attention sinks as part of its approach.Source: §5 · p. 11

Broader Impact Statement

  • RefAM depends on large foundation models and language prompts whose training data and biases may affect downstream performance across demographic groups or underrepresented domains.The exact composition of some foundation-model training datasets is not always publicly disclosed.Source: Broader Impact Statement · p. 11
  • The authors frame RefAM for research uses such as retrieval and visual understanding, not high-risk or sensitive decisions such as surveillance or biometric identification.Source: Broader Impact Statement · p. 11

A.1 Analysis on Stable Diffusion 3.5

The SD3.5 encoder ablation finds different responses to RefAM: T5 is structure-aware and sensitive to stop-word removal, while CLIP is largely insensitive but weaker on complex splits. The reported evaluation covers RefCOCO, RefCOCO+, and RefCOCOg.

A.1 Analysis on Stable Diffusion 3.5

  • Removing stop words causes significant performance drops for T5, which the authors characterize as structure-aware.Source: §A.1 · p. 16
  • CLIP acts largely as a bag-of-words model and is relatively insensitive to stop-word filtering, but does not reach T5's peak performance on complex splits.Source: §A.1 · p. 16, Table 7 · p. 16
  • T5 consistently outperforms CLIP across RefCOCO, RefCOCO+, and RefCOCOg, and is sensitive to RefAM stop words.Table 7 reports the full oIoU evaluation across the three benchmarks.Source: Table 7 · p. 16, §A.1 · p. 16

A.2 Timestep Robustness

The appendix reports robustness to timestep choice, caption substitution, and expression type. A mid-trajectory timestep performs consistently, empty image prompts have little effect, and performance is comparable across expression subsets, with attributes highest.

A.2 Timestep Robustness

  • Across the mid-trajectory band, oIoU stays within about 40–43 and mIoU within about 51–54; the default timestep reaches 42.67 / 52.17 on RefCOCO val.The sweep uses a representative 500-expression subset, and one fixed mid-trajectory choice per backbone is used across datasets.Source: §A.2 · p. 16, Figure 6 · p. 16

A.3 Caption Dependence on Images (RIS)

  • Replacing the COCO caption with an empty source prompt drops oIoU by only 1.3 points on 3,000 RefCOCO val expressions.For images, the method uses dataset-provided COCO captions rather than LLM-generated text.Source: §A.3 · p. 17, Table 8 · p. 17

A.4 Relational and Attribute Expressions (RIS)

  • RefAM performs comparably on spatial, attribute, and plain expressions, with the attribute subset highest, supporting localization beyond object category alone.The subsets distinguish relational/location cues, color/size/material attributes, and plain expressions.Source: §A.4 · p. 17, Table 9 · p. 17

A.5 Robustness to Random Seeds (RIS)

The appendix finds low seed-to-seed variation and limited sensitivity to the number of appended magnets. Performance gains level off after a handful of magnets, making the exact count relatively insensitive.

A.5 Robustness to Random Seeds (RIS)

  • Across five seeds on 500 RefCOCO val expressions, oIoU is 40.6 (std 1.2) and mIoU is 52.2 (std 1.0).The mean mIoU matches the main-table RefCOCO val result of 52.17; the table describes variation as about one point on both IoU metrics.Source: §A.5 · p. 17, Table 10 · p. 17
  • The only stochasticity is noise at the single feature-extraction timestep; the timestep and argmax point selection are fixed.Source: §A.5 · p. 17

A.6 Number of Appended Attention Magnets (RIS)

  • Adding magnets lifts performance by about 3 points from the no-magnet baseline to a plateau around 10–15 magnets; beyond a handful, changes are within seed-level noise.The exact magnet count is therefore not a sensitive hyperparameter in this RefCOCO val subset.Source: §A.6 · p. 18, Table 11 · p. 18

A.7 Referring Video Object Segmentation

The ablations compare attention representation, text conditioning, and SAM2 capacity in the referring video segmentation pipeline. They report advantages for cross-attention, captions, and the larger SAM2 variant.

A.7 Referring Video Object Segmentation

  • Cross-attention representations outperform attention-output representations under the same RefAM referring pipeline.The comparison holds the referring-expression tokens, aggregation, and argmax-then-SAM2 step constant; the representations differ in how the heatmap is built.Source: §A.7 · p. 18, Table 12 · p. 18, Figure 7 · p. 19
  • Caption conditioning performs better across all metrics than empty prompts, whose removal causes noticeable drops.The authors conclude textual prompts benefit diffusion-model feature extraction.Source: §A.7 · p. 18, Table 13 · p. 19
  • Replacing SAM2-H with SAM2-S drops J &F from 57.6 to 51.8, with decreases across all scores.The authors suggest higher model capacity matters for fine-grained spatial details in referring video segmentation.Source: §A.7 · p. 18

A.8 Comparison to ConceptAttention on Image Segmentation Task

RefAM uses cross-attention and added stop words to ground expressive descriptions without enumerating scene concepts. The section also describes stop-word filtering and noun-phrase and spatial-cue preprocessing, whose components contribute complementary gains.

A.8 Comparison to ConceptAttention on Image Segmentation Task

  • Unlike ConceptAttention, RefAM does not require advance enumeration of simple scene concepts and can ground expressive natural-language descriptions.RefAM uses cross-attention linked to referring expressions and appended stop words; ConceptAttention maps depend on predefined concepts, often one-word labels.Source: §A.8 · pp. 19–20
  • RefAM outperforms a range of diffusion, DINO, CLIP ViT, and Flux DiT interpretability techniques on ImageNet-Segmentation and PascalVOC (Single Class).The comparison follows ConceptAttention’s evaluation procedure; qualitative examples show better object coverage than ConceptAttention.Source: §A.8 · p. 19, Table 15 · p. 20

B Additional Stop Words & Filtering

  • Stop words often attract background regions under token-wise softmax normalization, so filtering their maps before averaging yields more focused referring-expression attention.Additional stop words act as background-attention magnets during softmax and are filtered before averaging as well.Source: B Additional Stop Words & Filtering · pp. 20–21

C Noun Phrase and Spatial Bias Extraction

  • Noun-phrase extraction improves semantic alignment, spatial bias improves localization, and together they yield the strongest results with attention magnets across benchmarks.The preprocessing parses object-centric phrases and spatial relations; relative cues use candidate-region centroids, while absolute cues use positional masks.Source: C Noun Phrase and Spatial Bias… · p. 21

D Limitations

The method benefits from high-quality captions, though empty prompts do not substantially degrade performance. In video referring segmentation, it currently ignores temporal aspects of expressions.

D Limitations

  • High-quality captions better guide semantic alignment, but when unavailable the method relies on LLM-generated descriptions; performance does not degrade substantially with empty prompts.Source: D Limitations · p. 21
  • Video referring object segmentation currently ignores temporal aspects of the expression.Source: D Limitations · p. 21

E Qualitative Examples

The qualitative examples show RefAM grounding relational and attribute expressions, alongside image and video examples and attention-map comparisons with and without added stop words.

E Qualitative Examples

Loading 2509.22650v3…