PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Everything: Emerging Localization Properties in Vision-Language Transformers

Walid Bousselham, Felix Petersen, Vittorio Ferrari, Hilde Kuehne

arXiv:2312.00878v3cs.CVcs.AI

TL;DR

Vision-language models trained with image-level objectives struggle to localize objects zero-shot, limiting their use for open-vocabulary localization. GEM adds a regularized self-self attention pathway to pretrained models without fine-tuning. It outperforms current training-free methods and reports state-of-the-art zero-shot point prediction results on OpenImages V7.

  • Problem

    Vision-language models trained with image-level objectives struggle with zero-shot localization and may assign object patches lower prompt similarity than background patches.

    Source: §1 · p. 1

  • Method

    GEM adds regularized self-self attention to pretrained vision-language models, using L2 normalization, adaptive temperature, repeated attention, and an ensemble of attention types.

    Source: §1 · p. 2

  • Results

    GEM improves over current training-free methods on zero-shot segmentation and achieves state-of-the-art results on OpenImages V7 zero-shot point prediction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports open-vocabulary localization without additional training, with regularizations intended to generalize across vision-language models and datasets.

    Source: §5 · p. 9

    Fine-tuned methods can miss rare entities when their curated image-text vocabulary reduces the diversity of learned concepts.

    Source: §4.6 · p. 9

Abstract

from arXiv · show

Vision-language foundation models have shown remarkable performance in various zero-shot settings such as image retrieval, classification, or captioning. But so far, those models seem to fall behind when it comes to zero-shot localization of referential expressions and objects in images. As a result, they need to be fine-tuned for this task. In this paper, we show that pretrained vision-language (VL) models allow for zero-shot open-vocabulary object localization without any fine-tuning. To leverage those capabilities, we propose a Grounding Everything Module (GEM) that generalizes the idea of value-value attention introduced by CLIPSurgery to a self-self attention path. We show that the concept of self-self attention corresponds to clustering, thus enforcing groups of tokens arising from the same object to be similar while preserving the alignment with the language space. To further guide the group formation, we propose a set of regularizations that allows the model to finally generalize across datasets and backbones. We evaluate the proposed GEM framework on various benchmark tasks and datasets for semantic segmentation. It shows that GEM not only outperforms other training-free open-vocabulary localization methods, but also achieves state-of-the-art results on the recently proposed OpenImagesV7 large-scale segmentation benchmark.

1. Introduction

Vision-language models excel at broad zero-shot tasks but struggle to localize objects, sometimes assigning lower similarity to object patches than to the background. GEM uses regularized self-self attention to form token groups while retaining language alignment, and the paper reports strong results on zero-shot segmentation and point prediction.

1. Introduction

  • Vision-language models trained with image-level objectives struggle with zero-shot localization and can assign object patches lower prompt similarity than background patches.The paper illustrates this inverse vision-language relation in Figure 1.Source: §1 · p. 1
  • Self-self attention increases similarity among similar tokens compared with standard q-k attention, while L2 normalization and adaptive temperature guide group formation.The method repeats self-self attention to strengthen grouping and ensembles its attention types.Source: §1 · p. 2
  • GEM improves over current training-free methods on zero-shot semantic segmentation and achieves state-of-the-art results on OpenImages V7 zero-shot point prediction.The paper also reports competitive results against approaches requiring fine-tuning.Source: §1 · p. 2
  • The paper presents training-free self-self attention localization and GEM regularizations as a way to generalize across vision-language models and datasets.Its evaluation covers open-vocabulary zero-shot semantic segmentation and point prediction.Source: §1 · p. 2

2. Related works

Prior work localizes objects using trained region or segmentation models, adapts vision-language models, or modifies pretrained models without training. These approaches trade off supervision, vocabulary scope, and preservation of language alignment.

2. Related works

  • Localization-first methods generate regions or masks and use vision-language models to label them, while their closed-vocabulary localizers constrain classification scope.OpenSeg and OVSeg are examples; GroundingSAM combines GroundingDINO with SAM and fine-grained supervision.Source: §2 · p. 3
  • Architecture-adaptation methods group visual features or train decoders with image-caption supervision, but some require dataset filtering that reduces free-form text characteristics.Examples include SegCLIP, GroupViT, ViL-Seg, OVSegmentor, ReCo, and PACL.Source: §2 · p. 3
  • Training-free methods adapt pretrained vision-language models for dense features; CLIPSurgery adds a parallel value-value attention pathway, which GEM extends to normalized self-self attention.MaskCLIP removes the final transformer layer's MLP, while CLIPSurgery aggregates outputs across layers with residual connections.Source: §2 · p. 3

3. Grounding with Self-Self Attention

The section generalizes value-value attention to self-self attention across visual projections and connects its token-grouping behavior to clustering. It adds normalization, adaptive temperature, iteration, and projection-ensemble choices to guide localization.

3.1. GEM: Grounding Everything Module

  • Query-query and key-key attention match or improve on value-value attention, and every self-self variant significantly outperforms standard query-key attention.Table 1 reports mIoU for these attention variants on PascalVOC and PascalContext.Source: §3.1 · p. 4, Table 1 · p. 4
  • L2-normalizing projected tokens and setting an adaptive temperature from pre-projection token norms and the ViT training temperature improve group formation and localization.The adaptive setting is intended to avoid hyperparameter tuning in the zero-shot setting.Source: §3.1 · p. 4
  • Iteratively applying normalized self-self attention gradually refines clusters; the paper fixes one additional iteration for most cases.The authors report that two successive self-self attentions are sufficient for most cases.Source: §3.1 · p. 4
  • The qkv ensemble combines iterative self-self attention over query, key, and value projections, and the reported comparison shows an improvement from ensembling normalized projections.Table 1 covers PascalVOC and PascalContext, with and without L2 normalization and adaptive temperature; no cell values are supplied here.Source: §3.1 · pp. 4–5, Table 1 · p. 4

3.2. Self-Self Attention for Clustering

  • Self-self attention acts like clustering: higher temperature or more iterations yields fewer, larger clusters, while lower temperature or fewer iterations yields more, smaller clusters.The simulation uses 20 random vectors; complex images with many classes might benefit from less clustering and fewer iterations.Source: §3.2 · p. 5, Figure 4 · p. 5
  • The proposed grouping effect is linked to projections that contract nearby tokens; measured Lipschitz constants are below 1 for value, key, and query projections.Reported constants are 0.51 ± 0.073 for value, 0.63 ± 0.091 for key, and 0.66 ± 0.104 for query.Source: §3.2 · p. 5

4. Evaluation

The evaluation tests GEM across zero-shot segmentation datasets and backbones, then analyzes its localization properties and qualitative successes and failures. GEM outperforms other training-free methods, with results varying by iteration count and backbone size.

4.1. Setup

  • The evaluation uses PascalVOC, Pascal Context, ADE20K, and OpenImages-V7, with mIoU for the first three and point-wise IoU for OpenImages-V7.The OpenImages-V7 validation set contains 36,702 images and 5,827 distinct class labels; all models use pretrained weights and GEM with one iteration.Source: §4.1 · pp. 5–6

4.2. Comparison to State-of-the-art

  • GEM consistently outperforms training-free methods on zero-shot semantic segmentation and exceeds trained or fine-tuned methods on OpenImagesV7 point prediction.On PascalContext and ADE20K, training-free methods also outperform localization-fine-tuned models except PACL; on OpenImagesV7, GEM exceeds GroundingSAM as well.Source: Table 2 · p. 6, Table 3 · p. 7, §4.2 · p. 7

4.3. Abalation

  • Normalization with the proposed adaptive temperature gives the highest mIoU on both PascalVOC and PascalContext.The proposed temperature’s multiplication factor is 1; the result is reported as consistent across both datasets.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Two self-self-attention iterations slightly improve PascalVOC, while one iteration is fixed for the paper’s zero-shot evaluations.Two iterations work slightly better on VOC, with fewer classes per image; fewer iterations work slightly better on PascalContext, with more classes per image.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Architecture and Model Size

  • GEM-ViT-B/16 consistently outperforms GEM-ViT-L/14, and BLIP tends to localize less well than models trained only with image-text contrastive loss.Increasing patch size from 16 to 32 slightly reduces performance for a fixed ViT-B size.Source: Table 5 · p. 8, §4.4 · p. 8

4.5. Analysis of Localization Properties

  • GEM shows higher object-background contrast than CLIPSurgery and CLIP, alongside improved text-object alignment.Patch-patch similarity rises from CLIP to CLIPSurgery and falls slightly from CLIPSurgery to GEM after normalization and temperature are added.Source: §4.5 · p. 8, Figure 6 · p. 8

4.6. Qualitative Analysis

  • Among the compared backbones, MetaCLIP produces sharper, more accurate localization and uniquely localizes “Glove”; BLIP’s masks are less precise.The comparisons cover CLIP, OpenCLIP, MetaCLIP, and BLIP.Source: §4.6 · p. 8, Figure 7 · p. 9
  • Some prompts segment related but unintended objects, and splitting “Vehicle registration plate” into “Vehicle” and “License plate” mitigates that failure.The “Human body” prompt also segments the car body; the authors attribute this failure type to the text encoder.Source: §4.6 · pp. 8–9, Figure 8 · p. 9

5. Conclusion

The conclusion presents GEM as a self-self attention pipeline that extracts latent localization information from vision-language models, using regularizations to support generalization across models and datasets for open-vocabulary localization without additional training.

  • GEM leverages latent localization capabilities in vision-language models trained on web-scale datasets.Source: §5 · p. 9
  • The method uses a self-self attention pipeline to extract localization information from vision-language models.Source: §5 · p. 9
  • Regularizations are intended to ensure generalizability across diverse models and datasets, enabling open-vocabulary localization without additional training.Source: §5 · p. 9

Supplementary Material

The supplementary material describes a demo, implementation details, ablations, localization-property analyses, and grouping-factor studies.

7. Colab Demo

The section provides a GoogleColab demo link.

  • A GoogleColab demo is provided at the linked address.Source: §7 · p. 13

8. Additional Implementation Details

GEM processes vision-transformer features in parallel through iterative, temperature-regularized self-self attention, then ensembles query-, key-, and value-based cues.

  • GEM processes input features from the vision transformer in parallel to the transformer.Source: §8 · p. 13
  • Each layer uses one self-self attention step on normalized projected features and one on values, applying the temperature heuristic to the value attention.Source: §8 · p. 13
  • The rollout applies self-self attention to query, key, or value projections, then to each projection applied to values, and finally ensembles the three projections.Source: §8 · p. 13

9. Additional Ablation

The depth and MLP ablations show that GEM performs best when started in middle layers, remains stable across a range of starting depths, and slightly benefits from omitting MLPs.

Impact of path length

  • On PascalVOC, GEM’s mIoU remains significantly stable for ViT-B/16 and ViT-B/32 when applied before the last layers, with best performance at depths of three to five layers.The authors fix GEM depth to d = 4 for all reported experiments.Source: §9 · p. 13

Impact of MLP

  • Adding the vision-transformer MLPs has a slight negative effect on PascalVOC mIoU for ViT-B/16 and ViT-B/32.The authors report that omitting MLPs generally leads to better results.Source: §9 · p. 13

10. Further Details on Cluster Analysis

Simulations show that more self-self-attention iterations and higher temperatures produce fewer, larger clusters.

10. Further Details on Cluster Analysis

  • In the Section 3.2 simulation, increasing self-self attention iterations (from top to bottom) and higher temperature each lead to fewer, larger clusters.Point-cluster associations reduced to two dimensions via PCA visualize cluster formation.Source: §10 · p. 13

11. Analysis of Localization Properties

The analysis treats localization as requiring both meaningful visual grouping and alignment of those groups with language. It examines visual distinctiveness, vision-language alignment, and how both change through GEM’s path.

11. Analysis of Localization Properties

  • The analysis frames localization performance around visual distinctiveness and vision-language alignment, with CLIP’s [CLS] token serving as the text-aligned reference for patch tokens.The [CLS] token was trained to correlate with text embeddings through contrastive learning.Source: §11 · pp. 13–14

11.1. Visual Distinctiveness

  • Visual distinctiveness requires similar patch tokens within an object while keeping tokens from distinct objects separated.The analysis uses patch-patch similarity and object-background contrast; the latter is evaluated with PascalVOC training-set masks.Source: §11.1 · pp. 14–15

11.3. Analysis

  • GEM reaches positive text-object contrast in later layers, while original CLIP remains negative; GEM’s object-background contrast recovers after three to four layers.Original CLIP’s patch-language alignment drops after layer six and recovers only in the last layer.Source: Figure 11 · p. 15, §11.3 · p. 15, Figure 12 · p. 15

11.2. Vision-Language Alignment

  • Vision-language alignment is measured by comparing text-embedding similarity to foreground patches against similarity to background patches.A higher positive global text-object-background contrast means foreground embeddings are closer to their corresponding text embeddings than background embeddings.Source: §11.2 · p. 15

Related papers

Loading 2312.00878v3…