PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation

Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, Patrick Pérez

arXiv:2312.12359v2cs.CV

TL;DR

CLIP has strong global open-vocabulary recognition but lacks dense features, while segmentation approaches can require annotations or change its original alignment. CLIP-DINOiser learns localization guidance to refine dense CLIP features. It reports state-of-the-art results across four segmentation benchmarks, though its granularity remains bounded by CLIP’s class-separation capability.

  • Problem

    CLIP aligns images with arbitrary text but struggles to provide dense open-vocabulary features, while segmentation commonly relies on annotated datasets for finite class sets.

    Source: §1 · p. 1

  • Method

    CLIP-DINOiser learns patch-correlation and objectness guidance from self-supervised features to refine MaskCLIP features without annotations.

    Source: §1 · p. 2

  • Results

    CLIP-DINOiser achieves state-of-the-art results on COCO, Pascal Context, Cityscapes, and ADE20K.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CLIP features already contain useful localization information that light convolutional layers can learn and use for zero-shot open-vocabulary segmentation in one CLIP pass.

    Source: §5 · p. 8

    CLIP-DINOiser’s granularity remains bounded by CLIP’s ability to separate classes.

    Source: §5 · p. 9

Abstract

from arXiv · show

The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both worlds and propose an open-vocabulary semantic segmentation method, which does not require any annotations. We propose to locally improve dense MaskCLIP features, which are computed with a simple modification of CLIP's last pooling layer, by integrating localization priors extracted from self-supervised features. By doing so, we greatly improve the performance of MaskCLIP and produce smooth outputs. Moreover, we show that the used self-supervised feature properties can directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference, no extra supervision nor extra memory and reaches state-of-the-art results on challenging and fine-grained benchmarks such as COCO, Pascal Context, Cityscapes and ADE20k. The code to reproduce our results is available at https://github.com/wysoczanska/clip_dinoiser.

1. Introduction

CLIP’s global image–text alignment supports open-vocabulary recognition but does not provide dense features, while segmentation methods often rely on annotations or alter that alignment. CLIP-DINOiser refines MaskCLIP features with learned localization guidance from self-supervised features, aiming for high-quality open-vocabulary masks with one CLIP pass.

1. Introduction

  • CLIP aligns images with arbitrary text and generalizes well globally, but struggles to produce dense open-vocabulary features.Segmentation methods commonly require manually annotated datasets for a finite set of classes.Source: §1 · pp. 1–2
  • Training or fine-tuning dense CLIP models can require long training or annotated data, while fine-tuning can alter vision-language associations.Heuristic densification of frozen CLIP features may require multiple forward passes; MaskCLIP is efficient but its local features can be noisy.Source: §1 · p. 2
  • Self-supervised features, particularly DINO’s, provide localization properties without manual annotation, but existing segmentation uses can require costly sampling or concept-specific prototypes.CLIP-DINOiser uses those localization cues to refine dense CLIP features.Source: §1 · p. 2
  • The method trains a convolutional layer on unlabeled data to pool CLIP features using patch-correlation guidance that mimics DINO localization cues.A second convolutional layer learns objectness information from CLIP features to improve segmentation of the background prompt.Source: §1 · p. 2
  • CLIP-DINOiser achieves state-of-the-art results on COCO, Pascal Context, Cityscapes, and ADE20K without annotations or retraining CLIP from scratch.It uses one CLIP forward pass and light pooling layers.Source: §1 · p. 2

2. Related Work

Prior zero-shot segmentation methods often depend on annotations, extensive training, or class-specific components. Work on dense CLIP features and self-supervised localization offers alternatives, but incurs computational costs or departs from open-vocabulary use.

2. Related Work

  • Zero-shot segmentation methods that generalize from seen to unseen classes can produce fine segmentations without inference overhead, but require pixel-level annotations for seen classes.They may extend to unseen classes using class information and relationships in word embeddings.Source: §2 · p. 2
  • Dense CLIP models are trained or fine-tuned with pixel-aligned objectives and varied annotations, but may require long training or costly annotations, and defined-vocabulary fine-tuning can alter open-vocabulary features.Examples include dense annotations, object masks, captions, and pseudo-labels.Source: §2 · pp. 2–3
  • Direct CLIP-feature densification aggregates multiple views or sliding windows, while MaskCLIP matches projected patch features to text after dropping global pooling, producing noisy dense predictions.Some densification approaches require multiple forward passes.Source: §2 · p. 3
  • Concept-specific visual-prototype codebooks can support cosegmentation but require expensive class-specific prototypes, unlike open-vocabulary approaches that avoid rebuilding them for new concepts.Related work also uses self-supervised features for localization; CLIP-DIY uses FOUND guidance with multiple CLIP passes, whereas this paper uses DINO patch correlations and an uncertainty constraint.Source: §2 · p. 3

3. Method

CLIP-DINOiser refines dense MaskCLIP features with localization information, aiming to preserve open-vocabulary segmentation while reducing noisy predictions. It first uses self-supervised patch correlations to guide pooling, then learns localization and objectness cues directly from CLIP features.

3.2. Preliminaries on MaskCLIP

  • MaskCLIP makes dense maps by matching projected patch features to text, but its predictions are noisy.The method modifies CLIP’s last pooling layer and produces dense maps in one forward pass.Source: §3.2 · p. 3

3.3. DINOising open-vocabulary features

  • DINO patch correlations are denser and more accurate than MaskCLIP’s, making them useful weights for locally consolidating MaskCLIP features.The resulting pooled features are compared with text queries to produce segmentation maps.Source: §3.3 · p. 4
  • Guided pooling smooths MaskCLIP’s high-frequency predictions and produces more accurate outputs.It averages patch features using affinity weights, with correlations below threshold γ zeroed out.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4. Teaching CLIP a first DINO trick: object correlations

  • A trained 3 × 3 convolution predicts DINO-like patch correlations from CLIP features, allowing CLIP-DINOiser to use guided pooling without DINO at inference.The authors report a similar boost over MaskCLIP, with one CLIP forward pass and a small extra layer.Source: §3.4 · pp. 4–5

3.5. Teaching CLIP a second DINO trick: background filtering

  • For background filtering, patches receive the background prompt only when they are both low-confidence and marked as background by the objectness mask.This relaxes FOUND’s otherwise restrictive mask for partially visible or cluttered objects.Source: §3.5 · p. 5
  • A 1 × 1 convolution learns FOUND’s objectness mask from CLIP features, and its predictions can replace FOUND’s mask with minimal performance drop.The learned objectness predictions are reported to be highly similar to those produced with DINO.Source: §3.5 · pp. 5–6

4. Experiments

The experiments evaluate CLIP-DINOiser on zero-shot open-vocabulary segmentation benchmarks and ablate its pooling and background-detection components. The method leads on four datasets without a background prompt and improves over MaskCLIP through feature-level pooling.

4.1. Experimental setup

  • The evaluation uses frozen OpenCLIP ViT-B/16 and compares zero-shot methods across eight benchmarks, reporting mIoU without post-processing.The benchmarks are divided according to whether they include a background prompt; CLIP-DINOiser uses only 1k unannotated images and no additional caption annotations.Source: §4.1 · p. 6

4.2. Open vocabulary semantic segmentation

  • CLIP-DINOiser achieves the best results on four no-background datasets, exceeding the second-best method by +2.2, +5.0, +6.7, and +5.1 mIoU.The evaluation reports mIoU without post-processing; Table 1 separates datasets with and without a background prompt.Source: §4.2 · p. 6, Table 1 · p. 7
  • With a background prompt, CLIP-DINOiser surpasses OVDiff by +2.3 mIoU on Context and is on par on Object, while outperforming methods without prototypes.On VOC20 without a background prompt, OVDiff is ahead by 0.8 mIoU; the authors attribute this possibly to its per-concept negative prototypes.Source: §4.2 · p. 7, Table 1 · p. 7
  • Qualitative examples show finer localization than CLIP-DIY and TCL on Cityscapes and ADE20k, while CLIP-DINOiser reduces MaskCLIP hallucinations as prompted query counts grow.The comparison is made without post-processing.Source: §4.2 · p. 7, Figure 8 · p. 8

4.3. Ablation study

  • Feature-level pooling with DINO-based weights improves over MaskCLIP by +6 to +19 mIoU across datasets, and trained CLIP-based weights improve results further.The comparison contrasts feature-level pooling with prediction-level refinement using CLIP keys.Source: §4.3 · p. 8, Table 2 · p. 8
  • Relaxing FOUND background detection with an uncertainty condition raises VOC to 62.1 mIoU, versus 60.1 mIoU using FOUND alone.The authors report similar results with CLIP-based objectness predictions and DINO-based or trained CLIP-based correlations, with trained CLIP correlations best overall.Source: §4.3 · p. 8, Table 2 · p. 8

5. Conclusions

CLIP-DINOiser uses lightweight convolutional layers guided by DINO to learn patch correlations and objectness from CLIP features. It performs open-vocabulary segmentation in one CLIP pass and is limited by CLIP’s class-separation granularity.

5. Conclusions

  • CLIP-DINOiser learns patch-correlation and objectness information from CLIP features using DINO as a guide, then segments in one CLIP pass with two lightweight convolutional layers.The method reaches state-of-the-art results on complex semantic-segmentation datasets.Source: §5 · p. 8
  • The method’s localization information is already present in CLIP features and can be exploited with lightweight convolutional layers.Source: §5 · p. 8
  • CLIP-DINOiser inherits CLIP’s class-separation granularity, which bounds its performance.The authors suggest better prompt engineering and image-text models could further improve performance.Source: §5 · p. 9

A.1. The impact of the training dataset

Training results are stable across random ImageNet subsets, and changing the training dataset or increasing its size shows no significant performance difference.

A.1. The impact of the training dataset

  • Across three random ImageNet training subsets, standard deviations are at most 0.1 mIoU in all reported cases.Source: §A.1 · p. 12, Table 3 · p. 12
  • Training on ImageNet or DUTS-TR produces no significant difference, and increasing ImageNet training data from 1k to 10k images does not seem to improve results.Source: §A.1 · p. 12, Table 3 · p. 12
  • Table 3 reports average scores and standard deviations for random ImageNet splits, separating datasets with and without a background class.Source: Table 3 · p. 12, §A.1 · p. 12

A.2. Self-supervised features discussion

Among DINO’s last-layer query, key, and value embeddings, value features yield the best segmentation results. Their correlations better separate background elements, and the resulting masks segment objects such as trees and sky.

A.2. Self-supervised features discussion

  • Unlike key embeddings commonly used for foreground-background separation, value-feature correlations better separate elements within the background.Background patches correlate with fewer other background patches, separating regions more clearly.Source: §A.2 · p. 12
  • DINO value embeddings produce the best segmentation among query, key, and value features, with more background objects such as trees and sky segmented and labeled.Source: §A.2 · p. 12, Figure 8 · p. 12

A.3. Background evaluation with FOUND

The background evaluation reports that foreground predictions learned from CLIP perform on par with or better than FOUND on unsupervised object discovery and saliency detection. The appendix also introduces additional qualitative comparisons and visual ablations of pooling and background detection.

A.3. Background evaluation with FOUND

  • The evaluation measures foreground/background quality through unsupervised object discovery and saliency detection.The setup uses the class-agnostic foreground/background protocol defined in.Source: §A.3 · pp. 12–13
  • CLIP-learned foreground predictions perform on par with or better than FOUND on unsupervised object discovery and saliency detection.Object discovery uses CorLoc on VOC07, VOC12, and COCO20k; saliency detection uses mIoU on DUT-OMRON, DUTS-TEST, and ECSSD.Source: §A.3 · p. 13

B. More qualitative results

B.1. Visual ablations

Visual ablations show that spatial pooling reduces noisy predictions, while background refinement removes uncertain or unannotated regions. In in-the-wild examples, CLIP-DINOiser produces less noisy masks than MaskCLIP, including when false-positive text queries are considered.

B.1. Visual ablations

  • Spatial pooling reduces prediction noise and helps produce good-quality segmentation across the shown examples.The visual comparisons are against MaskCLIP.Source: §B.1 · p. 13
  • Background refinement removes uncertain regions, including snow classified as “snowboard” and an unannotated cabinet.The examples compare results with and without the background refinement step.Source: §B.1 · p. 13

B.2. In-the-wild examples

  • CLIP-DINOiser produces less noisy masks than MaskCLIP and eliminates false-positive predictions in the shown in-the-wild examples.False-positive queries are prompts that appear in the final segmentation but are not depicted in the image.Source: §B.2 · p. 13

B.3. Limitations

The paper identifies several failure modes: CLIP can confuse frequently co-occurring concepts, small or distant objects may be missed, and text-query ambiguity remains. The authors also caution that evaluation inherited from fully supervised settings may limit assessment in an open-vocabulary paradigm.

B.3. Limitations

  • CLIP may produce similar features for frequently co-occurring concepts, such as “train” and “train tracks.”The authors also report a similar bias for “boat” and “sea.”Source: §B.3 · p. 13
  • Despite fine-grained segmentation, CLIP-DINOiser can miss small or far-away objects.A Cityscapes example illustrates this failure mode.Source: §B.3 · p. 13, Figure 12 · p. 14
  • Text-query ambiguity remains a limitation, with “house” mistaken for “building”; the authors say fully supervised evaluation may limit open-vocabulary assessment.They suggest redefining evaluation metrics could help address observed segmentation ambiguities.Source: §B.3 · p. 14

Related papers

Loading 2312.12359v2…