PaperlayerView exact revision on arXiv

Source-linked AI summary

Extract Free Dense Labels from CLIP

Chong Zhou, Chen Change Loy, Bo Dai

arXiv:2112.01071v2cs.CVcs.CL

TL;DR

The paper asks whether CLIP’s pretrained visual-language features can provide pixel-level semantics for semantic segmentation, an area traditionally dependent on labeled data. It introduces MaskCLIP and MaskCLIP+, finding that minimally adapted CLIP supports annotation-free open-vocabulary segmentation and substantially improves transductive zero-shot results.

  • Problem

    The paper investigates whether CLIP features, previously used mainly as global image representations, can support pixel-level semantic segmentation despite segmentation’s high dependence on labeled training data.

  • Method

    MaskCLIP extracts dense patch-level features while preserving CLIP’s visual-language association, and MaskCLIP+ uses its predictions as training-time pseudo labels with self-training.

  • Results

    50.5%, 46%, and 24.4% mIoU improvements on unseen classes are reported on PASCAL VOC, PASCAL Context, and COCO Stuff, respectively.

  • Takeaways & Limitations

    MaskCLIP can provide dense pseudo-label supervision for annotation-free, open-vocabulary segmentation across fine-grained, novel-concept, and moderately corrupted-input settings.

  • Takeaways & Limitations

    Conventional fine-tuning can break CLIP’s visual-language association, and manipulating text embeddings can fail on unseen classes.

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition. Many recent studies leverage the pre-trained CLIP models for image-level classification and manipulation. In this paper, we wish examine the intrinsic potential of CLIP for pixel-level dense prediction, specifically in semantic segmentation. To this end, with minimal modification, we show that MaskCLIP yields compelling segmentation results on open concepts across various datasets in the absence of annotations and fine-tuning. By adding pseudo labeling and self-training, MaskCLIP+ surpasses SOTA transductive zero-shot semantic segmentation methods by large margins, e.g., mIoUs of unseen classes on PASCAL VOC/PASCAL Context/COCO Stuff are improved from 35.6/20.7/30.3 to 86.1/66.7/54.7. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a new reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation. Source code is available at https://github.com/chongzhou96/MaskCLIP.

1 Introduction

The paper investigates whether CLIP’s visual-language features can support pixel-level semantic segmentation rather than only global image recognition. MaskCLIP preserves CLIP’s feature association for annotation-free, open-vocabulary prediction, while MaskCLIP+ uses pseudo labels and self-training to extend performance and architecture flexibility.

  • Motivation: CLIP features are explored for pixel-level dense prediction because prior work mainly used them as global image representations.The paper examines whether CLIP captures object-level and local semantics suitable for semantic segmentation.
  • Motivation: CLIP’s training on complex scenes encourages local semantics, open-vocabulary concepts, and contextual information useful for dense prediction.These properties include object co-occurrence, relations, and spatial-location priors.
  • MaskCLIP: Fine-tuning the CLIP image encoder and manipulating text embeddings failed to preserve zero-shot segmentation ability for unseen classes.The authors attribute this failure to breaking the original visual-language association through architectural changes, updated weights, or a seen-class mapper.
  • MaskCLIP: MaskCLIP extracts dense patch-level features and directly uses CLIP text embeddings as 1×1-convolution classifiers without breaking the visual-language association.It supports CLIP ResNet and ViT variants, with training-free key smoothing and prompt denoising for refinement.
  • MaskCLIP+: MaskCLIP+ deploys MaskCLIP during training as a pseudo-label annotator, enabling self-training with segmentation architectures beyond CLIP’s image encoder.This addresses MaskCLIP’s architectural constraint and supports models such as PSPNet and DeepLab.
  • Results: 50.5%, 46%, and 24.4% mIoU improvements on unseen classes are reported for PASCAL VOC, PASCAL Context, and COCO Stuff, respectively.The corresponding results improve from 35.6 to 86.1, 20.7 to 66.7, and 30.3 to 54.7.
  • Results: MaskCLIP+ supports annotation-free and open-vocabulary segmentation, including fine-grained classes, novel concepts, and moderately corrupted inputs.Examples include attribute-conditioned classes such as white car and red bus, and novel concepts such as Batman and Joker.
  • Conclusion: The study presents MaskCLIP as a source of dense pseudo labels for training existing segmentation methods without labeled training data.This positions CLIP-based features as supervision for annotation-free dense prediction.

2 Related Work

The related work situates MaskCLIP among methods transferring pretrained representations to downstream vision tasks. Unlike approaches that fine-tune CLIP for target segmentation datasets, MaskCLIP adapts CLIP for pixel-level prediction while preserving zero-shot transferability.

  • Transferable Representation Learning: Pretraining has long been used to improve dense prediction, while CLIP has enabled transfers to image manipulation, captioning, view synthesis, and object detection.These methods typically use CLIP as an image encoder for downstream tasks.
  • Transferable Representation Learning: MaskCLIP differs from DenseCLIP by adapting CLIP for pixel-level dense prediction without the target-dataset fine-tuning that sacrifices zero-shot transferability.DenseCLIP instead fine-tunes CLIP weights on target semantic-segmentation datasets.

3 Methodology

This section develops MaskCLIP by preserving CLIP’s visual-language association while extracting dense local features and using text embeddings as pixel-level classifiers.

  • 3.1 Preliminary on CLIP: CLIP jointly maps images and text into a unified representation space through visual-language pre-training.
  • 3.2 Conventional Fine-Tuning Hinders Zero-Shot Ability: Fine-tuning a CLIP-initialized DeepLab performs well on seen classes but fails to segment unseen classes satisfactorily.The approach replaces ImageNet weights with CLIP image-encoder weights and maps text embeddings to classifier weights.
  • 3.3 MaskCLIP: MaskCLIP’s global attention pooling computes a spatially weighted sum of local value features, whose outputs represent the whole image while retaining local semantic responses.The attention formulation uses an average-pooled feature as query and spatial features as key-value pairs.
  • 3.3 MaskCLIP: MaskCLIP extracts dense patch-level value features from CLIP’s image encoder and directly uses text embeddings as 1×1-convolution classification weights.Its image-encoder modification removes query and key embeddings while reformulating value and final linear layers as 1×1 convolutions.
  • 3.3 MaskCLIP: MaskCLIP keeps CLIP’s text encoder unchanged and produces pixel-level mask predictions for target classes described by prompts.The resulting class text embeddings serve as the classifier, preserving the original visual-language association.
  • 3.3 MaskCLIP: MaskCLIP inherits open-vocabulary, fine-grained, and corruption-robust segmentation capabilities from CLIP’s retained visual-language features.The paper also presents key smoothing, which uses similar key features to encourage similar local predictions.

4 Experiments

Experiments evaluate annotation-free and zero-shot segmentation across standard benchmarks, qualitative web images, robustness to corruption, and ablations. MaskCLIP+ consistently improves segmentation, including large gains on unseen classes, while some smoothing strategies introduce class-dominance failures.

  • Annotation-Free Segmentation: Experiments evaluate annotation-free segmentation on PASCAL Context and COCO Stuff without annotations, using mIoU and qualitative results.The baseline uses dense CLIP image-encoder features directly; MaskCLIP and MaskCLIP+ are compared on standard datasets.
  • Annotation-Free Segmentation: MaskCLIP+ yields the best qualitative results, while prompt denoising removes distraction classes and key smoothing produces less noisy but class-dominated outputs.The qualitative comparison is performed on PASCAL Context without annotation.
  • Annotation-Free Segmentation: 31.1 mIoU is obtained by MaskCLIP+(ViT-B/16) on PASCAL Context without human annotation.MaskCLIP+ uses MaskCLIP predictions as pseudo labels, and its outputs are less noisy and more accurate than MaskCLIP.
  • Annotation-Free Segmentation: MaskCLIP and MaskCLIP+ segment unseen web-image concepts without annotation, including fine-grained cars, celebrities, and animation characters.The examples cover cars differing in colors or imagery properties as well as named people and characters.
  • Robustness Under Corruption: MaskCLIP is evaluated on PASCAL Context under ImageNet-C corruptions, and a supervised FCN baseline is included for robustness comparison.The supplementary results report particularly strong performance for MaskCLIP under Gaussian, shot, and impulse noises.
  • Zero-Shot Segmentation: 50.5, 24.4, and 46.0 percentage-point gains in unseen mIoU are reported on PASCAL VOC, COCO Stuff, and PASCAL Context, respectively, over previous SOTA.MaskCLIP+ is compared with methods including SPNet, ZS3Net, CaGNet, and STRICT; the benchmark setup uses standard segmentation architectures.
  • Zero-Shot Segmentation: 86.1 unseen mIoU is reached on PASCAL VOC after self-training, up from 72.8 with MaskCLIP-guided learning and 3.7 with the adapted DeepLabv2.Self-training slightly lowers seen-class mIoU from 89.5 to 88.8, partially due to model drifting.

5 Conclusion

The paper demonstrates that CLIP can support pixel-level semantic segmentation without retraining, and introduces MaskCLIP+ to provide pseudo labels for more flexible architectures. The conclusion also highlights strong zero-shot performance and segmentation of challenging unseen concepts.

  • Conclusion: MaskCLIP directly applies CLIP’s image encoder to semantic segmentation without re-training, while MaskCLIP+ uses training-time pseudo labels for broader architectures.MaskCLIP+ extends the approach beyond the CLIP image encoder by supplying pseudo labels for unlabeled pixels.
  • Conclusion: MaskCLIP+ significantly improves previous state-of-the-art results on standard transductive zero-shot segmentation benchmarks.The method is described as applicable to segmentation-tailored architectures through pseudo-label training.
  • Conclusion: MaskCLIP+ can segment challenging unseen classes such as celebrities and animation characters.The paper presents this capability as an important extension beyond standard annotation-free segmentation benchmarks.
  • Conclusion: PASCAL Context lacks bear and teddy bear classes, and MaskCLIP predicts teddy bear pixels as bedclothes in the qualitative examples.The figure also includes a key-smoothing failure where every pixel is labeled as horse.

A Qualitative Results on Annotation-Free Segmentation

MaskCLIP and MaskCLIP+ produce annotation-free segmentations across fine-grained objects, imagery properties, novel concepts, car brands, and sports. MaskCLIP+ also separates athletes from audiences in sports scenes.

  • MaskCLIP+ yields the best PASCAL Context qualitative results through pseudo-label training, while key smoothing can produce class-dominated predictions.
  • MaskCLIP and MaskCLIP+ segment fine-grained classes, imagery-specific objects, and novel concepts without annotations.Examples include red or yellow cars, blurry cars, Batman, and Bill Gates.
  • MaskCLIP and MaskCLIP+ yield reasonable annotation-free segmentations of different car brands and sports in Web images.
  • MaskCLIP separates athletes playing basketball or football from audience members while distinguishing the two sports.

B Robustness Results on Annotation-Free Segmentation

MaskCLIP is evaluated under ImageNet-C corruptions across all severity levels and compared with a fully supervised ViT-B/16 FCN baseline. Its robustness is especially pronounced for several noise corruptions.

  • CLIP-ViT-B/16 consistently outperforms CLIP-ResNet-50 by large margins across the corruption-severity evaluation.
  • MaskCLIP performs particularly well on Gaussian, shot, and impulse noises compared with a fully supervised ViT-B/16 FCN baseline.
  • MaskCLIP is evaluated on PASCAL Context under ImageNet-C corruptions across all severity levels using mIoU.

C Quantitative Results on Zero-Shot Segmentation

MaskCLIP+ preserves performance on seen classes while achieving strong zero-shot segmentation results across evaluation metrics and supporting larger target vocabularies and input-resolution analyses.

  • Pseudo-label guidance instead of feature-matching distillation does not affect MaskCLIP+ performance on seen classes across three standard datasets.
  • MaskCLIP+ surpasses previous state-of-the-art methods by large margins in overall and unseen pAcc/mAcc, approaching fully supervised baselines.
  • pAcc and mAcc penalize false negatives but not false positives, whereas mIoU penalizes both.
  • MaskCLIP+ is more balanced between seen and unseen classes than methods that are more confident on seen classes.
  • Figure 7 evaluates open-vocabulary segmentation with a larger target text set.
  • Table 8 evaluates MaskCLIP on PASCAL Context across input resolutions and with a multi-scale ensemble.

D Vocabulary Used in Open-Vocabulary Segmentation

The open-vocabulary segmentation example uses named fictional characters as foreground classes and Cityscapes categories as background classes, exposing confusion from shared local features.

  • The larger-vocabulary example uses Batman, Joker, James Gordon, The Penguin, Robin, Alfred, Catwoman, and Harley Quinn as foreground classes.
  • All Cityscapes classes except person serve as background classes in the example.
  • Shared local features cause Batman’s jaw to be segmented as James Gordon and part of Joker’s suit to be classified as The Penguin.

E Input Resolution and Multi-Scale Ensemble

MaskCLIP faces a resolution trade-off: matching CLIP’s 224x224 input preserves resolution and positional-encoding compatibility but produces smaller outputs. A 336x336 input is an empirical sweet spot, while multi-scale ensembles mitigate the resolution problem.

  • 336x336 is the empirically identified input-resolution sweet spot for MaskCLIP.
  • 224x224 preserves resolution and positional-encoding matching with CLIP but yields smaller outputs.
  • Multi-scale ensembles mitigate MaskCLIP’s input-resolution problem.

F Pseudo Code of MaskCLIP+

MaskCLIP+ trains a target segmentation model in two stages: MaskCLIP-guided learning followed by self-training. Both stages use pseudo labels and cross-entropy optimization with SGD updates.

  • Initialization: MaskCLIP+ initializes a target model with ImageNet-pretrained weights and loads target-class text embeddings into its classifier.
  • MaskCLIP-guided learning: During MaskCLIP-guided learning, the target model is trained against pseudo labels generated by MaskCLIP.
  • MaskCLIP-guided learning: Each guided-learning iteration computes cross-entropy loss between model predictions and MaskCLIP pseudo labels, then updates the model with SGD.
  • Self-training: After guided learning, self-training uses pseudo labels generated by the current target model for further optimization.
  • Self-training: Each self-training iteration applies cross-entropy loss to current-model predictions and self-generated pseudo labels before an SGD update.
Loading 2112.01071v2…