PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning

Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip H. S. Torr, Ser-Nam Lim

arXiv:2212.04994v1cs.CV

TL;DR

Pixel-level segmentation annotations are costly, and standard CLIP training does not ensure alignment between image patches and text. PACL modifies contrastive compatibility to align vision patches with text, enabling annotation-free zero-shot segmentation. It outperforms prior methods across four segmentation datasets and improves zero-shot classification across 12 datasets.

  • Problem

    Pixel-level segmentation annotations are costly, and CLIP’s contrastive objective aligns image and text CLS tokens without ensuring patch-level alignment.

    Source: §1 · pp. 1–2

  • Method

    PACL modifies image-text contrastive compatibility to align vision patch tokens with the text CLS token.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    Across four zero-shot segmentation datasets, PACL with pre-trained CLIP beats previous approaches (72.3 mIoU on Pascal VOC) and improves classification accuracy across 12 datasets.

    Source: §1 · p. 2, §6.1 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The learned patch-text alignment lets models identify text-relevant image regions for zero-shot segmentation without segmentation annotations or masks during training.

    Source: §7 · p. 9

    Training models from scratch with PACL as an independent contrastive method and exploring image-text retrieval remain outside this work’s scope.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabulary zero-shot segmentation on 4 different segmentation benchmarks: Pascal VOC, Pascal Context, COCO Stuff and ADE20K. Furthermore, we show that PACL is also applicable to image-level predictions and when used with a CLIP backbone, provides a general improvement in zero-shot classification accuracy compared to CLIP, across a suite of 12 image classification datasets.

1. Introduction

Pixel-level segmentation annotations are costly, while existing open-vocabulary segmentation methods either rely on such annotations or cannot leverage pre-trained vision encoders. PACL aligns image patches with text so image-text-trained models can transfer to zero-shot segmentation, with reported gains across segmentation and classification benchmarks.

1. Introduction

  • Pixel-level annotations require substantial human effort and expense, impeding the large-scale training of supervised segmentation models.Existing annotation-free unsupervised methods are nevertheless outperformed by fully supervised counterparts on most segmentation benchmarks.Source: §1 · p. 1
  • CLIP’s standard contrastive loss aligns image and text CLS tokens but does not ensure patch-level alignment, a prerequisite the authors target for dense predictions.Earlier open-vocabulary segmentation approaches include methods that require segmentation annotations or masks, while GroupViT cannot leverage pre-trained vision encoders.Source: §1 · p. 2
  • PACL with pre-trained CLIP consistently beats previous zero-shot segmentation baselines on four datasets, including methods trained with segmentation annotations or masks.The reported datasets are Pascal VOC, Pascal Context, COCO Stuff, and ADE20K; PACL also improves zero-shot classification across 12 datasets.Source: §1 · p. 2
  • PACL modifies contrastive compatibility to align vision patch representations with the text CLS representation, enabling text-conditioned region identification and zero-shot open-vocabulary segmentation.The compatibility function weights vision tokens by their cosine similarity to the text CLS token.Source: §1 · p. 2

2. Related Work

Related work spans supervised segmentation, annotation-free methods using self-supervised features, and language-supervised models with open-vocabulary potential. Prior open-vocabulary segmentation methods vary in their annotation requirements and in whether they can use pre-trained vision encoders.

Supervised semantic segmentation

  • Supervised segmentation datasets provide pixel-level class labels, but annotation expense leaves them limited to a relatively small number of classes.The cited examples include Pascal VOC, ADE20K, and Cityscapes.Source: §2 · p. 2

Unsupervised semantic segmentation

  • Unsupervised segmentation methods leverage self-supervised features to segment without dense annotations, but such approaches remain difficult to scale to many visual concepts.STEGO uses localized feature correspondences in self-supervised models such as DINO.Source: §2 · pp. 2–3

Natural language supervision

  • Large-scale image-text models transfer to downstream tasks and are expected to recognize visual concepts expressible in natural language.Examples of downstream tasks include image-text retrieval, visual question answering, zero-shot image classification, and object detection.Source: §2 · p. 3

Natural language supervision for zero-shot segmentation

  • Existing open-vocabulary segmentation methods differ in their training supervision: LSeg uses segmentation annotations, OpenSeg uses class-agnostic annotations, and ViL-Seg and GroupViT use image-text data alone.GroupViT introduces a modified ViT architecture for grouping tokens, whereas this work modifies CLIP compatibility to align vision patches with text CLS tokens.Source: §2 · p. 3

3. Patch Level Alignment in CLIP

The section tests whether pre-trained CLIP patch tokens align with text by applying its usual zero-shot classification method at the patch level. The reported accuracies indicate poor patch-level alignment, despite CLIP’s image-level capabilities.

3. Patch Level Alignment in CLIP

  • Patch-level alignment is evaluated by classifying each dataset patch with the standard zero-shot method and treating accuracy as a measure of vision-text alignment.The method applies the usual cosine-similarity classification to each patch by using a vision encoder that returns token representations.Source: §3 · p. 3
  • 52.49% for ViT-B/16 and 27.91% for ViT-L/14 are the patch classification accuracies reported for pre-trained CLIP on Pascal VOC.The authors interpret these low accuracies as evidence of poor patch-level image-text alignment; the table caption says PACL increases accuracy for both encoders.Source: §3 · pp. 3–4, Table 1 · p. 4
  • ViT-L/14, despite better image-level prediction performance than ViT-B/16, has lower patch-level accuracy, and the authors conclude pre-trained CLIP alone cannot support open-vocabulary segmentation.The section also points to a CLIP-over-DINO semantic-coherence comparison, without giving its numerical values here.Source: §3 · p. 4, Figure 3 · p. 4

4. Semantic Coherence in Vision Encoders

The section evaluates whether CLIP vision patches are semantically coherent enough to support patch-level alignment with text. CLIP ViT-B/16 outperforms DINO on the reported semantic-coherence test, with qualitative comparisons also favoring CLIP or showing it at par.

4. Semantic Coherence in Vision Encoders

  • Patch pairs are classified as sharing a target label when their cosine similarity supports that prediction, with patch labels assigned by majority vote over segmentation labels.The authors use performance on this binary task as an indicator of semantic coherence.Source: §4 · p. 4
  • Figure 4 visualizes cosine similarity from a marked dog-image patch to patches in the same image and a different dog image for both encoders.Panels b–c show CLIP; panels d–e show DINO.Source: Figure 4 · p. 4
  • CLIP ViT-B/16 outperforms DINO on semantic coherence by AUROC and is qualitatively at par or better on same- and different-image dog patches.Figure 4 compares patch-level cosine similarities for CLIP and DINO using a marked patch in one dog image.Source: §4 · p. 4, Figure 4 · p. 4

5. Patch Aligned Contrastive Learning (PACL)

PACL modifies CLIP’s contrastive compatibility function to align vision patch tokens with the text CLS representation despite the lack of patch-level text annotations. The method uses similarity-weighted patch features in an InfoNCE objective, and the reported patch-level classification results show markedly improved alignment over vanilla CLIP.

5. Patch Aligned Contrastive Learning (PACL)

  • Because image–text datasets do not provide ground-truth text for each patch, the alignment is learned weakly by modifying CLIP’s contrastive loss.The method replaces CLS vision features with patch tokens and embeds each patch in the shared space.Source: §5 · pp. 4–5
  • Applying CLIP’s semantic coherence to unsupervised segmentation approaches such as STEGO is identified as a candidate direction, but further study is outside this work’s scope.Source: §5 · p. 4
  • PACL computes patch-to-text cosine similarities, softmaxes them across patches, and uses the resulting weights to pool vision embeddings before computing image–text compatibility.The modified compatibility function is trained with InfoNCE.Source: §5 · p. 5
  • PACL produces a stark improvement in patch-level alignment over vanilla CLIP on the reported patch classification task.The authors relate the improvement to InfoNCE attending to image regions corresponding to the text and assigning them higher similarity.Source: §5 · p. 5
  • Unlike the related phrase-grounding approach [18], PACL uses CLIP patch tokens rather than detector-provided ROI features and fits CLIP’s contrastive setting.The cited approach targets weakly supervised phrase grounding, whereas PACL targets zero-shot open-vocabulary semantic segmentation.Source: §5 · p. 5
  • At inference, the compatibility function supports image-level predictions, while class-specific patch similarities form segmentation masks followed by a softmax across classes.The passage describes predictions for a supplied set of class names.Source: §5 · pp. 5–6

6. Experiments & Discussion

PACL is evaluated for zero-shot semantic segmentation and image classification using pre-trained encoders. It outperforms segmentation baselines across four datasets and is competitive with or better than vanilla CLIP on most of 12 classification datasets.

6.1. Zero-shot Semantic Segmentation

  • PACL keeps the pre-trained image and text encoders frozen and trains only a small vision embedder, using CLIP’s patch representations for segmentation transfer.The vision embedder is a residual block with two linear layers in its main branch and one in its residual branch.Source: §6.1 · p. 6
  • Training uses about 30M samples from public image-text datasets, with extracted nouns and prompted sentences added to descriptions, and no segmentation annotations or masks.The datasets are GCC3M, GCC12M and YFCC15M.Source: §6.1 · p. 6
  • A 4 × 4 patch-extraction stride at inference provides finer-grained patches, and the resulting segmentations are upscaled with bilinear interpolation.Source: §6.1 · p. 6
  • PACL with DINO ViT-B/16 reaches 55.4 mIoU on Pascal VOC, while CLIP ViT-B/16 consistently outperforms ViT-L/14 and DINO ViT-B/16.The authors relate the encoder ranking to semantic coherence and identify it as an important factor for PACL's success.Source: §6.1 · p. 7, Table 3 · p. 8
  • mIoU scores of 72.3, 50.1, 38.8 and 31.4 on Pascal VOC, Pascal Context, COCO Stuff and ADE20K show CLIP + PACL outperforming prior approaches under stricter annotation and class-evaluation constraints.Table 2 reports the four-dataset comparison; the method uses no segmentation annotations and is evaluated on all classes.Source: §6.1 · p. 7, Table 2 · p. 8

6.2. Image Classification & Future Work

  • PACL + CLIP outperforms vanilla CLIP on 10 of 12 datasets with ViT-B/16 and 7 of 12 with ViT-L/14, and generally improves results on ImageNet and its distribution shifts.The ImageNet exception is ImageNet-R with ViT-L/14.Source: §6.2 · p. 8, Figure 8 · p. 8
  • Training PACL from scratch and testing it on image-text retrieval are left for future work, as are studies of cross-attention fusion for patch alignment and dense prediction.Source: §6.2 · p. 8

7. Conclusion

The conclusion presents PACL as a patch-to-text alignment objective for zero-shot semantic segmentation without segmentation annotations or masks. It also reports segmentation gains across four datasets and classification improvements across 12 datasets with CLIP.

7. Conclusion

  • PACL aligns ViT vision patch tokens with the text CLS token, enabling zero-shot semantic segmentation without segmentation annotations or masks and outperforming prior approaches on four datasets.With a pre-trained CLIP encoder, PACL also improves classification accuracy across 12 image-classification datasets.Source: §7 · pp. 8–9

A. Additional Implementation Details

A.1. Vision Embedder Architecture

The vision embedder is a small trainable component on top of frozen pre-trained CLIP encoders.

A.1. Vision Embedder Architecture

  • Only 1.1M vision-embedder parameters are trained atop frozen CLIP encoders, using a residual block with two main-branch linear layers and one residual linear layer.The full CLIP ViT-B/16 architecture has 150M parameters, and frozen encoders require no gradient propagation.Source: §A.1 · p. 12

A.2. Training details for Vision Embedder

This appendix section covers training details for the vision embedder and the prompt engineering used with a pre-trained CLIP text encoder.

A.2. Training details for Vision Embedder

  • The appendix describes pre-trained encoder architectures, PACL training hyperparameters, and CLIP prompt engineering for the text encoder.Source: §A.2 · p. 12

A.2.1 Architecture and Hyperparameters

PACL trains a vision embedder on top of a pre-trained vision encoder, projecting patch-wise representations into the shared multimodal space. The setup supports multiple encoder combinations and uses prompt-averaged text embeddings for inference.

A.2.1 Architecture and Hyperparameters

  • PACL trains a vision embedder over pre-trained CLIP ViT-B/16 or ViT-L/14 encoders, and also tests DINO ViT-B/16 with CLIP ViT-B/16 text.The models are trained using the vision embedder described in the appendix.Source: §A.2.1 · p. 12
  • The vision embedder maps the encoder’s patch-wise representations into the shared multimodal space.It operates on the vision encoder’s token or patch representations.Source: Figure 9 · p. 12
  • Training uses image-text data for 10 epochs, batch size 4096, and AdamW, without segmentation annotations or class-agnostic masks.The stated setup uses four NVIDIA A100 GPUs and a cosine annealing learning-rate schedule.Source: §A.2.1 · p. 12

A.2.2 Prompt Engineering

  • At inference, class text embeddings are averaged across seven prompts and compared by cosine similarity with vision patch representations.The prompt set uses varied class-name templates; vanilla CLIP instead compares against the vision encoder’s CLS token.Source: §A.2.2 · p. 12

A.3. Training and Evaluation Datasets

PACL is trained using image-text datasets and evaluated on segmentation and image-classification benchmarks. Additional qualitative comparisons report that CLIP’s vision encoder is competitive with or better than DINO in semantic coherence.

A.3. Training and Evaluation Datasets

  • Training data include GCC-3M, GCC-12M, and a 15-million-sample YFCC-100M subset, with GCC-12M intended primarily for pre-training and GCC-3M described as less noisy for fine-tuning.The datasets consist of web image-text pairs.Source: §A.3 · pp. 12–13
  • Zero-shot segmentation is evaluated on Pascal VOC, Pascal Context, COCO Stuff, and ADE20K using mean intersection over union.The passage specifies each benchmark’s class and validation-image counts.Source: §A.3 · p. 13
  • Zero-shot classification is evaluated on 12 datasets: ImageNet, four ImageNet distribution shifts, and seven other image-classification datasets.The seven include CIFAR-10, CIFAR-100, Stanford Cars, Caltech-101, Food-101, Oxford-IIIT Pets, and Flowers.Source: §A.3 · p. 13

B.1. Semantic Coherence in CLIP

  • CLIP ViT-B/16 performs competitively or better than DINO in semantic coherence, covering more patches of relevant objects as similar to a selected patch.Semantic coherence means semantically similar image regions have similar patch-level representations.Source: §B.1 · p. 13

B.2. Qualitative Segmentation Results

Qualitative examples illustrate zero-shot segmentation predictions on Pascal VOC and ADE20K. PACL broadly outperforms vanilla CLIP on zero-shot classification across the evaluated datasets.

B.3. Zero-shot Image Classification

  • PACL with ViT-B/16 outperforms vanilla CLIP on 10 of 12 classification datasets, and PACL with ViT-L/14 does so on 7 of 12.Table 4 compares the models across eight standard classification datasets and four ImageNet distribution shifts.Source: Table 4 · p. 14, §B.3 · p. 14

B.2. Qualitative Segmentation Results

  • Figure 10 compares CLIP and DINO patch-similarity heatmaps for selected patches across same-class images.The examples include birds, aeroplanes, and sheep; the caption describes panels for the original image and each encoder’s similarities.Source: Figure 10 · p. 14
  • Qualitative examples show zero-shot segmentation predictions for varied concepts on Pascal VOC and indoor and outdoor scenes on ADE20K, despite no segmentation-annotation or mask training.Figure 11 presents original images, ground-truth labels, and predictions from the best-performing PACL model; Figure 12 gives corresponding ADE20K examples.Source: §B.2 · pp. 13–14, Figure 11 · p. 14

Related papers

Loading 2212.04994v1…