PaperlayerView exact revision on arXiv

Source-linked AI summary

GroupViT: Semantic Segmentation Emerges from Text Supervision

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang

arXiv:2202.11094v5cs.CV

TL;DR

Pixel-supervised segmentation is costly and tied to labeled categories, motivating a model that can learn semantic segments from text alone. GroupViT uses hierarchical visual grouping with jointly trained image-text contrastive learning, achieving zero-shot segmentation at 52.3% mIoU on PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context without fine-tuning.

  • Problem

    Pixel-supervised segmentation requires costly per-pixel labels and cannot generalize beyond its labeled categories, motivating zero-shot segmentation across vocabularies from text alone.

  • Method

    GroupViT hierarchically groups visual tokens into irregular-shaped segments and trains jointly with a text encoder on paired image-text data using contrastive losses.

  • Results

    52.3% mIoU on PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context are achieved for zero-shot semantic segmentation without fine-tuning.

  • Takeaways & Limitations

    Text supervision can transfer beyond image classification to finer-grained semantic segmentation across different label vocabularies without pixel-wise labels.

  • Takeaways & Limitations

    Performance is lower on PASCAL Context because background classes are less likely to appear in training text, and the architecture lacks segmentation-specific enhancements.

Abstract

from arXiv · show

Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision. We open-source our code at https://github.com/NVlabs/GroupViT .

1. Introduction

GroupViT addresses the annotation and category-generalization limits of pixel-supervised segmentation by learning hierarchical semantic grouping from image-text pairs. The resulting model transfers to zero-shot semantic segmentation without pixel-level labels or fine-tuning.

  • Pixel-supervised segmentation requires costly per-pixel labels and is restricted to labeled categories, limiting generalization to unseen ones.
  • The paper asks whether semantic segmentation can be learned purely from text supervision without per-pixel annotations while generalizing across object vocabularies.
  • GroupViT hierarchically merges visual tokens into irregular-shaped segments through Transformer stages and grouping modules.Its global self-attention supports non-grid-like segments, while successive stages merge smaller segments into larger ones.
  • GroupViT and a jointly trained text encoder learn from paired image-text data using contrastive losses, with image-level embeddings compared against textual embeddings.Positive pairs use corresponding image-text examples, while text from other images supplies negative pairs.
  • 52.3% mIoU on PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context are achieved in zero-shot semantic segmentation without fine-tuning.The model is trained with text supervision alone and transfers to segmentation by matching image segments with label-text embeddings.
  • The contributions include introducing hierarchical bottom-up grouping, learning without pixel-level labels, and establishing a baseline for zero-shot segmentation across vocabularies.

2. Related Work

Prior work includes grid-based Vision Transformers, text-supervised representation learning, and weakly supervised segmentation. GroupViT differs by learning grouping and segmentation from noisy public text supervision without human annotations.

  • Vision Transformer: Most Vision Transformer variants retain grid-like representations, whereas GroupViT learns arbitrary-shaped visual segments through hierarchical grouping.
  • Representation Learning with Text Supervision: Text-supervised representation learning has transferred successfully to downstream recognition tasks and categories beyond ImageNet.
  • Weakly-supervised Semantic Segmentation: Weakly supervised semantic segmentation reduces supervision but still relies on manual labels, finite vocabularies, and carefully curated datasets.
  • Related Text-supervised Segmentation: Unlike concurrently developed methods using private datasets with 400M–1.8B image-text pairs, GroupViT trains from scratch on noisier public data totaling 30M images while achieving competitive performance.

3. Method

GroupViT is a hierarchical Transformer that progressively groups visual tokens into irregular-shaped segments, then learns image representations from paired text using contrastive supervision. Its grouping blocks use hard assignments to merge similar segments, while multi-label text prompting supplies additional positive supervision.

  • Grouping Vision Transformer: GroupViT hierarchically groups visual tokens into progressively larger arbitrary-shaped segments through Transformer stages and grouping blocks.Each stage merges smaller groups into larger ones rather than preserving a regular image-grid structure.
  • Grouping Vision Transformer: Each grouping block compares learned group tokens with image-segment tokens and merges tokens assigned to the same group into new segment tokens.Assignments are computed with Gumbel-Softmax and converted to one-hot hard assignments.
  • Grouping Vision Transformer: Hard assignment produces more effective grouping than soft assignment in the reported experiment.The comparison is reported in Table 1.
  • Learning from Image-Text Pairs: GroupViT trains jointly with a Transformer text encoder using image-text contrastive learning that pulls matched pairs together and pushes unmatched pairs apart.The final image embedding averages output segment tokens, while text is encoded into the same embedding space.
  • Learning from Image-Text Pairs: Multi-label contrastive learning generates additional text labels by extracting nouns and prompting them with sentence templates.This provides each image with multiple positive text pairs instead of only one matched pair in the batch.

4. Experiments

Experiments evaluate GroupViT’s training setup, grouping variants, and zero-shot segmentation behavior. Hierarchical grouping, hard assignment, multi-label contrastive learning, and increased group tokens improve segmentation-related outcomes, while visualizations show progressively structured semantic concepts.

  • Implementation Details: GroupViT uses ViT-S with 12 Transformer layers and evaluates 1-stage and 2-stage grouping architectures that both output 8 tokens.The 1-stage model groups after layer 6, whereas the 2-stage model groups after layers 6 and 9.
  • Ablation Study: Adding multi-label contrastive loss improves performance by 13.1% with hard assignment and 2.6% with soft assignment.The loss adds noun-based text labels and uses a prompted format during training and inference.
  • Ablation Study: Increasing group tokens consistently improves performance, while 8 output tokens are found optimal.The paper interprets group tokens as representations of distinct semantic concepts that can encode multiple concepts in a 384-D space.
  • Ablation Study: The 2-stage model improves mask mIoU by 1.8% and boundary mIoU by 1.9% over the 1-stage variant.With combined CC and YFCC training data, the 2-stage model improves both metrics by approximately 7%, and produces smoother segmentation maps.
  • Visualization: Qualitative results show plausible segmentation for single objects, multiple same-class objects, and multiple objects from different classes.Visualizations indicate that first-stage group tokens attend to mid-level concepts, while second-stage tokens represent higher-level concepts that can aggregate earlier concepts.
  • Zero-Shot Transfer: Zero-shot transfer compares GroupViT with zero-shot baselines and fully supervised transfer on PASCAL VOC 2012 and PASCAL Context.The evaluation reports mIoU on validation splits, and the paper states that GroupViT outperforms ViT trained with CLIP by a large margin while remaining comparable to ViT on ImageNet classification.

5. Discussion

GroupViT transfers representations learned from noisy image-text pairs to semantic segmentation without explicit human supervision or fine-tuning. The authors identify lower PASCAL Context performance and missing segmentation-specific architectural enhancements as future-work areas.

  • Conclusion: GroupViT transfers representations learned from large-scale noisy image-text pairs to semantic segmentation in a zero-shot manner.The conclusion frames this as learning semantic segmentation with text alone and without explicit human supervision.
  • Conclusion: Text supervision transfers beyond image classification to finer-grained vision tasks, opening a research direction.The authors describe this transfer as not previously explored in this context.
  • Limitations and Future Work: PASCAL Context performance is lower than PASCAL VOC because background classes are less likely to be labeled in text and correctly grouped segments can receive incorrect textual classes.The limitation specifically mentions ground and road as examples of background classes.
  • Limitations and Future Work: GroupViT does not currently integrate segmentation-specific enhancements such as dilated convolutions, pyramid pooling, or a U-Net.These components are identified as potential future improvements.

A.1. Architecture

The implementation uses a ViT-S-based GroupViT and a jointly trained Transformer text encoder, with MLP-Mixer connections between grouping stages. Fully supervised baselines append a 1×1 convolution for pixel-wise classification and use specified augmentation and optimization settings.

  • Architecture: GroupViT is based on ViT-S with 12 Transformer layers, each containing multi-head self-attention, an MLP block, and layer normalization.Group tokens across grouping stages are connected through MLP-Mixer layers.
  • Baselines: Fully supervised transfer baselines fine-tune a pre-trained ViT with an appended 1×1 convolutional layer for pixel-wise classification.Training uses random image scaling from [0.5, 2] and random 224×224 crops.
  • Baselines: The baseline training procedure uses Adam, weight decay 0.05, learning rate 0.001, and 4k iterations with a specified batch size.The supplied passage truncates after introducing the batch-size setting.

B. Qualitative Results

Qualitative results show GroupViT segmenting both object and context-related stuff classes on PASCAL datasets. The examples include single objects, repeated categories, mixed categories, and object-context combinations.

  • PASCAL VOC 2012: GroupViT successfully groups and correctly classifies single objects and multiple objects across same-category and different-category scenarios on PASCAL VOC 2012.The examples are presented as challenging qualitative cases.
  • PASCAL Context: On PASCAL Context, GroupViT segments both object classes such as cat and dog and stuff classes such as window and water.The passage gives examples combining objects with contextual regions.

C. Additional Experiments and Analysis

Additional analyses compare GroupViT with ViT for text-supervised ImageNet classification and evaluate generated masks against baseline masks. The supplied materials identify the ImageNet comparison but do not provide mask-analysis outcomes.

  • ImageNet Classification: GroupViT and ViT are compared on ImageNet using text-only image-text contrastive supervision.The evaluation reports zero-shot and linear probing accuracy on the ImageNet validation split following the CLIP setting.
  • ImageNet Classification: GroupViT’s ImageNet classification performance is comparable to, or better than, ViT under the described text-supervised evaluation.The passage presents this as evidence that the grouping mechanism enhances the baseline.
  • Mask Analysis: The mask analysis evaluates GroupViT and ViT by deriving attention masks and computing Jaccard similarity for attention heads.The supplied passage truncates before reporting the resulting comparison.

C.3. Limitations

GroupViT’s PASCAL Context performance is constrained mainly by classification of background groups, even when those groups are correctly formed. Qualitative results and oracle comparisons indicate that grouping is stronger than assigning groups to textual classes.

  • GroupViT’s mIoU is significantly lower on PASCAL Context than on PASCAL VOC.
  • Qualitative examples show GroupViT grouping multiple objects and background regions into semantic groups before labels are assigned.The examples include single objects, multiple objects of the same or different categories, and stuff classes.
  • Correctly grouped background regions such as ground, road, and wall can still be assigned incorrect classes through image-text similarity.The text notes these classes are less likely to appear in training descriptions and reports low IoU for some background classes.
  • The large gap between original and oracle mIoU on PASCAL Context indicates room to improve classification of groups into segmentation labels.The oracle assigns each output group the class with maximum ground-truth-mask IoU, using ground-truth masks to label groups.

C.4. COCO Dataset

On COCO, GroupViT is evaluated for zero-shot semantic segmentation across 80 object classes, with qualitative examples illustrating object and stuff segmentation and a documented stuff-class failure mode.

  • GroupViT transfers to COCO semantic segmentation, which contains 80 object classes, by combining instance masks from the same category.Performance is reported using semantic-segmentation mIoU.
  • Qualitative PASCAL Context examples show GroupViT segmenting both object and stuff classes, including cat and window or dog and water.
  • A PASCAL Context failure case shows correctly grouped stuff regions being assigned incorrect classes through visual-text embedding similarity.The cited example includes ground, road, and wall.

C.5. Training on RedCaps

Training GroupViT on RedCaps provides a generalization check across image-text datasets, yielding similar zero-shot segmentation accuracy to filtered YFCC on several benchmarks.

  • Replacing filtered YFCC with RedCaps yields similar zero-shot accuracy on PASCAL VOC, PASCAL Context, and COCO.RedCaps contains 12 million Reddit image-text pairs and is described as similar in size to filtered YFCC.
  • The RedCaps experiment demonstrates transfer to multiple image-segmentation benchmark datasets.
  • The comparison evaluates whether GroupViT’s results generalize across alternative training datasets beyond Conceptual Captions and filtered YFCC.
Loading 2202.11094v5…