PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Vocabulary Universal Image Segmentation with MaskCLIP

Zheng Ding, Jieke Wang, Zhuowen Tu

arXiv:2208.08984v2cs.CV

TL;DR

Open-vocabulary universal image segmentation asks models to segment arbitrary text-described categories across semantic, instance, and panoptic tasks, beyond fixed training labels. MaskCLIP integrates mask tokens with a pretrained ViT-based CLIP encoder in an encoder-only Transformer design, and reports encouraging results across these tasks while avoiding heavy retraining. The paper also reports faster mask-feature extraction than a CLIP baseline and notes confusion between similar open-vocabulary classes.

  • Problem

    Existing segmentation methods use fixed categorical definitions, motivating segmentation for arbitrary text-described categories beyond the training set.

  • Method

    MaskCLIP is a Transformer-based encoder-only approach that couples a pretrained ViT-based CLIP image encoder with mask tokens and Relative Mask Attention for mask refinement and class prediction.

  • Results

    MaskCLIP achieves encouraging results for open-vocabulary semantic, instance, and panoptic segmentation and extracts features for 100 masks in ~0.6s versus ~3s for the CLIP baseline on a single 3090 GPU.

  • Takeaways & Limitations

    Pretrained CLIP partial/dense features can support a unified open-vocabulary segmentation approach without the time-consuming student-teacher training process.

  • Takeaways & Limitations

    Similar classes may be mistakenly classified in the open-vocabulary setting, such as predicting house when the ground truth is building.

Abstract

from arXiv · show

In this paper, we tackle an emerging computer vision task, open-vocabulary universal image segmentation, that aims to perform semantic/instance/panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time. We first build a baseline method by directly adopting pre-trained CLIP models without finetuning or distillation. We then develop MaskCLIP, a Transformer-based approach with a MaskCLIP Visual Encoder, which is an encoder-only module that seamlessly integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction. MaskCLIP learns to efficiently and effectively utilize pre-trained partial/dense CLIP features within the MaskCLIP Visual Encoder that avoids the time-consuming student-teacher training process. MaskCLIP outperforms previous methods for semantic/instance/panoptic segmentation on ADE20K and PASCAL datasets. We show qualitative illustrations for MaskCLIP with online custom categories. Project website: https://maskclip.github.io.

1. Introduction

Open-vocabulary universal image segmentation extends semantic, instance, and panoptic segmentation to arbitrary text-described categories. MaskCLIP addresses this setting by combining pretrained CLIP features with mask-token-based Transformer processing.

  • Motivation: Existing panoptic and instance segmentation methods use fixed category definitions represented as categorical labels without semantic relations.This limits their handling of categories outside the predefined label set.
  • Motivation: Open-vocabulary universal image segmentation targets categories beyond the training set using text-based descriptions at inference time.It unifies semantic segmentation for background regions and instance segmentation for foreground objects under a broader setting.
  • Approach: MaskCLIP builds on pretrained CLIP image and text embeddings mapped to the same space, first establishing a training-free open-vocabulary panoptic baseline.The paper then develops a dedicated segmentation algorithm rather than relying only on direct CLIP adoption.
  • Approach: MaskCLIP uses a Transformer-based encoder-only design that couples a pretrained CLIP image encoder with a mask-token encoder for mask representation and class prediction.Its Relative Mask Attention module integrates mask tokens with the ViT-based CLIP backbone and performs mask refinement.
  • Results: MaskCLIP demonstrates encouraging and competitive results for open-vocabulary semantic, instance, and panoptic segmentation.The stated contribution covers all three segmentation tasks under the universal image segmentation setting.

2. Related Work

Related work spans open-vocabulary recognition, universal segmentation, class-agnostic mask generation, and reuse or distillation of CLIP representations. The paper emphasizes that these approaches differ in task formulation, training data, and testing scope.

  • Open vocabulary: Open-vocabulary methods seek recognition and detection for categories beyond those present in training data.Prior efforts include CLIP-based approaches for object detection and related vision tasks.
  • Universal segmentation: Universal segmentation unifies semantic, instance, and panoptic segmentation by treating stuff and instances similarly and producing masks for each.Earlier work generally handled these tasks separately with different methods.
  • Open-vocabulary universal segmentation: Open-vocabulary universal segmentation methods are not directly comparable when their problem definitions, training data, and testing scopes differ.The paper presents a summary of recent open-vocabulary applications to clarify these setting differences.
  • Open-vocabulary universal segmentation: Some prior methods have limited open-vocabulary scope because they rely on pseudo-mask supervision or image features tied to training-label categories.The paper identifies these constraints for XPM and LSeg, while describing OpenSeg as potentially applicable to instance or panoptic segmentation.
  • Class-agnostic segmentation: Class-agnostic segmentation removes class supervision to generate masks for novel classes that may appear during testing.This contrasts with class-aware methods that predict classes together with corresponding labels.
  • CLIP model distillation/reuse: CLIP reuse and distillation form a related line of work built on large-scale image-text paired captioning datasets and subsequent vision-language models.The cited methods include approaches that distill or adapt CLIP knowledge for detection and segmentation.

3. Method

MaskCLIP uses a two-stage pipeline that proposes class-agnostic masks, then refines them and extracts mask-specific CLIP features through a frozen ViT-based visual encoder.

  • Pipeline: The first stage generates class-agnostic mask proposals, while the second MaskCLIP Visual Encoder consumes images and coarse masks.The proposal network removes class supervision and uses binary positive/negative classification.
  • Mask Class Tokens: Mask Class Tokens extract partial/dense features for corresponding mask areas while preserving global image context without independently encoding each mask.The attention design lets mask tokens attend to their corresponding image regions while image tokens continue interacting with one another.
  • Mask Class Tokens: The mask attention matrix routes each Mask Class Token to image tokens selected by its corresponding mask, while preventing interactions among Mask Class Tokens.True positions are masked out, and False positions remain attendable.
  • Relative Mask Attention: Relative Mask Attention adds a mask-aware attention matrix to CLIP ViT attention using image tokens and mask patch tokens.The design changes the attention matrix rather than CLIP features directly, preserving the learned image-text feature relationship while incorporating detailed mask information.
  • Relative Mask Attention: The attention-derived mask residual updates coarse masks into refined masks through a learnable nonlinear mapping.The update uses the coarse mask, a residual, and sigmoid/inverse-sigmoid transformations.

4. Experiments

The experiments evaluate MaskCLIP across open-vocabulary semantic, instance, and panoptic segmentation using COCO training data and several external benchmarks. Results show stronger segmentation performance than baselines, improved efficiency through mask-token feature extraction, and some remaining confusion between similar classes.

  • Experimental Setup: The evaluation trains on COCO and tests open-vocabulary semantic, instance, and panoptic segmentation across ADE20K, PASCAL Context, and LVIS.ADE20K and PASCAL Context support semantic or panoptic evaluation, while LVIS supports instance segmentation.
  • Experimental Setup: MaskCLIP uses class-agnostic mask proposal networks, pretrained CLIP features, and mask-token baselines for open-vocabulary classification.The CLIP baseline masks each image and sends the resulting images to CLIP, whereas MaskCLIP w/o RMA uses Mask Class Tokens for partial or dense features.
  • Efficiency Analysis: ~4x faster: MaskCLIP w/o RMA extracts features for 100 masks in ~0.6s, compared with ~3s for the CLIP baseline on one 3090 GPU.MaskCLIP extracts multiple mask-area features in one CLIP run instead of independently processing N masked images.
  • Open-Vocabulary Semantic Segmentation: MaskCLIP outperforms both baselines on four open-vocabulary semantic-segmentation datasets and reaches state-of-the-art results on three benchmarks.The P-59 result is slightly lower than LSeg+; all reported semantic-segmentation results use mIoU.
  • Open-Vocabulary Panoptic Segmentation: MaskCLIP w/o RMA performs better on all reported panoptic-segmentation metrics, while MaskCLIP predicts floor correctly where the CLIP baseline predicts skyscraper.The qualitative difference is attributed in the passage to retaining global context information.
  • Open-Vocabulary Instance Segmentation: MaskCLIP outperforms the CLIP baseline and MaskCLIP w/o RMA in general under cross-dataset instance segmentation, and outperforms previous results across generalized COCO-split categories.Under the constrained COCO-split setting, it remains competitive on base and target categories.

5. Ablation Study

The ablations examine MaskCLIP with ground-truth mask proposals and mask refinement, while qualitative figures illustrate panoptic segmentation and user-specified classes.

  • Ground-Truth Mask Proposals: Ground-truth mask proposals substantially improve performance, and MaskCLIP outperforms OpenSeg in this setting.The setup isolates classification capability by removing mask-proposal quality effects.
  • Qualitative Results: Qualitative ADE20K results compare MaskCLIP with two baselines using the dataset’s 150 class names as text inputs.The examples use validation images and directly supplied class names.
  • Qualitative Results: User-specified category examples illustrate distinctions such as real versus toy rabbits and other fine-grained object properties.Bold labels denote novel classes absent from COCO training labels.
  • Mask Refinement: MaskCLIP performs slightly better with mask refinement than without it, particularly on segmentation quality measured by SQ.The ablation focuses on whether attention information can refine masks.

6. Conclusion

The paper concludes that MaskCLIP addresses open-vocabulary universal image segmentation by integrating mask queries with a pretrained ViT-based CLIP backbone. It reports encouraging results across semantic, instance, and panoptic segmentation.

  • Contribution: MaskCLIP tackles open-vocabulary universal image segmentation with mask queries and a ViT-based CLIP backbone.Its Relative Mask Attention module is integrated with the pretrained CLIP model.
  • Contribution: MaskCLIP integrates segmentation mask refinement and image feature extraction within an encoder module built on a pretrained CLIP image model.This distinguishes the approach from prior open-vocabulary semantic segmentation and object detection methods.
  • Results: The method obtains encouraging experimental results on open-vocabulary semantic, instance, and panoptic segmentation.

A. CLIP Baseline Details

The CLIP baseline combines class-agnostic mask proposals with CLIP image and text embeddings to classify each cropped mask region. Its algorithm computes a text embedding once, then scores each proposed region against all category names.

  • Baseline Pipeline: The baseline precomputes text embeddings for C category names and obtains N masks from a mask proposal network.The image has dimensions H × W × 3, while embeddings use hidden dimension D.
  • Baseline Pipeline: For each proposal, the baseline crops the image by element-wise masking, encodes the region visually, and compares it with text embeddings.The cropped region is R_i = M_i ⊙ I and its visual embedding is produced by the CLIP visual encoder.
  • Classification: The final classification score for each mask is a softmax over dot products between its visual embedding and the category text embeddings.
  • Components: Algorithm 1 requires a mask proposal network, a CLIP visual encoder, and a CLIP text encoder.

B. Ablation on using Relative Mask Attention in Different Layers

This ablation studies which pretrained CLIP layers should be replaced with Relative Mask Attention. The authors report that the last layer is important because features are progressively learned through the attention layers.

  • Layer Selection: The study compares four combinations of layers for inserting Relative Mask Attention while keeping the pretrained CLIP model fixed during training.All methods are trained on COCO and evaluated on the ADE20K validation set.
  • Layer Selection: The last layer is a key part of the results because features are gradually learned through the attention layers.The CLIP ViT-L/14@336px backbone has 24 layers, with four replaced by Relative Mask Attention.

C. More Visualization Results on Arbitrary Categories

MaskCLIP is visualized on user-specified categories, including closely related prompts and novel classes, while preserving dense/local CLIP features for person identification.

  • MaskCLIP classifies closely related user-specified prompts, including “four-leg animal” versus “two-leg animal” and “car,” “truck,” versus “SUV.”
  • The visualizations include person identification, demonstrating that the model preserves dense/local CLIP features well.
  • Figure 7 marks novel classes that do not exist in the COCO training-data labels in bold.
Loading 2208.08984v2…