PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, Liang-Chieh Chen

arXiv:2308.02487v2cs.CV

TL;DR

Open-vocabulary segmentation must segment and recognize categories beyond those available during training, but existing CLIP-based approaches often extract image features multiple times in two-stage pipelines. FC-CLIP unifies mask generation and classification with a shared frozen convolutional CLIP backbone, achieving state-of-the-art results with substantially lower computational cost.

  • Problem

    Open-vocabulary segmentation seeks to recognize arbitrary classes beyond the limited semantic vocabularies typically available in annotated panoptic datasets.

  • Method

    FC-CLIP is a single-stage framework that uses one shared frozen convolutional CLIP backbone for mask generation and open-vocabulary classification.

  • Results

    FC-CLIP achieves state-of-the-art performance across open-vocabulary segmentation benchmarks while training 7.5× and testing 6.6× faster than ODISE.

  • Takeaways & Limitations

    The results establish a simple, strong, and fast baseline for efficient single-stage open-vocabulary segmentation.

  • Takeaways & Limitations

    The two-stage setting is constrained by duplicate feature extraction and by using high-resolution inputs for masks but low-resolution inputs for CLIP classification.

Abstract

from arXiv · show

Open-vocabulary segmentation is a challenging task requiring segmenting and recognizing objects from an open set of categories. One way to address this challenge is to leverage multi-modal models, such as CLIP, to provide image and text features in a shared embedding space, which bridges the gap between closed-vocabulary and open-vocabulary recognition. Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through a mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using a shared Frozen Convolutional CLIP backbone, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields a better accuracy-cost trade-off. The proposed FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining. When training on COCO panoptic data only and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Additionally, the training and testing time of FC-CLIP is 7.5x and 6.6x significantly faster than the same prior art, while using 5.9x fewer parameters. FC-CLIP also sets a new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code at https://github.com/bytedance/fc-clip

1 Introduction

Open-vocabulary segmentation addresses the limited class vocabularies of conventional panoptic segmentation by using language-aligned representations. FC-CLIP unifies mask generation and open-vocabulary recognition with a shared frozen convolutional CLIP backbone, achieving strong benchmark performance and lower computational cost.

  • Motivation: Panoptic segmentation typically covers only dozens or hundreds of semantic classes, limiting application to settings with unlimited possible categories.Open-vocabulary segmentation instead uses category-name text embeddings to classify a broader range of categories.
  • Prior limitations: Existing open-vocabulary methods commonly use two stages: a mask generator produces proposals, then frozen CLIP classifies masked image crops.This design extracts image features multiple times and can be computationally heavy.
  • Design challenges: Fine-tuning a shared CLIP backbone can disrupt image-text alignment, especially harming out-of-vocabulary recognition, while separate mask-generation backbones increase model size and cost.CLIP’s usual low-resolution pretraining also conflicts with the higher resolutions preferred for dense prediction.
  • FC-CLIP: FC-CLIP builds the mask generator and classifier on a shared frozen convolutional CLIP backbone, preserving alignment while using CLIP features for mask generation.Its design relies on frozen CLIP’s open-vocabulary classification ability and CNN-based features’ stronger generalization across input sizes.
  • Efficiency and performance: 238M frozen and 21M trainable parameters give FC-CLIP a smaller model than ODISE’s 1494M frozen and 28M trainable parameters.Training takes 25.6 V100 GPU days, 7.5× faster than ODISE’s 192 V100 GPU days, and inference is 6.6× faster.
  • Efficiency and performance: 16.8 AP on ADE20K surpasses ODISE by +2.4, while FC-CLIP also exceeds SAN by +1.1 mIoU on both A-847 and PC-459.The same model is trained only on COCO panoptic data and evaluated without task-specific fine-tuning.
  • Conclusion: FC-CLIP establishes a simple, strong, and fast baseline that advances performance across multiple open-vocabulary segmentation benchmarks.The authors present the framework as a basis for future efficient single-stage research.

2 Related Work

Prior work adapts vision-language models and segmentation architectures to recognize categories beyond training labels, often through separate mask-generation and CLIP-classification stages. FC-CLIP instead presents a shared frozen convolutional CLIP single-stage pipeline that reuses semantic features for both functions.

  • Vision-language models: CLIP and ALIGN learn aligned image-text representations from large-scale image-text data, supporting open-vocabulary segmentation.Earlier vision-language approaches commonly used pretrained detectors for visual representations before downstream language-supervised adaptation.
  • Segmentation architectures: Modern closed-vocabulary segmentation commonly uses end-to-end mask-classification architectures built from pixel and mask decoders.FC-CLIP builds on the pixel decoder and mask decoder design of Mask2Former.
  • Pipeline comparison: Existing two-stage pipelines extract image features multiple times and use different input scales for mask segmentation and CLIP classification.These choices increase computation and can produce sub-optimal performance when dense prediction requires higher resolution.
  • Open-vocabulary segmentation: MaskCLIP and related methods use a class-agnostic mask generator followed by a frozen CLIP encoder for cross-modal alignment.This two-stage arrangement separates segmentation from open-vocabulary recognition.
  • FC-CLIP: FC-CLIP uses one shared frozen convolutional CLIP backbone for a mask generator, an in-vocabulary classifier, and an out-of-vocabulary classifier.Its out-of-vocabulary classifier uses frozen CLIP features, while the overall design follows the single-stage pipeline shown in the comparison.

3 Method

FC-CLIP addresses the inefficiency and alignment risks of two-stage open-vocabulary segmentation with a shared frozen convolutional CLIP backbone. The framework supports mask generation and classification in one stage while preserving open-vocabulary recognition.

  • Problem Definition: Open-vocabulary segmentation uses distinct training and testing category sets, with novel categories potentially appearing only during inference.Category names for test classes are assumed available as natural-language representations during testing.
  • Two-Stage Open-Vocabulary Segmentation: Two-stage methods extract image features separately for mask generation and mask classification, increasing computation and limiting backbone scalability.They also process high-resolution inputs for mask generation and lower-resolution inputs for CLIP classification.
  • Naïve Single-Stage Open-Vocabulary Segmentation: Fine-tuning a shared CLIP backbone can misalign image and text features, causing sub-optimal performance on novel unseen classes.A naïve single-stage design also increases training costs by 2.1× to 52.8 GPU days.
  • FC-CLIP: FC-CLIP builds a class-agnostic mask generator and two classifiers on a shared frozen convolutional CLIP backbone.The mask generator uses a multi-scale deformable-attention pixel decoder and mask decoders with object queries.
  • In-Vocabulary Classifier: The in-vocabulary classifier compares pooled mask embeddings with category text embeddings in a common embedding space.Category text embeddings are generated once with a CLIP-pretrained text encoder and kept in memory.
  • Out-of-Vocabulary Classifier: The out-of-vocabulary classifier reuses frozen CLIP features, and geometric ensembling combines its scores with in-vocabulary predictions.The ensemble uses separate α and β weights for training-seen and novel categories.

4 Experimental Results

FC-CLIP is evaluated as a zero-shot open-vocabulary segmentation model across multiple datasets, showing strong accuracy, transferability, and efficiency. Experiments also examine its architecture, cross-dataset generalization, and the effect of freezing CLIP.

  • Evaluation Setup: FC-CLIP is trained on COCO panoptic data and evaluated zero-shot on ADE20K, Cityscapes, Mapillary Vistas, and semantic-segmentation benchmarks.The evaluation covers open-vocabulary panoptic and semantic segmentation.
  • Street-View Panoptic Segmentation: +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes are achieved over ODISE, despite a slightly lower SQ.The lower SQ indicates a weaker mask generator than ODISE’s larger-backbone system.
  • Semantic Segmentation: +6.6, +8.2, +10.4, and +12.5 mIoU over MaskCLIP are reported on A-847, PC-459, A-150, and PC-59, respectively.The comparisons use the same training annotations, while FC-CLIP is trained solely with panoptic annotations.
  • Efficiency: 6.61× and 7.08× faster inference than ODISE are measured on ADE20K and COCO, respectively.The FPS comparison averages runtime over the validation set, including post-processing.
  • Ablation: Freezing CLIP preserves novel-class recognition, whereas fine-tuning harms performance on unseen concepts by disrupting image-text feature alignment.The comparison uses the same mask proposals for 10 seen and 10 unseen classes.

5 Conclusion

The paper presents FC-CLIP as a simple, effective single-stage framework for open-vocabulary segmentation. It uses a shared frozen convolutional CLIP backbone to reduce costs while achieving strong benchmark performance.

  • FC-CLIP builds mask generation and classification on a shared frozen convolutional CLIP backbone.The framework is presented as a single-stage alternative for open-vocabulary segmentation.
  • The framework reduces training and testing costs while establishing a strong baseline across multiple benchmarks.The conclusion also positions FC-CLIP as evidence that pretrained CLIP can support dense prediction tasks.

Appendix

The appendix provides additional experiments, comparisons, visualizations, and dataset details concerning FC-CLIP’s design and evaluation.

  • Supplementary experiments analyze FC-CLIP’s design, comparisons with open-vocabulary semantic segmentation and ensemble methods, and hyperparameter tuning.
  • The appendix quantitatively compares ViT-based and CNN-based CLIP across varying input sizes and includes visualizations and comprehensive dataset details.

6 Additional Experimental Results

Additional experiments examine backbone freezing, classifier ensembling, grounding, resolution scaling, and architectural portability. Together, these studies characterize FC-CLIP’s design choices and performance across segmentation settings.

  • Backbone analysis: Building all three FC-CLIP modules on a single frozen CLIP backbone attains the best performance.The comparison evaluates fine-tuning and freezing choices for each module.
  • Grounding segmentation: FC-CLIP achieves state-of-the-art grounding segmentation performance despite using no grounding loss during training.The evaluation uses ground-truth classes as text-query inputs.
  • Classifier ensembling: Geometric ensembling slightly outperforms arithmetic ensembling, favoring α ≤0.5 and β ≥0.5.These settings bias seen classes toward the in-vocabulary classifier and unseen classes toward the out-of-vocabulary classifier.
  • Resolution scaling: CNN-based CLIP is compared with ViT-based CLIP across input resolutions for classification and panoptic segmentation.The comparisons use varying resolution and, for panoptic segmentation, identical mask proposals from ODISE.
  • Architectural portability: FC-CLIP maintains superior performance across different backbones and segmentation frameworks, including ResNet and kMaX-DeepLab.The reported experiments are trained on COCO and tested zero-shot on other datasets.
  • Visualization: Visualization examples show FC-CLIP predictions on the ADE20K validation set after COCO panoptic training and zero-shot evaluation.The examples are presented in Figure 5.

7 Datasets Information and Licenses

The paper trains FC-CLIP on COCO panoptic data and evaluates it across several zero-shot segmentation datasets. The section also records dataset sizes, class variants, validation settings, and license information.

  • Training dataset: COCO provides 118k training images and 5k validation images under the 2017 splits used for FC-CLIP training.Unless otherwise specified, the model is trained on the COCO train split and evaluated on various datasets.
  • Evaluation datasets: ADE20K contains 2k validation images and is evaluated in both A-847 and A-150 class versions.A-847 has 847 classes, while A-150 contains 150 frequent categories.
  • Evaluation datasets: Cityscapes focuses on urban street-scene understanding and contributes 500 fine-data validation images..
  • Evaluation datasets: Mapillary Vistas is a traffic-related dataset with 2k validation images..
  • PASCAL datasets: Pascal Context is evaluated using PC-459 with 459 classes and PC-59 with 59 classes across 5k validation images.The dataset covers indoor and outdoor scenes.
  • PASCAL datasets: Pascal VOC contains 1.5k validation images with 20 foreground classes and one background class, reported as PAS-21 or PAS-20.PAS-21 includes the background class, while PAS-20 excludes it.
Loading 2308.02487v2…