PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation

Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, Xingang Wang

arXiv:2303.17225v1cs.CV

TL;DR

Open-vocabulary segmentation must support arbitrary categories without fragmenting semantic, instance, and panoptic tasks across specialized models. FreeSeg addresses this with one-shot unified training, a two-stage mask-and-classification framework, and adaptive prompt learning, reporting improved performance and generalization across three tasks.

  • Problem

    Existing open-vocabulary segmentation methods use task-specific architectures and parameters, fragmenting segmentation across tasks.

  • Method

    FreeSeg trains one all-in-one model using universal mask proposals, zero-shot mask classification, and adaptive task- and category-aware prompts.

  • Results

    FreeSeg establishes new state-of-the-art performance and generalization across semantic, instance, and panoptic segmentation tasks.

  • Takeaways & Limitations

    A single architecture and inference parameter set can support open-vocabulary semantic, instance, and panoptic segmentation in deployment.

Abstract

from arXiv · show

Recently, open-vocabulary learning has emerged to accomplish segmentation for arbitrary categories of text-based descriptions, which popularizes the segmentation system to more general-purpose application scenarios. However, existing methods devote to designing specialized architectures or parameters for specific segmentation tasks. These customized design paradigms lead to fragmentation between various segmentation tasks, thus hindering the uniformity of segmentation models. Hence in this paper, we propose FreeSeg, a generic framework to accomplish Unified, Universal and Open-Vocabulary Image Segmentation. FreeSeg optimizes an all-in-one network via one-shot training and employs the same architecture and parameters to handle diverse segmentation tasks seamlessly in the inference procedure. Additionally, adaptive prompt learning facilitates the unified model to capture task-aware and category-sensitive concepts, improving model robustness in multi-task and varied scenarios. Extensive experimental results demonstrate that FreeSeg establishes new state-of-the-art results in performance and generalization on three segmentation tasks, which outperforms the best task-specific architectures by a large margin: 5.5% mIoU on semantic segmentation, 17.6% mAP on instance segmentation, 20.1% PQ on panoptic segmentation for the unseen class on COCO.

1. Introduction

FreeSeg addresses the fragmentation of open-vocabulary segmentation by using one framework for semantic, instance, and panoptic tasks. Its unified training, adaptive prompts, and reported experiments target arbitrary categories and multi-task deployment.

  • 1. Introduction: Existing segmentation tasks rely on labor-intensive pixel-level annotations and limited category vocabularies, restricting scalability relative to real-world language.Open-vocabulary methods address custom categories, but existing approaches use task-specific architectures and parameters.
  • 1. Introduction: FreeSeg uses one architecture and inference parameter set for open-vocabulary semantic, instance, and panoptic segmentation.This unified design is intended to handle multiple tasks and arbitrary categories in one model.
  • 1. Introduction: FreeSeg combines universal mask proposals with zero-shot classification, optimized through one-shot training on multi-task labels.The two-stage framework extracts masks first and classifies them against text descriptions second.
  • 1. Introduction: Adaptive prompt learning encodes task-aware and category-sensitive concepts to improve generalization to arbitrary text descriptions.Semantic context interaction and test-time prompt tuning further improve cross-model alignment and unseen-class generalization.
  • 1. Introduction: FreeSeg evaluates semantic, instance, and panoptic segmentation across COCO, ADE20K, and VOC 2012, reporting new state-of-the-art performance and generalization.The reported evaluation also emphasizes reduced research effort and feasibility for multi-task deployment.

2. Related Work

Related work spans open-vocabulary segmentation, universal segmentation, and prompt learning. These lines of research respectively address category generalization, task unification, and adaptation of vision-language representations.

  • 2. Related Work: Open-vocabulary segmentation maps visual features into semantic spaces or aligns them with pretrained vision-language models.These approaches target categories unavailable during training.
  • 2. Related Work: Universal segmentation seeks one architecture for semantic, instance, and panoptic tasks, with MaskFormer casting segmentation as mask classification.Existing universal approaches constrain multiple tasks to a unified training paradigm.
  • 2. Related Work: Prompt learning adapts pretrained language or vision-language models for downstream prediction through optimized or task-specific text prompts.Prior work applies prompt optimization and text-visual feature matching to dense prediction tasks.

3. Methodology

FreeSeg uses a two-stage, all-in-one framework that learns universal mask proposals with multi-task supervision and classifies them through adaptive task- and category-aware prompts. The same model supports semantic, instance, and panoptic segmentation, including arbitrary categories through CLIP-based matching and test-time prompt tuning.

  • 3.1. FreeSeg Framework: FreeSeg first extracts universal mask proposals, then uses CLIP-based zero-shot classification to produce task-oriented segmentation for arbitrary categories.The mask proposal extractor is trained once with multi-task labels, while inference matches mask representations against adaptive textual embeddings.
  • 3.1. FreeSeg Framework: One-shot training supervises an all-in-one mask proposal extractor across semantic, instance, and panoptic tasks, selecting one task label per iteration to reduce gradient conflicts.The mask loss combines Focal and Dice losses, while per-iteration task selection avoids simultaneously optimizing conflicting task gradients.
  • 3.2. Adaptive Prompt Learning: Adaptive prompts concatenate learnable task and class representations into joint text embeddings, allowing the model to encode task-specific characteristics and generalize to unseen categories.Task prompts distinguish semantic, instance, and panoptic segmentation; class prompts accept arbitrary category names through the CLIP text encoder.
  • 3.3. Semantic Context Interaction: Semantic context interaction uses cross-attention between adaptive text embeddings and multi-scale visual features to emphasize visual evidence relevant to the supplied text classes.Projected visual queries interact with text-derived keys and values before an output projection enhances the visual features.
  • 3.4. Test Time Prompt Tuning: Test Time Prompt Tuning selects low-entropy, high-confidence unseen-class predictions and optimizes the adaptive class prompt to improve cross-modal alignment.Entropy filtering uses threshold τ, because lower entropy indicates higher prediction confidence.

4.1. Datasets and Evaluation Metrics

The evaluation spans open-vocabulary semantic, instance, and panoptic segmentation across COCO, ADE20K, and VOC2012, using task-specific metrics for seen and unseen classes.

  • COCO, ADE20K, and VOC2012 provide the evaluation datasets, with seen/unseen category splits defined for open-vocabulary testing.
  • Semantic segmentation is evaluated with mIoU for seen and unseen classes and hIoU for their combined performance.
  • Instance segmentation reports mAP for seen and unseen classes, while panoptic segmentation uses PQ-related metrics for open-vocabulary evaluation.

4.2. Implementation Details

Experiments use established segmentation and vision-language components with dataset-specific training schedules, while Table 2 reports seen- and unseen-class instance-segmentation mAP.

  • Mask2Former extracts mask proposals, ResNet101 serves as the backbone, and CLIP uses a ViT-B/16 visual backbone on COCO.COCO training uses 8×A100 GPUs, AdamW, 60,000 iterations, and 640×640 inputs.
  • COCO training uses batch size 32 per GPU, learning rate 0.0002, weight decay 0.0002, and task/class prompt sizes of 8×512 and 16×512.
  • Table 2 compares state-of-the-art open-vocabulary instance-segmentation methods using mAP for seen and unseen classes.

4.3. Comparison to State-of-the-art Methods

FreeSeg achieves strong open-vocabulary results across semantic, instance, and panoptic segmentation, with especially large gains on unseen classes.

  • Open Vocabulary Semantic Segmentation: 49.1% mIoU on unseen COCO classes surpasses ZSSeg by +5.5%, while 28.6% mIoU on ADE20K surpasses it by +8.3%.
  • Open Vocabulary Instance Segmentation: 20.6% unseen-class mAP on COCO outperforms ZSI by +7.0% mAP, although ZSI has higher seen-class mAP under box-level supervision.FreeSeg also achieves 16.3% / 15.4% results on ADE20K, as reported in the supplied passage.
  • Open Vocabulary Panoptic Segmentation: 29.8% unseen-class PQ, 79.2% SQ, and 37.6% RQ outperform ZSSeg by 20.1%, 7.5%, and 25.4%, respectively.

4.4. Generalization Analysis

Cross-dataset evaluation trains FreeSeg on one dataset and tests it on another without finetuning, treating all target classes as unseen.

  • COCO-trained and ADE20K-trained models are directly evaluated on other datasets without finetuning, despite differences in data distributions and domains.
  • 24.6% mIoU, 6.5% mAP, 16.3% PQ, and 21.6% RQ on ADE20K after COCO training exceed MaskCLIP by 0.9%, 0.6%, 1.2%, and 2.4%.The same setting reports 71.8% SQ.
  • When trained on ADE20K and validated on COCO, FreeSeg achieves 21.7% mIoU, 6.6% mAP, 16.5% PQ, 72.0% SQ, and 21.6% RQ.

4.5. Ablation Study

Ablations show that multi-task training and adaptive prompting improve FreeSeg’s open-vocabulary segmentation, especially for unseen classes, across semantic, instance, and panoptic tasks.

  • Component Analysis: 2.8% and 1.3% mIoU gains follow from adding the adaptive task prompt and semantic context interaction, respectively, after adaptive class prompting.Without text guidance, the primary vision model reaches only 4.9% mIoU and 0.7% mAP on unseen classes.
  • Component Analysis: Test-time prompt tuning improves unseen-class performance during inference and supports stronger cross-modal alignment of visual and text features.The same module trends are reported for open-vocabulary instance and panoptic segmentation.
  • Multi-Task Analysis: 43.3% mIoU on unseen semantic classes is achieved by unified multi-task training, while unseen-class performance also improves for instance and panoptic segmentation.The unified model uses one-shot training with multi-task labels and reduces training costs by nearly two-thirds.
  • Adaptive Prompt Analysis: 9.6% mIoU improvement on unseen semantic classes is obtained with adaptive prompting over fixed prompts, alongside gains on instance and panoptic segmentation.Adaptive prompts improve mAP by 2.2% on unseen instance segmentation and PQ by 7.5% on unseen panoptic segmentation.

4.6. Qualitative results

Qualitative comparisons show that FreeSeg segments unseen categories that CLIP misses and supports arbitrary class specification across universal segmentation tasks.

  • 4.6. Qualitative results: FreeSeg accurately segments unseen classes such as giraffe and grass for semantic segmentation, while CLIP misses some unseen instances such as cow and skateboard.The examples also illustrate arbitrary-class specification for instance and panoptic segmentation.

5. Conclusion

The conclusion presents FreeSeg as a single universal model for unified open-vocabulary segmentation, reducing training and deployment costs while improving generalization.

  • 5. Conclusion: FreeSeg uses one architecture and shared inference parameters for open-vocabulary semantic, instance, and panoptic segmentation.Compared with single-task training, it reduces training cost by about two-thirds and achieves better generalization performance.
  • 5. Conclusion: One unified model reduces computational capacity, memory cost, and bandwidth during real-scene deployment.
Loading 2303.17225v1…