PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple Framework for Open-Vocabulary Segmentation and Detection

Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, Lei Zhang

arXiv:2303.08131v3cs.CV

TL;DR

Open-vocabulary detection and segmentation differ in vocabulary, task objectives, and annotation granularity, motivating a unified model. OpenSeeD jointly trains an encoder-decoder on separate detection and segmentation data using shared semantics, decoupled queries, and conditioned mask decoding. It achieves stronger or competitive transfer across both tasks, including state-of-the-art segmentation and competitive detection results.

  • Problem

    Detection and segmentation use different vocabularies, task targets, and spatially granular box or mask supervision, limiting straightforward joint learning across separate datasets.

  • Method

    OpenSeeD jointly trains one encoder-decoder using a shared text-defined semantic space, separate foreground/background queries, and conditioned mask decoding.

  • Results

    OpenSeeD achieves stronger or competitive zero-shot and transfer performance across segmentation and detection, including improved open-vocabulary segmentation benchmarks and competitive detection performance.

  • Takeaways & Limitations

    OpenSeeD provides a strong baseline for a single open-vocabulary model jointly serving segmentation and detection.

Abstract

from arXiv · show

We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground objects and background stuff, while detection merely cares about the former; $ii$) data discrepancy -- box and mask annotations are with different spatial granularity, and thus not directly interchangeable. To address these issues, we propose a decoupled decoding to reduce the interference between foreground/background and a conditioned mask decoding to assist in generating masks for given boxes. To this end, we develop a simple encoder-decoder model encompassing all three techniques and train it jointly on COCO and Objects365. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection. Specifically, OpenSeeD beats the state-of-the-art method for open-vocabulary instance and panoptic segmentation across 5 datasets, and outperforms previous work for open-vocabulary detection on LVIS and ODinW under similar settings. When transferred to specific tasks, our model achieves new SoTA for panoptic segmentation on COCO and ADE20K, and instance segmentation on ADE20K and Cityscapes. Finally, we note that OpenSeeD is the first to explore the potential of joint training on segmentation and detection, and hope it can be received as a strong baseline for developing a single model for both tasks in open world.

1. Introduction

OpenSeeD addresses the vocabulary, task, and annotation-granularity gaps between open-vocabulary detection and segmentation by jointly training one model on both data types. It combines a shared semantic space, decoupled decoding, and conditioned mask decoding to support both tasks.

  • Detection and segmentation differ in vocabulary size and supervision granularity, with Objects365 providing boxes for 365 concepts across 1.7M images versus COCO masks for 133 categories across 0.1M images.
  • OpenSeeD jointly learns from separate detection and segmentation datasets using a single open-vocabulary model, unlike aligned-annotation joint training or pretrain-then-fine-tune approaches.
  • Segmentation covers foreground objects and background stuff, whereas detection focuses only on foreground objects; mask annotations can be converted to boxes but not reliably in reverse.
  • A shared text encoder aligns concepts in a common semantic space, while foreground and background queries reduce task interference in decoding.
  • Conditioned mask decoding learns masks from boxes and supplies mask assistance for detection data, helping reconcile the different spatial supervision available to the two tasks.
  • Joint training yields new state-of-the-art segmentation results for zero-shot and task-transfer settings, alongside competitive zero-shot detection performance across varied datasets.

2. Related Work

Prior work connects detection and segmentation either through joint training on datasets with aligned box and mask annotations or through detection pretraining followed by segmentation transfer. These approaches do not provide a single open-vocabulary model jointly trained across separate detection and segmentation datasets.

  • Mask R-CNN and HTC jointly train detection and segmentation on a single dataset containing both box and mask annotations.
  • Joint-training approaches are constrained to foreground instances rather than the broader background-stuff requirements of semantic and panoptic segmentation.
  • Detection models such as HTC and Mask DINO can be pretrained with box supervision and then transferred to a specific segmentation dataset.
  • Pretrain-and-finetune protocols produce two separate models rather than one model jointly serving detection and segmentation.

3. Method

OpenSeeD jointly learns open-vocabulary segmentation and detection with a shared image-text encoder-decoder, then reconciles their task and supervision differences through specialized query decoding and conditioned mask prediction.

  • Basic framework: OpenSeeD encodes images and vocabulary concepts with separate image and text encoders, aligning visual semantics before decoding masks, boxes, and classification scores.The decoder cross-attends image features, while classification scores come from similarity between decoded semantics and text features.
  • Basic loss formulation: A unified loss jointly trains segmentation mask, box, and classification objectives across the union of segmentation and detection datasets.Segmentation masks provide derived boxes, while detection contributes box and classification losses; one set of weights supports both tasks.
  • Decoupled decoding: OpenSeeD divides decoder queries into foreground, background, and conditioned types to address differences between object detection and segmentation supervision.Foreground queries serve both tasks, background queries handle stuff categories in segmentation, and conditioned queries support mask generation from boxes and concepts.
  • Decoupled decoding: Foreground and background decoding use separate query groups and matching procedures, applying both groups to segmentation but only foreground decoding to detection.This design reduces interference while allowing foreground supervision from both datasets to cooperate.
  • Query construction: Language-guided selection supplies foreground queries from text-related image features, while learnable background queries represent segmentation stuff categories.Selecting text-relevant tokens improves query initialization and supports transfer to novel vocabulary; background queries are not matched to detection ground truth.
  • Conditioned mask decoding: Conditioned mask decoding learns the mapping from concepts and boxes to masks using segmentation data, addressing the coarser spatial supervision of detection boxes.On ADE20K, conditioned decoding raises mask AP from 8.6 to 46.4 in the reported pilot study.

4. Experiment

OpenSeeD is jointly pretrained on COCO segmentation and Objects365 detection, then evaluated through zero-shot and task-specific transfer, ablations, and in-the-wild benchmarks. It improves open-vocabulary segmentation across several datasets while retaining strong detection and closed-set transfer performance.

  • Experimental Setup: OpenSeeD is jointly pretrained using COCO panoptic segmentation data and Objects365 object-detection data, with separate dataset settings for its tiny and large models.COCO2017 provides around 110k segmentation images; Objects365 provides 660k v1 or 1.7M v2 detection images.
  • Experimental Setup: OpenSeeD uses one set of weights for zero-shot open-vocabulary evaluation across panoptic, instance, semantic segmentation, and detection tasks.The evaluation reports PQ, mask AP, mIoU, and box AP for the corresponding tasks.
  • Open-Vocabulary Benchmarking: OpenSeeD improves instance-segmentation mask AP over ODISE and X-Decoder on ADE20K, Cityscapes, and LVIS, while remaining competitive on panoptic and semantic segmentation.Reported gains include 1.1 and 1.9 mask AP on ADE20K, 10.2 and 8.3 on Cityscapes, and 9.8 on LVIS versus the cited X-Decoder comparisons.
  • Direct and Task-Specific Transfer: 59.5 PQ is achieved for COCO panoptic segmentation without fine-tuning, while task-specific transfer reaches 53.7 PQ and 42.6 AP on ADE20K and 48.5 AP on Cityscapes.The ADE20K results use 1280 × 1280 image size.
  • Segmentation and Detection in the Wild: More than 10 AP improvement is observed on the SeginW benchmark when detection supervision is combined with segmentation supervision under the same setting.SeginW contains 25 datasets and reports average and median mAP.
  • Segmentation and Detection in the Wild: OpenSeeD’s tiny model outperforms GLIP-T by 2.8 AP in average on the 35-dataset ODinW benchmark under a setting using Objects365 detection data.The comparison is conducted zero-shot with jointly pretrained model weights.
  • Ablation: Ablations show that combining tasks and using detection data improves segmentation across backbones, while offline pseudo-mask assistance significantly improves mask AP and box AP.The pseudo masks are generated on Objects365 conditioned on boxes using the large OpenSeeD model.

5. Conclusion

OpenSeeD unifies open-vocabulary segmentation and detection through joint learning, decoupled decoding, and conditioned mask decoding. The model improves open-segmentation, maintains reasonable detection performance, and transfers to closed-vocabulary tasks, while leaving richer grounding and image-text training for future work.

  • OpenSeeD jointly learns segmentation and detection from different datasets with a single open-vocabulary model.
  • Decoupled decoding separates foreground and background queries, while language-guided foreground selection bridges their task gap.
  • Conditioned mask decoding provides an interactive segmentation interface and assists detection data during training.
  • The unified model significantly improves open-segmentation while retaining reasonable detection performance and transferring to close-vocabulary tasks.
  • OpenSeeD does not use referring or grounding data or large-scale image-text pairs, leaving broader joint training to future work.
Loading 2303.08131v3…