PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation

Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, Heung-Yeung Shum

arXiv:2206.02777v3cs.CV

TL;DR

Detection and segmentation models were not unified, limiting cooperation between tasks and datasets. Mask DINO extends DINO with a mask branch and shared query-based training, and it reports the best results across instance, panoptic, and semantic segmentation among models under one billion parameters. Its large-scale setting remains constrained by additional segmentation-head memory, which requires smaller images and fewer queries than DINO.

  • Problem

    Best-performing Transformer-based detection and segmentation models remained separate, preventing task and data cooperation between the two tasks.

  • Method

    Mask DINO extends DINO with a parallel mask branch that reuses content queries to dot-product a high-resolution pixel embedding map, while extending query selection, denoising, and matching for segmentation.

  • Results

    Mask DINO achieves the best results on instance, panoptic, and semantic segmentation among models under one billion parameters, while improving detection and segmentation over compared models.

  • Takeaways & Limitations

    The framework shows that detection and segmentation can help each other in query-based models, with semantic and panoptic segmentation benefiting from detection pre-training.

  • Takeaways & Limitations

    In the large-scale setting, the segmentation head requires additional GPU memory, forcing smaller images and fewer queries than DINO and impacting detection performance.

Abstract

from arXiv · show

In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Some key components in DINO are extended for segmentation through a shared architecture and training process. Mask DINO is simple, efficient, and scalable, and it can benefit from joint large-scale detection and segmentation datasets. Our experiments show that Mask DINO significantly outperforms all existing specialized segmentation methods, both on a ResNet-50 backbone and a pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K) among models under one billion parameters. Code is available at \url{https://github.com/IDEACVR/MaskDINO}.

1. Introduction

Mask DINO addresses the lack of a unified Transformer framework for detection and segmentation by extending DINO with mask prediction and shared task cooperation. It reports stronger performance across detection and instance, panoptic, and semantic segmentation under comparable settings.

  • Transformer-based detection and segmentation models remained specialized, limiting task and data cooperation despite evidence that unified architectures can improve multiple tasks.
  • Mask DINO extends DINO with a parallel mask prediction branch, reusing content queries to predict binary masks from a high-resolution pixel embedding map.
  • The framework extends DINO components for segmentation, including query selection, mask denoising training, and hybrid bipartite matching for boxes and masks.
  • +0.8 AP on COCO detection over DINO is reported under the same ResNet-50 setting.
  • +2.6 AP, +1.1 PQ, and +1.5 mIoU are reported over Mask2Former for COCO instance, COCO panoptic, and ADE20K semantic segmentation, respectively.

2. Related Work

Transformer-based detectors have advanced end-to-end, query-based detection, while earlier DETR-like approaches remained inferior to specialized segmentation models. The section introduces instance, semantic, and panoptic segmentation as the main segmentation tasks.

  • Detection: DETR introduced end-to-end, query-based Transformer detection with set prediction and bipartite matching.
  • Detection: DAB-DETR represents queries as 4D anchor boxes and refines predictions layer by layer.
  • Detection: Earlier DETR-like models showed feasibility for joint detection and segmentation but remained inferior to specialized segmentation models.
  • Segmentation: Instance segmentation predicts a mask and category for each object instance, whereas semantic segmentation classifies every pixel, including background, by category.
  • Segmentation: Panoptic segmentation is presented as a unified segmentation task alongside instance and semantic segmentation.

3. Mask DINO

Mask DINO extends DINO with a parallel mask branch and adapts query selection, initialization, denoising, and matching for unified detection and segmentation. Its design addresses the mismatch between region-level detection queries and pixel-level segmentation queries while retaining DINO’s detection architecture.

  • Motivation: DINO’s queries encode positional and semantic information for detection, whereas Mask2Former’s queries are not designed for region-level localization.This difference helps explain why simply adding task heads produces weak cross-task performance.
  • Architecture: Mask DINO adds a parallel mask-prediction branch to DINO while preserving its core detection architecture.The framework minimally modifies DINO’s decoder and extends several detection components for segmentation.
  • Segmentation branch: Mask DINO constructs a 1/4-resolution pixel embedding map by fusing backbone and encoder features, then dot-products content queries with it for mask classification.The segmentation head maps channels to the Transformer hidden dimension and upsamples encoder features before fusion.
  • Unified and Enhanced Query Selection: Unified query selection uses encoder classification, box, and segmentation heads to select content queries and initialize both content and anchor-box queries.Selected features regress boxes and predict masks; these predictions are supervised and used as decoder anchors.
  • Unified and Enhanced Query Selection: Mask-enhanced box initialization derives boxes from initially predicted masks, using the more accurate early mask predictions to initialize decoder anchor boxes.The paper presents this as task cooperation between pixel-level segmentation and region-level detection.
  • Training: Mask DINO extends denoising and hybrid bipartite matching to masks, combining classification, box, and mask losses for consistent query-to-ground-truth assignment.The matching cost is λclsLcls + λboxLbox + λmaskLmask.

4. Experiments

Experiments evaluate Mask DINO across detection and instance, panoptic, and semantic segmentation, including ResNet-50 and SwinL settings. The results show strong performance, faster convergence, task cooperation, and scalability through shared training and detection pre-training.

  • Experimental Setup: Mask DINO is evaluated on object detection, instance, panoptic, and semantic segmentation using COCO, ADE20K, and Cityscapes.Main results use ResNet-50 and SwinL backbones; the experiments also include SOTA comparisons with DINO pre-trained on Objects365.
  • Instance Segmentation and Object Detection: 44.2 AP is achieved for instance segmentation in 24 epochs, surpassing Mask2Former with fewer than half the training epochs.Mask DINO also exceeds Mask2Former by +2.7 AP and +2.6 AP in the 12-epoch and 50-epoch settings, respectively.
  • Instance Segmentation and Object Detection: +1.2 AP improves detection after mask-enhanced box initialization, exceeding DINO by 0.8 AP.The authors report the best detection and instance segmentation performance among DETR-like models with a SwinL backbone without extra data.
  • Panoptic Segmentation: 1.0 PQ and 1.1 PQ improvements over previous best models are obtained in the 12-epoch and 50-epoch panoptic settings.The model also improves mask AP_pan by 2.6 to 44.3 AP, 0.6 higher than Mask2Former’s 43.7 AP.
  • Semantic Segmentation: 1.6 and 0.6 mIoU gains over Mask2Former are reported on ADE20K and Cityscapes, respectively.These semantic-segmentation comparisons use a ResNet-50 backbone and 100 queries.
  • Comparison with SOTA Models: 54.5 AP, 59.4 PQ, and 60.8 mIoU are achieved for instance, panoptic, and semantic segmentation among models under one billion parameters.With SwinL and Objects365 detection pre-training, Mask DINO outperforms Mask2Former on all three tasks, including a +2.5 AP instance-segmentation gain; unified training requires 10 to 20 fine-tuning epochs versus 100 for Mask2Former.

5. Conclusion

Mask DINO unifies Transformer-based object detection and image segmentation with minimal extensions to DINO. It achieves strong cross-task performance while exposing limitations in panoptic assistance and large-scale detection.

  • Mask DINO unifies object detection with instance, panoptic, and semantic segmentation in a Transformer-based framework.The framework extends DINO from detection to segmentation with minimal modifications.
  • Mask DINO outperforms previous specialized models across all three segmentation tasks among models under one billion parameters.
  • Detection and segmentation can help each other in query-based models, including through detection-pretrained visual representations for semantic and panoptic segmentation.
  • In COCO panoptic segmentation, segmentation tasks do not achieve mutual assistance because mask AP lags behind instance-only training.
  • Large-scale training has not produced new state-of-the-art detection performance because segmentation requires additional GPU memory.Smaller image sizes and fewer queries are used than in DINO, affecting final detection performance.

A. Visualization analysis

The paper motivates Mask DINO by showing why detection information and improved query mechanisms can strengthen segmentation in a unified architecture.

  • Detection is typically easier than instance segmentation, with Box AP usually more than 4 AP higher than mask AP.The paper argues that box prediction can guide attention toward meaningful regions for mask prediction.
  • DINO’s query selection and deformable attention improvements are presented as mechanisms that can also benefit segmentation.
  • Mask2Former’s learnable decoder queries cannot use selected top K encoder features to guide mask predictions.

B. Implementation details

The implementation details are provided in supplementary materials, alongside more detailed descriptions of the model implementation.

  • The code is available in the supplementary materials.
  • The supplementary materials provide detailed descriptions of the implementation.

B.1. General settings

Mask DINO is evaluated across detection and segmentation datasets using standard task-specific metrics, multiple backbones, joint losses, and visualization comparisons.

  • Dataset and metrics: Mask DINO is evaluated on COCO 2017 for detection, instance segmentation, and panoptic segmentation, and on ADE20K for semantic segmentation.
  • Dataset and metrics: Object detection and instance segmentation use standard AP and mask AP, while panoptic and semantic segmentation use PQ and mIoU.
  • Dataset and metrics: Panoptic evaluation also reports AP_th and AP_st for thing and stuff categories, respectively.
  • Backbone: Experiments use ResNet-50 and SwinL backbones, with Objects365 detection pre-training for the large SwinL setting.The pre-trained model is fine-tuned on corresponding datasets for all tasks.
  • Loss function: Joint training combines classification, box, and mask losses, including focal, L1, GIOU, cross-entropy, and dice terms.Point loss is also used within mask loss for efficiency.
  • Visualization: Figure 2 visualizes ground-truth and decoder-layer predicted masks for Mask2Former and Mask DINO.The displayed query is selected by its final assignment to the ground-truth mask, then dot-produced with the pixel embedding map.
  • Comparison: COCO test-dev comparisons report Mask DINO alongside existing state-of-the-art models, with TTA and O365 identified in the table caption.

B.2. Denoising training

Mask DINO extends denoising training to jointly reconstruct objects and masks from noisy boxes and labels. Noise-derived labels and boxes provide decoder content and positional queries during training, with specified noise strengths.

  • Denoising training: Noised objects are concatenated with original decoder queries for training and removed during inference.The model reconstructs ground-truth objects from the noised inputs.
  • Label noise: Label flipping with probability p = 0.2 produces content queries through label embeddings.Each noised label is embedded into a high-dimensional vector for the decoder.
  • Box noise: Box noise perturbs center position and scale, with λ1 = λ2 = 0.4 controlling the maximum shifts and scaling range.The box is represented as (x, y, w, h) and serves as the decoder’s positional query.

C. Large models setting

Large SwinL models are pre-trained on Objects365 for object detection and then fine-tuned for COCO and ADE20K segmentation. The settings vary by task in training duration, image scale, hardware, and query count.

  • Pre-training: SwinL models are pre-trained on Objects365 for object detection before segmentation fine-tuning.The pre-training follows DINO’s setting.
  • COCO fine-tuning: COCO instance and panoptic models are fine-tuned for 24 epochs at 1280 × 1280 resolution using 16 A100 GPUs.The image scale is 1.2× larger than the stated baseline setting.
  • ADE20K fine-tuning: ADE20K semantic models are fine-tuned for 160k iterations with 3× more queries.These settings differ from the COCO instance and panoptic configuration.

D. SOTA Results on COCO test-dev

The paper presents Mask DINO’s COCO test-dev results in Table 16. This section directs readers to that table for the reported evaluation results.

  • COCO test-dev: COCO test-dev results are reported in Table 16.The cited table is the designated location for the section’s results.
  • COCO test-dev: Table 16 contains the paper’s presented COCO test-dev evaluation results.The passage identifies the table but does not state individual metric values.
  • COCO test-dev: The section uses COCO test-dev as the evaluation setting for the results shown in Table 16.No further comparison or quantitative outcome is stated in the passage.
Loading 2206.02777v3…