PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask Scoring R-CNN

Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, Xinggang Wang

arXiv:1903.00241v1cs.CV

TL;DR

Instance segmentation commonly uses classification confidence as a mask-quality score, although it is not well correlated with mask IoU. Mask Scoring R-CNN adds a MaskIoU head that uses mask and feature information to recalibrate scores, yielding consistent COCO improvements. The method is simple, effective, and applicable to other instance segmentation networks.

  • Problem

    Classification confidence does not reliably measure mask quality, even though scoring affects instance segmentation evaluation and AP.

  • Method

    Mask Scoring R-CNN adds a MaskIoU head that regresses mask IoU from the mask-head output and RoI feature, then multiplies predicted MaskIoU by classification confidence.

  • Results

    AP improves consistently by about 1.5% with various backbone networks, and Mask Scoring R-CNN consistently outperforms Mask R-CNN on COCO.

  • Takeaways & Limitations

    The approach provides a simple mask-scoring direction and can be applied to other instance segmentation networks to obtain more reliable mask scores.

  • Takeaways & Limitations

    MaskIoU-head training samples must have box-level IoU above 0.5 with ground truth, although their MaskIoU may be below 0.5.

Abstract

from arXiv · show

Letting a deep network be aware of the quality of its own predictions is an interesting yet important problem. In the task of instance segmentation, the confidence of instance classification is used as mask quality score in most instance segmentation frameworks. However, the mask quality, quantified as the IoU between the instance mask and its ground truth, is usually not well correlated with classification score. In this paper, we study this problem and propose Mask Scoring R-CNN which contains a network block to learn the quality of the predicted instance masks. The proposed network block takes the instance feature and the corresponding predicted mask together to regress the mask IoU. The mask scoring strategy calibrates the misalignment between mask quality and mask score, and improves instance segmentation performance by prioritizing more accurate mask predictions during COCO AP evaluation. By extensive evaluations on the COCO dataset, Mask Scoring R-CNN brings consistent and noticeable gain with different models, and outperforms the state-of-the-art Mask R-CNN. We hope our simple and effective approach will provide a new direction for improving instance segmentation. The source code of our method is available at \url{https://github.com/zjhuang22/maskscoring_rcnn}.

1. Introduction

Instance segmentation needs mask scores that reflect pixel-level mask quality, because classification confidence can misrepresent inaccurate masks. Mask Scoring R-CNN learns mask quality and improves COCO performance by recalibrating these scores.

  • Instance segmentation advances from box-level recognition to precise pixel-level classification, requiring more detailed labels and delicate network design.
  • Classification confidence is a poor mask-quality measure because it identifies proposal categories without assessing mask quality or completeness.
  • Mask Scoring R-CNN learns a mask score from the predicted mask and instance feature instead of reusing classification confidence.
  • MaskIoU is learned directly and combined with classification confidence so the final score reflects both semantic category and mask completeness.
  • AP improves consistently by about 1.5% with various backbone networks when MS R-CNN mask scores replace classification confidence.

2. Related Work

Prior instance segmentation methods generally lack reliable alignment between mask scores and actual mask quality. Mask Scoring R-CNN addresses this gap by processing masks for quality estimation rather than relying on box-level or averaged pixel-level scores.

  • Instance Segmentation: Detection-based methods generate regions with detectors before predicting masks, whereas segmentation-based methods group pixel-level predictions into instances.
  • Instance Segmentation: Methods without explicit instance-mask quality scores use averaged pixel-level classification scores as an alternative.
  • Instance Segmentation: Misalignment between mask score and mask quality can cause higher-IoU masks to receive lower priority, degrading final AP.
  • Detection Score Correction: Unlike bounding-box score correction methods, Mask Scoring R-CNN estimates mask IoU through a regression task and processes the instance mask for completeness.
  • Detection Score Correction: Mask scoring is presented as a new direction for improving instance segmentation performance.

3. Method

Mask Scoring R-CNN augments Mask R-CNN with a MaskIoU head that estimates predicted-mask quality and calibrates mask scores using both classification confidence and mask completeness.

  • Motivation: Mask R-CNN can assign high classification confidence to low-quality masks because classification confidence does not measure mask completeness.Background clutter and occlusion can contribute to this misalignment.
  • Mask scoring: Mask scoring decomposes the mask score as smask = scls · siou, combining category classification with MaskIoU regression.scls is taken from the R-CNN classification score, while siou estimates mask quality.
  • MaskIoU head: The MaskIoU head concatenates RoIAlign features with the predicted mask and regresses their IoU against the ground-truth mask.A max-pooling layer aligns the predicted mask's spatial size with the RoI feature.
  • Training: MaskIoU-head training uses RPN proposals whose proposal-to-ground-truth box IoU exceeds 0.5, with targets generated from thresholded predicted masks.The predicted mask is binarized using a threshold of 0.5 before computing the MaskIoU regression target.
  • Inference: During inference, the MaskIoU head processes the top-k target masks selected after SoftNMS to calibrate their classification scores.The standard procedure uses k = 100 selected boxes before mask generation and MaskIoU prediction.

4. Experiments

Experiments on COCO evaluate Mask Scoring R-CNN across datasets, backbones, frameworks, MaskIoU-head designs, training targets, and prediction quality. The method consistently improves mask scoring while adding negligible computation.

  • Experimental setup: Experiments use COCO evaluation metrics, including AP, AP@0.5, AP@0.75, and scale-specific APS, APM, and APL.Unless noted, AP is evaluated using mask IoU.
  • Quantitative results: Mask Scoring R-CNN improves Mask R-CNN by about 1.5 AP overall and about 2 points at AP@0.75 across backbone networks.The reported gains are stable across ResNet-18/50/101 and robust across Faster R-CNN, FPN, and DCN+FPN frameworks.
  • MaskIoU-head design: The MaskIoU head combines the predicted mask score map with RoI features, with alternative fusion designs evaluated in ablation studies.The target-class score map can be pooled and concatenated with or multiplied by RoI features; all-class and high-resolution variants are also considered.
  • MaskIoU-head design: Concatenating the target score map with RoI features achieves the best fusion performance and becomes the default choice.This choice is evaluated alongside other ways of combining mask predictions and RoI features.
  • Training targets: Training the target category’s MaskIoU is preferred: predicting all categories degrades performance drastically, while regressing all positive categories is inferior to the default.The default target learns MaskIoU for the target category while ignoring other categories in the proposal.
  • Training targets: Using all examples to train the MaskIoU head produces the best performance among thresholds that retain only samples with MaskIoU greater than τ.The threshold experiment compares training on all examples with selecting samples above a specified MaskIoU threshold.
  • Prediction quality: MaskIoU predictions correlate with ground-truth MaskIoU at about 0.74 for two backbone configurations, exceeding the cited 0.617 box-IoU correlation.The evaluated predictions come from 500,000 instances across 5,000 COCO validation images.
  • Efficiency: The MaskIoU head adds about 0.39G FLOPs per proposal, while measured speed remains about 0.132 or 0.202 seconds per image for the two tested configurations.These speeds are reported as approximately identical for Mask R-CNN and Mask Scoring R-CNN.

5. Conclusion

Mask Scoring R-CNN addresses instance-mask scoring by adding a MaskIoU head to Mask R-CNN, aligning mask scores with mask quality. On COCO, it consistently and obviously outperforms Mask R-CNN and can extend to other instance segmentation networks.

  • Mask Scoring R-CNN adds a MaskIoU head to Mask R-CNN to align mask scores with mask quality.
  • On the COCO benchmark, Mask Scoring R-CNN consistently and obviously outperforms Mask R-CNN.
  • The approach can be applied to other instance segmentation networks to obtain more reliable mask scores.
Loading 1903.00241v1…