PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick

arXiv:1703.06870v3cs.CV

TL;DR

Instance segmentation must detect every object while precisely separating each instance, motivating a flexible framework for this combined task. Mask R-CNN extends Faster R-CNN with a parallel mask branch and surpasses prior single-model results across COCO instance segmentation, detection, and keypoint tasks.

  • Problem

    Instance segmentation requires detecting all objects while precisely segmenting each instance, creating a need for an enabling framework for the combined task.

  • Method

    Mask R-CNN extends Faster R-CNN with a parallel RoI mask branch and uses quantization-free RoIAlign to preserve pixel-to-pixel spatial alignment.

  • Results

    Mask R-CNN surpasses previous single-model state-of-the-art results on COCO instance segmentation, object detection, and human pose estimation.

  • Takeaways & Limitations

    Mask R-CNN provides a flexible framework for instance-level recognition that can be readily extended to more complex tasks.

  • Takeaways & Limitations

    The design is not optimized for speed, leaving better speed/accuracy trade-offs outside the paper’s scope.

Abstract

from arXiv · show

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code has been made available at: https://github.com/facebookresearch/Detectron

1. Introduction

Mask R-CNN extends Faster R-CNN with a parallel mask-prediction branch to provide a simple, flexible, and fast framework for instance segmentation. RoIAlign addresses pixel-to-pixel misalignment, while the framework achieves strong COCO results and generalizes to human pose estimation.

  • Framework: Mask R-CNN extends Faster R-CNN with a parallel branch that predicts a pixel-to-pixel segmentation mask for each Region of Interest.The mask branch is a small fully convolutional network applied independently to each RoI.
  • Core design: RoIAlign removes coarse spatial quantization, improving mask accuracy by relative 10% to 50%, with larger gains under stricter localization metrics.The method also decouples mask and class prediction by predicting a binary mask for each class independently.
  • Results: Mask R-CNN surpasses previous state-of-the-art single-model results on COCO instance segmentation, including the heavily engineered 2016 competition winner.Ablation experiments evaluate basic instantiations to demonstrate robustness and analyze core factors.
  • Efficiency: 200ms per frame on a GPU and one to two days of COCO training on a single 8-GPU machine demonstrate the framework’s speed.The authors emphasize that fast training and testing, flexibility, and accuracy can support future instance-segmentation research.
  • Generalization: Mask R-CNN applies to human pose estimation by representing each keypoint as a one-hot binary mask, surpassing the 2016 COCO keypoint competition winner at 5 fps.This generalization requires minimal modification and illustrates the framework’s flexibility for instance-level recognition.

2. Related Work

Related instance-segmentation methods largely build on R-CNN-style detection, segment proposals, fully convolutional prediction, or semantic segmentation. Mask R-CNN instead follows an instance-first strategy, contrasting with segmentation-first approaches and addressing limitations noted in prior work.

  • R-CNN: R-CNN evaluates convolutional networks independently on candidate object regions, while RoIPool and Faster R-CNN improve speed, accuracy, and learned attention.Faster R-CNN learns the attention mechanism with a Region Proposal Network.
  • Instance Segmentation: Segment-proposal methods predict candidate segments before recognition, a strategy described as slow and less accurate.DeepMask and related methods learn segment candidates that are subsequently classified by Fast R-CNN.
  • Instance Segmentation: FCIS combines segment proposals and object detection through position-sensitive channels that jointly address classes, boxes, and masks, but produces errors on overlapping instances and spurious edges.Its fully convolutional design makes the system fast.
  • Instance Segmentation: Semantic-segmentation-driven methods separate same-category pixels into instances, whereas Mask R-CNN uses an instance-first rather than segmentation-first strategy.These methods start from per-pixel classification results such as FCN outputs.

3. Mask R-CNN

Mask R-CNN extends Faster R-CNN with a parallel branch that predicts a binary mask for each RoI alongside class and bounding-box outputs. Its key design elements are decoupled per-class mask prediction, fully convolutional mask representation, and RoIAlign for preserving pixel-to-pixel alignment.

  • Mask R-CNN: Mask R-CNN adds a third branch that predicts an object mask in parallel with Faster R-CNN’s class label and bounding-box offset outputs.The method retains Faster R-CNN’s two-stage procedure and adds binary-mask prediction in the second stage.
  • Mask R-CNN: L = Lcls + Lbox + Lmask combines classification, box-regression, and mask losses, with K class-specific m × m masks trained using per-pixel sigmoid binary cross-entropy.For an RoI with ground-truth class k, the mask loss is applied only to the corresponding class mask.
  • Mask Representation: The mask branch predicts each m × m mask with an FCN, preserving explicit spatial layout and avoiding the vector collapse of fully connected mask predictors.The fully convolutional representation requires fewer parameters and is more accurate in the reported experiments.
  • RoIAlign: RoIAlign removes RoIPool’s coordinate quantization by using bilinear interpolation at four sampled locations per bin, preserving alignment between extracted features and the input.The interpolated values are aggregated using max or average pooling, and the exact sampling locations are not critical when quantization is avoided.
  • RoIAlign: RoIAlign produces large improvements, whereas quantized RoIWarp performs on par with RoIPool, demonstrating that alignment—not bilinear resampling alone—is crucial.The comparison is reported with further experimental details in Table 2c.
  • Network Architecture: Mask R-CNN is instantiated with ResNet and ResNeXt backbones of 50 or 101 layers, and its heads add a fully convolutional mask branch to prior Faster R-CNN box heads.The evaluated backbones include C4 and Feature Pyramid Network configurations.

4. Experiments: Instance Segmentation

On COCO instance segmentation, Mask R-CNN outperforms prior state-of-the-art methods, including challenge winners, while ablations show that RoIAlign, independent mask prediction, deeper backbones, and fully convolutional mask heads are important. The model also maintains practical training and inference speed, and multi-task training improves box detection.

  • State-of-the-art comparison: Mask R-CNN outperforms prior state-of-the-art instance-segmentation models, including MNC and FCIS, and ResNet-101-FPN surpasses FCIS+++ without additional bells and whistles.FCIS+++ uses multi-scale training/testing, horizontal-flip testing, and online hard example mining.
  • Mask branch: Using fully convolutional mask heads improves mask AP by 2.1 points over multilayer perceptrons because FCNs explicitly exploit masks’ spatial layout.The comparison uses a ResNet-50-FPN backbone with unpretrained convolutional layers for a fair comparison.
  • Architecture and mask prediction: Deeper backbones and advanced designs, including FPN and ResNeXt, improve Mask R-CNN, while independent per-class sigmoid masks avoid the severe 5.5-point mask-AP loss from multinomial prediction.Class-agnostic masks remain nearly as effective, achieving 29.7 mask AP versus 30.3 for class-specific masks on ResNet-50-C4.
  • RoIAlign: RoIAlign improves AP by about 3 points over RoIPool, outperforms RoIWarp, and yields larger gains with stride-32 features, including 7.3 mask AP and 10.5 mask AP75.With RoIAlign, stride-32 C5 features achieve 30.9 AP versus 30.3 AP for stride-16 C4 features; FPN adds 1.5 mask AP and 0.5 box AP.
  • Bounding-box detection: Mask R-CNN’s ResNet-101-FPN configuration surpasses prior state-of-the-art box detectors, while multi-task training raises box AP by 0.9 points over Faster R-CNN with RoIAlign.The model’s mask and box AP differ by only 2.7 points: 37.1 mask AP versus 39.8 box AP.
  • Inference and training: The shared-feature ResNet-101-FPN model runs in 195ms per image plus 15ms CPU resizing, achieves statistically identical mask AP to the unshared model, and trains in 44 hours on synchronized 8-GPU hardware.ResNet-50-FPN training takes 32 hours, while the ResNet-101-C4 variant takes approximately 400ms per image.

5. Mask R-CNN for Human Pose Estimation

Mask R-CNN extends naturally to human pose estimation by predicting one mask per keypoint type, using minimal pose-specific knowledge. On COCO, the ResNet-50-FPN system achieves 62.7 APkp, improves to 63.1 with person segmentation, runs at 5 fps, and benefits substantially from RoIAlign.

  • Framework and implementation: Mask R-CNN predicts K one-hot masks, one for each keypoint type, treating keypoints independently.For each visible ground-truth keypoint, training minimizes cross-entropy over an m^2-way softmax output.
  • Main results and ablations: 62.7 APkp is 0.9 points higher than the COCO 2016 keypoint detection winner, while using a considerably simpler and faster method.The experiments primarily demonstrate framework generality and exploit minimal human-pose domain knowledge.
  • Main results and ablations: 4.4 points is the APkp improvement from RoIAlign over RoIPool, showing that alignment is essential for pixel-level localization in keypoints and masks.Keypoint detections are especially sensitive to localization accuracy, even with the finer strides of ResNet-50-FPN.
  • Framework and generality: The effectiveness of Mask R-CNN for boxes, masks, and keypoints suggests it can serve as a framework for other instance-level tasks.The unified model’s flexibility is demonstrated through its extension from instance segmentation to human pose estimation.

Appendix A: Experiments on Cityscapes

On Cityscapes, Mask R-CNN achieves state-of-the-art instance segmentation without using coarse annotations, with especially large gains for overlapping person and car instances. COCO pre-training further improves performance in the dataset’s low-data and domain-shifted categories.

  • Results: 26.2 AP on Cityscapes test exceeds the previous best DIN result by over 30% relatively and surpasses concurrent SGN’s 25.0 AP, without coarse data.The best fine-data-only entry achieves 17.4 AP, making Mask R-CNN’s result approximately 50% better.
  • Overlapping instances: ∼40% relative improvement on person raises AP from 21.8 to 30.5, while ∼20% improvement on car raises AP from 39.4 to 46.9.Cityscapes averages 6 people and 9 cars per image, creating substantial within-category overlap.
  • Low-data regime: About 200–500 training samples in truck, bus, and train expose a central low-data challenge for Cityscapes instance segmentation.These categories also contribute substantially to the observed validation/test bias and domain shift.
  • COCO pre-training: 32.0 AP on test from COCO pre-training improves nearly 6 points over the fine-only model.The model initializes seven Cityscapes categories from COCO and randomly initializes rider, then fine-tunes for 4k iterations.
  • Domain shift: 28.8/22.8, 53.5/32.2, and 33.0/18.6 are the fine-only val/test AP pairs for truck, bus, and train, respectively.COCO pre-training improves these categories most, but the domain shift persists.

Appendix B: Enhanced Results on COCO

Appendix B describes techniques that improve Mask R-CNN’s original COCO results and notes its use as the framework for all three winning teams in the COCO 2017 instance segmentation competition.

  • Enhanced Results on COCO: Mask R-CNN is compatible with complementary detection and segmentation techniques, including improvements to Fast/Faster R-CNN and fully convolutional networks.The appendix presents techniques that improve over the original results.
  • Enhanced Results on COCO: All three winning teams in the COCO 2017 instance segmentation competition used Mask R-CNN as their framework and significantly outperformed the previous state of the art.The passage attributes this adoption to Mask R-CNN’s generality and flexibility.

Instance Segmentation and Object Detection

Mask R-CNN’s enhanced results improve both mask and box AP, with gains from end-to-end training, stronger pretraining, augmentation, and deeper architectures. A non-local model and test-time augmentation further achieve strong single-model results, forming the foundation of the COCO 2017 submission.

  • Enhanced results: 5.1 mask AP points and 7.7 box AP points are gained overall, increasing mask AP from 36.7 to 41.8 and box AP from 39.6 to 47.3.Each model improvement consistently increases both mask AP and box AP, supporting the generalization of the framework.
  • Enhanced results: 0.6 mask AP and 1.2 box AP are gained through approximate end-to-end training that jointly trains RPN and Mask R-CNN.The approximation computes partial RoIAlign gradients while ignoring gradients with respect to RoI coordinates.
  • Enhanced results: 0.6 mask AP and 0.8 box AP are gained through train-time scale augmentation using randomly sampled scales from [640, 800] pixels.Training is extended to 260k iterations, with learning-rate reductions at 200k and 240k iterations.
  • Enhanced results: 40.3 mask AP and 45.0 box AP are achieved with a non-local model without test-time augmentation, running at 3fps on an Nvidia Tesla P100 GPU.The result uses the recently proposed non-local model.
  • Enhanced results: 41.8 mask AP and 47.3 box AP are obtained by combining evaluations across scales from [400, 1200] pixels and horizontal flips.This is reported as a single-model result and serves as the foundation of the COCO 2017 competition submission.

Keypoint Detection

Mask R-CNN keypoint detection improves with longer training, stronger backbones, data distillation, and test-time augmentation. These enhancements raise APkp from the updated baseline to 70.4.

  • Keypoint Detection: 130k training iterations, with learning-rate reductions at 100k and 120k, improve APkp by about 1 point.The updated schedule reduces the learning rate by 10 at both milestones.
  • Keypoint Detection: 66.1 and 67.3 are the APkp values obtained by replacing ResNet-50 with ResNet-101 and ResNeXt-101, respectively.ResNeXt-101 produces the higher reported APkp.
  • Keypoint Detection: 1.8 points raise Mask R-CNN APkp to 69.1 when data distillation exploits 120k additional unlabeled COCO images.Data distillation self-trains by predicting annotations for unlabeled images and updating the model with them.
  • Keypoint Detection: 70.4 is the APkp achieved using the same test-time augmentation as instance segmentation.This augmentation further boosts APkp after data distillation.
Loading 1703.06870v3…