Source-linked AI summary
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features
Liang-Chieh Chen, Alexander Hermans, George Papandreou, Florian Schroff, Peng Wang, Hartwig Adam
TL;DR
Instance segmentation requires jointly addressing object detection and semantic segmentation. MaskLab combines box detection with semantic and direction prediction, achieving comparable performance with state-of-the-art models on COCO.
Problem
Instance segmentation requires simultaneously solving object detection and semantic segmentation.
Method
MaskLab builds on a detector and combines semantic segmentation with direction prediction to perform foreground/background segmentation within detected regions.
Results
MaskLab shows comparable performance with state-of-the-art models on the COCO instance segmentation benchmark for mask segmentation and box detection.
Takeaways & Limitations
MaskLab provides a unified instance-segmentation model using box detection, semantic segmentation, and direction prediction.
Takeaways & Limitations
Qualitative failures mainly arise from missed detections, wrong class predictions, and coarse segmentation boundaries.
Abstract
from arXiv · showhide
In this work, we tackle the problem of instance segmentation, the task of simultaneously solving object detection and semantic segmentation. Towards this goal, we present a model, called MaskLab, which produces three outputs: box detection, semantic segmentation, and direction prediction. Building on top of the Faster-RCNN object detector, the predicted boxes provide accurate localization of object instances. Within each region of interest, MaskLab performs foreground/background segmentation by combining semantic and direction prediction. Semantic segmentation assists the model in distinguishing between objects of different semantic classes including background, while the direction prediction, estimating each pixel's direction towards its corresponding center, allows separating instances of the same semantic class. Moreover, we explore the effect of incorporating recent successful methods from both segmentation and detection (i.e. atrous convolution and hypercolumn). Our proposed model is evaluated on the COCO instance segmentation benchmark and shows comparable performance with other state-of-art models.
1. Introduction
MaskLab addresses instance segmentation by combining box detection with semantic and direction-based mask refinement. It evaluates this design on COCO and reports performance comparable to state-of-the-art models.
- Instance segmentation jointly localizes object instances and assigns pixels to them with pixel-level accuracy.
- MaskLab builds on Faster R-CNN and uses semantic segmentation to distinguish different classes, including background, while direction prediction separates instances sharing a class.
- The model incorporates atrous convolution, hypercolumn features, multi-grid context, and deformable crop-and-resize operations.
- MaskLab is evaluated on COCO and shows comparable mask-segmentation and box-detection performance to state-of-the-art models.
2. Related Work
Prior instance-segmentation methods generally begin from detection or segmentation. MaskLab combines these approaches by using detection boxes with semantic and direction features for instance separation.
- Deep-neural-network instance-segmentation methods are categorized by whether they begin with detection or segmentation modules.
- Detection-based methods use object detectors to classify mask regions or refine predicted boxes into masks.
- Segmentation-based methods first produce pixel-level predictions and then cluster them into object instances.
- MaskLab combines detection-based and segmentation-based advantages through Faster R-CNN, semantic segmentation, and direction features.
3. MaskLab
MaskLab predicts boxes, semantic logits, and direction logits, then combines them within each box to produce instance masks. Hypercolumn refinement and deformable crop-and-resize further improve mask processing.
- MaskLab produces refined box predictions, semantic segmentation logits, and direction prediction logits using shared convolutional features.
- Semantic logits distinguish semantic classes and background, while direction logits estimate each pixel’s direction toward its instance center.
- The model selects the semantic channel matching each predicted label, crops it to the box, assembles direction logits, and combines both for foreground/background segmentation.
- Hypercolumn features are concatenated with coarse mask logits and processed by extra convolutional layers to predict refined masks.
- Deformable crop-and-resize learns offsets for sub-boxes before repeating crop-and-resize on the deformed regions.
4. Experimental Evaluation
MaskLab’s COCO evaluation studies implementation and feature choices, showing gains from direction-aware refinement, lower-level hypercolumn features, pretraining, and denser feature maps. Final evaluations report competitive mask and box detection performance, while qualitative analysis identifies detection and coarse-boundary failures.
- Experimental setup: MaskLab is trained end-to-end on COCO, with ablations on minival and final evaluation on test-dev using mask-IoU mAP.The implementation uses TensorFlow, atrous convolution with output stride = 8, and groundtruth boxes for training semantic and direction branches.
- Feature ablations: Using both semantic and direction features achieves 29.72% mAP@0.75, rising to 30.57% with 4-bin distance quantization.Direction-only features reach 27.4%, compared with 24.44% for semantic-only features; 4 bins split each direction region into four distance regions.
- Mask refinement: 33.89% mAP@0.75 is obtained with conv1 and conv2 mask-refinement features, improving over 30.57% without these lower-level features.Adding more lower-level features produces no further improvement, and the refinement network uses three 5 × 5 convolution layers with 64 filters.
- Atrous convolution: Changing atrous output stride from 8 to 16 lowers performance from 40.41% to 38.61% mAP@0.75.The experiment supports using denser feature maps for this model configuration.
- Test-dev results: On COCO test-dev, MaskLab performs competitively with leading methods for both mask segmentation and box detection.MaskLab+ is 1.9% better than the base ResNet-101 model for mask results, while the best box results reach 41.9% mAP with ImageNet pretraining and 43.0% with JFT pretraining.
- Qualitative results: Qualitative failures mainly arise from missed detections, wrong class predictions, and coarse mask boundaries.The semantic branch can also activate on non-person regions because it is trained with groundtruth boxes without negative examples; the detection branch filters those wrong box predictions.
5. Conclusion
MaskLab addresses instance segmentation with three outputs and uses semantic segmentation and direction prediction for foreground/background segmentation. Its effectiveness was demonstrated on the COCO benchmark with promising results.
- MaskLab produces box detection, semantic segmentation, and direction prediction outputs for instance segmentation.
- Built on a state-of-the-art detector, MaskLab performs foreground/background segmentation using semantic segmentation and direction prediction.
- MaskLab demonstrated effectiveness on the challenging COCO instance segmentation benchmark and showed promising results.