Source-linked AI summary
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Jifeng Dai, Kaiming He, Jian Sun
TL;DR
Instance-aware semantic segmentation requires distinguishing individual objects, which conventional semantic segmentation methods do not do effectively. The paper introduces a shared-feature, three-stage Multi-task Network Cascade with end-to-end training through causal dependencies. It reports 63.5% mAPr at 360ms per image on PASCAL VOC and first place in the COCO segmentation track.
Problem
Conventional semantic segmentation predicts pixel categories but does not identify individual object instances, while prior mask proposal methods are slow.
Method
Multi-task Network Cascades use shared convolutional features across stages that differentiate instances, estimate masks, and categorize objects, trained end-to-end through a differentiable spatial transformation.
Results
63.5% mAPr on PASCAL VOC was achieved at 360ms per image, and the method won first place in the COCO segmentation track.
Takeaways & Limitations
The cascade provides fast, accurate instance segmentation while allowing stronger shared features and deeper models to benefit the full system.
Takeaways & Limitations
Boundary refinement with a CRF is beyond the paper’s scope and left for future investigation.
Abstract
from arXiv · showhide
Semantic segmentation research has recently witnessed rapid progress, but many leading methods are unable to identify object instances. In this paper, we present Multi-task Network Cascades for instance-aware semantic segmentation. Our model consists of three networks, respectively differentiating instances, estimating masks, and categorizing objects. These networks form a cascaded structure, and are designed to share their convolutional features. We develop an algorithm for the nontrivial end-to-end training of this causal, cascaded structure. Our solution is a clean, single-step training framework and can be generalized to cascades that have more stages. We demonstrate state-of-the-art instance-aware semantic segmentation accuracy on PASCAL VOC. Meanwhile, our method takes only 360ms testing an image using VGG-16, which is two orders of magnitude faster than previous systems for this challenging problem. As a by product, our method also achieves compelling object detection results which surpass the competitive Fast/Faster R-CNN systems. The method described in this paper is the foundation of our submissions to the MS COCO 2015 segmentation competition, where we won the 1st place.
1. Introduction
Instance-aware semantic segmentation remains challenging because conventional semantic segmentation predicts pixel categories without distinguishing individual objects. Multi-task Network Cascades address this gap with a shared-feature, three-stage causal cascade and end-to-end training.
- Conventional fully convolutional networks predict category labels per pixel but do not identify individual object instances.
- Existing CNN-based instance-aware methods rely on slow mask proposal modules that can bottleneck inference and segmentation accuracy.
- MNCs decompose the task into differentiating instances, estimating masks, and categorizing objects.
- The three stages share convolutional features, while later stages depend causally on earlier outputs.
- A differentiable spatial-coordinate layer enables theoretically valid backpropagation through predicted boxes and supports clean, single-step end-to-end training.
- 63.5% mAPr on PASCAL VOC was achieved at 360ms per image, about 3.0% above prior same-VGG results and two orders faster than previous systems.
- The method also achieved first place in the COCO segmentation track using ResNet-101 and benefited from stronger shared features.
2. Related Work
Prior instance-aware segmentation systems commonly combine region-based recognition with computationally expensive proposal mechanisms. The paper positions shared convolutional features and the MNC cascade as alternatives to these bottlenecks.
- R-CNN-style object detection uses region proposals followed by CNN-based classification, while shared convolutional features accelerate later systems.
- Earlier instance-aware segmentation methods rely on mask-level proposals, making proposal computation a major inference bottleneck.MCG, for example, takes 30 seconds per image.
- Fully convolutional semantic segmentation enables end-to-end per-pixel prediction but cannot distinguish instances of the same category.
3. Multi-task Network Cascades
MNCs use a three-stage cascade that proposes boxes, regresses class-agnostic masks, and categorizes instances. The stages share convolutional features and are trained with a unified loss despite causal dependencies.
- The cascade proposes box-level instances, regresses mask-level instances, and categorizes each instance.
- The unified loss is non-independent because later-stage losses depend on earlier outputs, so the entire cascade is trained end-to-end.
- 3.1. Regressing Box-level Instances: Stage 1 predicts class-agnostic bounding boxes with objectness scores using an RPN-like fully convolutional structure.
- 3.2. Regressing Mask-level Instances: Stage 2 takes shared features and stage-1 boxes to produce a class-agnostic pixel-level mask for each proposal.
- 3.2. Regressing Mask-level Instances: Mask regression uses only a few proposed boxes rather than dense sliding windows, reducing computation while sharing features across stages.
- 3.3. Categorizing Instances: Stage 3 combines box- and mask-based pathways before predicting category scores for each instance.
4. End-to-End Training
The cascade is trained end-to-end with a unified loss, requiring differentiable handling of predicted box coordinates so gradients can pass through later stages. A differentiable RoI warping layer supplies this capability while preserving efficient feature extraction.
- Unified cascade loss: The cascade loss sums stage-specific losses, while later-stage terms condition on earlier-stage outputs rather than remaining independent.The three terms use unit balance weights and are minimized jointly with respect to the network parameters.
- Differentiable dependencies: The main training challenge is backpropagating through RoI pooling when predicted boxes and convolutional features both depend on the network parameters.Unlike fixed proposals, predicted boxes require gradients with respect to their spatial coordinates.
- Differentiable dependencies: Differentiable RoI warping crops and interpolates features inside each predicted box into a fixed spatial resolution before standard max pooling.The layer represents cropping and warping as a linear transform of the full-image feature map and operates independently on each channel.
- Backpropagation: Bilinear interpolation makes the RoI transform differentiable with respect to box center, width, and height, enabling gradients through predicted box positions.Because the interpolation kernel is nonzero only locally, each output computation uses very few terms.
- Backpropagation: The resulting cascade can be trained by stochastic gradient descent in a single end-to-end framework, with the RoI warping module providing the needed backpropagation components.The same framework supports the dependencies of subsequent masking operations and other standard network components.
5. Cascades with More Stages
The MNC framework extends to additional cascade stages by adding class-wise box regression and reusing later-stage structures during inference. A 3-stage trained model can therefore use a 5-stage inference cascade, while training can be aligned with that structure.
- Additional stages: Class-wise bounding-box regression is added to stage 3 as a sibling of the classifier, following the joint classification-and-regression design of Fast R-CNN.The regression layer has dimension 4(N+1).
- Five-stage inference: During inference, regressed stage-3 boxes become new proposals, and stages 2 and 3 are rerun on them, producing a 5-stage inference structure.The added stages share the structures of stages 2 and 3 while using the updated stage-3 boxes as inputs.
- Training and scope: The cascade can be trained end-to-end with the same backpropagation algorithm so its training-time structure matches inference-time structure.The paper reports that this consistency improves accuracy in experiments.
- Training and scope: Although more stages are possible, the paper presents cascades with at most five stages because of concerns about fast inference.This is an explicit scope boundary for the presented MNC configurations.
6. Implementation Details
Implementation uses proposal filtering, IoU-based sample definitions, pretrained initialization, image-centric optimization, and five-stage inference with mask voting. These choices control computational cost and determine how training examples and predictions are processed.
- Proposal processing: Stage 1 produces approximately 10^4 regressed boxes, which NMS reduces to the top 300 proposals for stage 2.The NMS threshold is 0.7 IoU, and only pathways determined by these proposals carry later-stage training signals.
- Sample definitions: Stage-2 positives require proposal-to-ground-truth-mask IoU greater than 0.5, while stage 3 uses box-level and mask-level IoU thresholds of at least 0.5.Stage-2 mask targets are proposal–ground-truth-mask intersections resized to m × m pixels.
- Training setup: Shared convolutional and corresponding 4096-d fully connected layers are initialized from ImageNet-pretrained models, while extra layers are initialized randomly.Training is image-centric, computing shared features over the entire image and sampling RoIs for loss computation.
- Training setup: Training uses one image per mini-batch per GPU across 8 GPUs, with learning rates of 0.001 for 32k iterations and 0.0001 for the next 8k.Images are resized so their shorter side has 600 pixels, without multi-scale training or testing.
- Inference post-processing: Inference returns 600 instances from stages 3 and 5, then applies category-wise NMS and weighted mask averaging for similar predictions.Mask voting improves accuracy by approximately 1% over the NMS outcome.
7. Experiments
Experiments show that end-to-end training and deeper cascades improve MNC accuracy, while shared features and avoiding external proposals enable fast inference. The method performs strongly on PASCAL VOC, object detection, and MS COCO segmentation.
- Training strategies: End-to-end training raises mAPr from 60.5% with untuned shared features to 62.6% for a 3-stage cascade.The model structure is held constant across these training-strategy comparisons, so the reported improvement is attributed to end-to-end training.
- Training strategies: 63.5% mAPr is obtained by end-to-end training a 5-stage cascade, improving over the 3-stage result under the same 5-stage inference strategy.The comparisons are also observed with ZF net, supporting the generality of the findings.
- PASCAL VOC segmentation: 360ms per image makes the method about two orders of magnitude faster than previous instance-aware segmentation systems.The comparison includes all testing steps on an Nvidia K40 GPU; prior methods use MCG proposals costing about 30s.
- Object detection: 75.9% mAPb is obtained on PASCAL VOC object detection using the union of VOC 2007 trainval+test and 2012 trainval, substantially exceeding Fast/Faster R-CNN.Using box-regression outputs alone gives 73.5% mAPb, while mask-level outputs give 70.9%.
- MS COCO segmentation: 24.6%/44.3% is achieved on MS COCO test-dev with ResNet-101, compared with 19.5%/39.7% using VGG-16.The metrics are mAPr@IoU=[0.5:0.95] and mAPr@IoU=0.5, respectively.
- MS COCO segmentation: 28.2%/51.5% on the COCO test-challenge set wins first place in the ILSVRC & COCO 2015 segmentation track.The final result adds global context modeling, multi-scale testing, and ensembling to the baseline.
8. Conclusion
The paper presents Multi-task Network Cascades as a fast, accurate approach to instance segmentation and notes that boundary refinement with CRFs remains outside its scope.
- Multi-task Network Cascades target fast and accurate instance segmentation through a multi-task network-cascade framework.
- CRF-based refinement of instance-mask boundaries is orthogonal to the method and left for future investigation.