Source-linked AI summary
Fully Convolutional Instance-aware Semantic Segmentation
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, Yichen Wei
TL;DR
Instance-aware semantic segmentation must classify and segment individual object instances, while conventional and prior fully convolutional approaches leave this joint problem unresolved. FCIS provides an end-to-end fully convolutional solution that shares convolutional representations and score maps across both subtasks and all ROIs. It achieves state-of-the-art accuracy and efficiency, including a 0.24-second COCO inference time and first place in the COCO 2016 segmentation competition.
Problem
Instance-aware semantic segmentation requires detecting and segmenting individual object instances, but conventional FCNs cannot assign different semantics to the same pixel across regions.
Method
FCIS jointly predicts instance masks and categories using shared fully convolutional representations and score maps across segmentation, detection, and ROIs.
Results
FCIS achieves state-of-the-art accuracy and efficiency, winning the COCO 2016 segmentation competition and reaching 0.24 seconds per image during COCO inference.
Takeaways & Limitations
The integrated architecture improves the state of the art in both accuracy and efficiency for instance-aware semantic segmentation.
Takeaways & Limitations
Prior position-sensitive mask-proposal methods remain limited by fixed-size square windows, multi-scale scanning, category blindness, and downstream classification requirements.
Abstract
from arXiv · showhide
We present the first fully convolutional end-to-end solution for instance-aware semantic segmentation task. It inherits all the merits of FCNs for semantic segmentation and instance mask proposal. It performs instance mask prediction and classification jointly. The underlying convolutional representation is fully shared between the two sub-tasks, as well as between all regions of interest. The proposed network is highly integrated and achieves state-of-the-art performance in both accuracy and efficiency. It wins the COCO 2016 segmentation competition by a large margin. Code would be released at \url{https://github.com/daijifeng001/TA-FCN}.
1. Introduction
Instance-aware semantic segmentation requires distinguishing individual objects, which conventional FCNs cannot do because translation-invariant responses assign the same pixel the same semantics across regions. FCIS addresses this with a fully convolutional joint segmentation-and-detection design that shares representations and avoids per-ROI warping.
- Motivation: Conventional FCNs produce per-pixel category score maps but are unaware of individual object instances.Translation invariance makes the same pixel receive identical responses regardless of its relative position in context.
- Limitations of prior methods: Existing instance-aware methods use ROI pooling and per-ROI fully connected networks, which lose spatial detail, over-parameterize masks, and repeat computation across ROIs.These drawbacks particularly degrade accuracy for large objects and make inference slow with many ROIs.
- Limitations of prior methods: Prior fully convolutional mask-proposal methods encode relative position with position-sensitive maps but require downstream classification and fixed-size sliding-window image-pyramid scanning.They generate category-agnostic proposals rather than performing end-to-end instance-aware segmentation.
- FCIS contribution: FCIS jointly predicts object masks and categories using shared convolutional representations and score maps, without extra parameters, warping, or resizing.It operates on box proposals, with simple and fast per-ROI computation.
- Results: FCIS achieves state-of-the-art accuracy and efficiency, wins the COCO 2016 segmentation competition, and runs inference in 0.24 seconds per image with ResNet-101.The reported inference time was measured on an Nvidia K40.
2. Our Approach
The approach extends position-sensitive score maps into a joint inside/outside formulation for instance detection and segmentation. A fully convolutional architecture shares computation across tasks and ROIs, preserving spatial structure while keeping per-ROI processing negligible.
- Position-sensitive representation: FCNs are translation invariant, so a single pixel cannot receive different foreground and background semantics for different object instances.Position-sensitive maps introduce relative-position information needed for instance-aware prediction.
- Position-sensitive representation: Prior position-sensitive mask proposals use fixed-size square windows, multi-scale scanning, and separate classification, limiting their direct use for instance-aware segmentation.The proposals are category-agnostic and require a downstream network for object categories.
- Joint formulation: FCIS shares one convolutional representation and one set of score maps between object segmentation and detection, eliminating separate task-specific subnetworks and extra parameters.The design targets the tight correlation between the two subtasks.
- Joint formulation: The joint formulation converts inside and outside scores into detection and segmentation outputs using max, softmax, and average-pooling operations.Detection distinguishes object-bearing ROIs, while segmentation separates pixels inside versus outside the instance boundary.
- End-to-end architecture: FCIS produces score maps fully convolutionally, preserves original-image aspect ratios and local weight sharing, and avoids feature warping, resizing, and fully connected layers.Its per-ROI operations are limited to cell division, score-map copying, softmax, max, and average pooling.
- End-to-end architecture: The end-to-end system uses an RPN to generate ROIs, a fully convolutional score-map branch, and a bounding-box regression branch for ROI refinement.Training combines detection, segmentation, and bounding-box regression losses, with the latter two active only for positive ROIs.
3. Related Work
Related work covers semantic segmentation, category-agnostic object segment proposals, and instance-aware semantic segmentation. Existing instance-aware approaches generally separate classification and segmentation or rely on complex post-processing rather than an end-to-end fully convolutional solution.
- Semantic image segmentation: Semantic image segmentation assigns every pixel a category label without distinguishing object instances, and FCNs dominate this task.Extensions include global context, multi-scale feature fusion, deconvolution, CRFs, and domain transforms.
- Object segment proposal: Object segment proposal methods generate category-agnostic object segments using either low-level image features or deep learning.DeepMask and SharpMask are examples, while a fully convolutional proposal method motivates FCIS.
- Instance-aware semantic segmentation: Instance-aware semantic segmentation requires both classification and segmentation of object instances, but most state-of-the-art approaches perform these subtasks separately.Representative systems include SDS, Hypercolumn, CFM, MNC, MultiPathNet, and iterative approaches.
- Instance-aware semantic segmentation: Attempts to extend FCNs through output grouping or clustering rely on complex hand-crafted post-processing, are not end-to-end, and have unsatisfactory performance.This motivates an integrated fully convolutional formulation.
- FCNs for object detection: R-FCN adapts position-sensitive score maps for object classification but remains unaware of instance segmentation and must be combined with another method for that task.A straightforward combination with InstFCN uses separately trained FCNs for mask prediction and classification.
4. Experiments
Experiments show that FCIS benefits from translation-sensitive score maps and joint mask-classification modeling, outperforming fully convolutional baselines while achieving strong accuracy and efficiency on COCO.
- PASCAL VOC Ablation: 59.1% and 36.0% mAPr are achieved by the naïve MNC baseline at IoU thresholds of 0.5 and 0.7, respectively.These scores are 5.5% and 12.9% lower than those of original MNC.
- PASCAL VOC Ablation: FCIS achieves the best PASCAL VOC result, while translation-invariant and separate-score-map variants perform worse.The ablation supports the importance of position-sensitive score maps and the joint formulation.
- COCO Comparison: 28.8% mAPr@[0.5:0.95] is achieved by FCIS on COCO test-dev without OHEM, 4.2% absolutely higher than MNC.FCIS is also approximately 6× faster in inference and approximately 4× faster in training than MNC.
- Network Depth: Accuracy improves from ResNet-50 to ResNet-101 and saturates at ResNet-152.This result is reported for FCIS networks of different depths.
- COCO Challenge Experiments: 29.2% mAPr@[0.5:0.95] is achieved by the baseline FCIS system, already exceeding MNC+++ from the 2015 challenge.Multi-scale testing adds 2.8% accuracy, while horizontal flip and multi-scale training add 0.7% and 0.9%, respectively.
- COCO Challenge Experiments: 37.6% final accuracy is obtained with an ensemble of six networks, 3.8% absolutely higher than G-RMI, the 2016 second-place entry.The entry wins the 2016 COCO segmentation challenge and exceeds MNC+++ by 9.2% absolutely.
- COCO Detection: 39.7% mAPb@[0.5:0.95] is achieved for COCO box-level detection, ranking FCIS second on the detection leaderboard.The detected boxes are obtained as enclosing boxes of the instance masks.
5. Conclusion
The paper presents a fully convolutional method for instance-aware semantic segmentation, emphasizing an integrated architecture and a novel joint formulation for accuracy and efficiency.
- The method is the first fully convolutional approach for instance-aware semantic segmentation.
- Its high performance is attributed to a highly integrated and efficient network architecture.
- A novel joint formulation is identified as a particularly important component of the architecture.