Source-linked AI summary
Instance-sensitive Fully Convolutional Networks
Jifeng Dai, Kaiming He, Yi Li, Shaoqing Ren, Jian Sun
TL;DR
FCNs are effective for semantic segmentation but are not directly instance-aware. InstanceFCN addresses this gap with relative-position score maps and a simple assembling module, avoiding mask-resolution-dependent high-dimensional layers. It achieves compelling proposal results on PASCAL VOC and MS COCO and competitive downstream instance semantic segmentation.
Problem
FCNs produce semantic outputs but are not directly applicable to producing or distinguishing object-instance segments.
Method
InstanceFCN classifies pixels by relative positions on a regular grid, then assembles the resulting instance-sensitive score maps into candidate segments.
Results
InstanceFCN yields compelling instance segment proposal results on PASCAL VOC and MS COCO, comparing favorably with several proposal methods.
Takeaways & Limitations
Instance segmentation can be addressed in the FCN fashion without high-dimensional layers dependent on mask resolution.
Abstract
from arXiv · showhide
Fully convolutional networks (FCNs) have been proven very successful for semantic segmentation, but the FCN outputs are unaware of object instances. In this paper, we develop FCNs that are capable of proposing instance-level segment candidates. In contrast to the previous FCN that generates one score map, our FCN is designed to compute a small set of instance-sensitive score maps, each of which is the outcome of a pixel-wise classifier of a relative position to instances. On top of these instance-sensitive score maps, a simple assembling module is able to output instance candidate at each position. In contrast to the recent DeepMask method for segmenting instances, our method does not have any high-dimensional layer related to the mask resolution, but instead exploits image local coherence for estimating instances. We present competitive results of instance segment proposal on both PASCAL VOC and MS COCO.
1 Introduction
InstanceFCN extends fully convolutional networks from semantic segmentation to instance segment proposal by classifying relative positions and assembling instance-sensitive score maps. It avoids DeepMask’s mask-resolution-dependent high-dimensional layer and achieves compelling results on PASCAL VOC and MS COCO.
- Motivation: FCNs produce effective pixel-wise semantic score maps but cannot directly distinguish or produce object instances.Previous instance segmentation methods generally relied on off-the-shelf segment proposal methods.
- Method: InstanceFCN replaces one category score map with instance-sensitive maps whose pixels classify relative positions within object instances.A regular k×k grid yields k^2 maps; for k = 3, the network produces 9 maps.
- Method: A simple assembling module combines outputs from the relative-position maps to generate an instance segment in each sliding window.The assembled output forms a candidate instance while preserving a fully convolutional pipeline.
- Comparison with DeepMask: Unlike DeepMask, InstanceFCN has no high-dimensional layer whose size depends on the mask resolution.DeepMask uses an m^2-dimensional fully connected layer, for example m = 56 and m^2 = 3136.
- Results: InstanceFCN obtains compelling instance segment proposal results on both PASCAL VOC and MS COCO, comparing favorably with several proposal methods.The authors also report competitive instance semantic segmentation when using downstream classifiers.
2 Related Work
Related work established fully convolutional prediction for semantic segmentation, restoration, and box-level proposals, but these approaches did not directly provide mask-level instance proposals.
- Fully convolutional models: Fully convolutional models apply sliding-window computation as convolution and support pixel-wise regression for image restoration tasks.Examples include denoising and super-resolution, where each output pixel predicts an intensity value.
- Semantic segmentation: FCNs enabled end-to-end, pixel-to-pixel semantic segmentation by treating each output pixel as a classifier for its receptive field.However, the original FCN cannot distinguish object instances.
- Region proposals: The fully convolutional RPN in Faster R-CNN proposes box-level instances using bounding-box regression and objectness classification.It does not generate mask-level proposals.
3 Instance-sensitive FCNs for Segment Proposal
InstanceFCN extends the FCN paradigm to instance proposals by classifying relative positions, assembling masks from a small set of score maps, and scoring each window for objectness. Its design exploits local coherence so mask prediction remains independent of mask resolution and uses fewer parameters than a sliding fully connected layer.
- Instance-sensitive score maps: Relative-position classifiers distinguish neighboring object instances that a conventional semantic FCN cannot separate.Each output pixel classifies whether it belongs to a relative position such as an instance’s left or right side.
- Instance-sensitive score maps: A k×k grid defines k2 instance-sensitive score maps, with each map corresponding to one relative position.For example, k = 3 produces 9 score maps that are trained end-to-end.
- Instance assembling module: The assembling module copies corresponding sub-windows from the score maps and places them according to relative position to form an m×m instance mask.The module is used during training and inference; inference densely slides windows to predict an instance at each position.
- Local coherence: Local coherence lets neighboring windows reuse predictions for the same image pixel, unlike DeepMask’s sliding m2-dimensional fully connected layer.Reuse is exact except for a few pixels near relative-position partition boundaries.
- Local coherence: Because mask-prediction layers are independent of m×m resolution, InstanceFCN reduces computational cost and parameter count, lowering overfitting risk on small datasets.The paper reports that its mask-prediction layer can have hundreds times fewer parameters than DeepMask.
- Network architecture: The architecture uses separate fully convolutional branches for instance-sensitive maps and per-window objectness scoring, followed by assembly.The objectness branch produces one score for each sliding window that generates one instance.
4 Experiments
Experiments on PASCAL VOC and MS COCO show that InstanceFCN provides competitive segment proposals and semantic instance segmentation while using substantially fewer mask-generation parameters than DeepMask. Performance is robust to the relative-position grid size and improves with full-image training.
- Evaluation: Average Recall (AR) measures proposal quality across mask-level IoU thresholds from 0.5 to 1.0 at fixed proposal counts.The reported metrics are denoted AR@N, where N is the number of proposals.
- Relative-position ablation: k = 3 already performs well, while k = 5 and k = 7 are comparable and slightly outperform the k = 3 baseline.The gain from finer relative-position division saturates around k = 5, which is used thereafter.
- Comparison with DeepMask: 53M parameters are used by the controlled ∼DeepMask fully connected mask-generation layers, versus 0.1M in InstanceFCN’s final k^2-dimensional convolutional layer.The InstanceFCN layer has 1/500 as many parameters as DeepMask’s corresponding fully connected layers.
- Comparison with DeepMask: InstanceFCN outperforms ∼DeepMask under crop-based training, and full-size-image training improves its result further.The latter gain is presented as evidence of the benefit of the fully convolutional scheme.
- Segment proposals: CNN-based methods, including InstanceFCN, ∼DeepMask, and MNC, perform better than Selective Search and MCG on PASCAL VOC proposals.InstanceFCN has AR@100 and AR@1000 similar to MNC, but its AR@10 is 5.5% higher.
- Semantic instance segmentation: 43.0% mAP@0.7 is InstanceFCN’s highest score among competitors, 1.5% above the closest competitor; its mAP@0.5 is second-best.Its mAP@0.5 result is lower than MNC’s, while the reported two-step-training result is 61.5% versus 60.2% for MNC’s step-by-step counterpart.
- MS COCO: InstanceFCN achieves higher AR scores than DeepMask and DeepMaskZoom on the first 5k MS COCO validation images.The comparison uses the multiple training and testing scales used in DeepMask’s evaluation.
5 Conclusion
The conclusion presents InstanceFCN as a fully convolutional approach that proposes instances from relative-position score maps and a simple assembling module. It avoids mask-resolution-dependent high-dimensional layers while relying on local coherence.
- Conclusion: InstanceFCN proposes segment instances by classifying pixels according to their relative positions to instances.This produces a set of instance-sensitive score maps.
- Conclusion: A simple assembling module generates segment instances from the instance-sensitive score maps.The module is part of the fully convolutional proposal pipeline.
- Conclusion: The architecture handles instance segmentation without high-dimensional layers whose dimensions depend on mask resolution.This design is contrasted with mask-generation approaches that use such layers.