Source-linked AI summary
Convolutional Feature Masking for Joint Object and Stuff Segmentation
Jifeng Dai, Kaiming He, Jian Sun
TL;DR
Existing raw-image masking methods risk artificial boundaries and require extensive computation. The paper masks proposal segments on convolutional feature maps, extends this representation to joint object and stuff segmentation, and reports state-of-the-art benchmark results with major speed gains.
Problem
Raw-image masking can introduce artificial boundaries that may degrade segment features, while processing thousands of image regions is time-consuming.
Method
The method projects proposal segments onto convolutional feature maps, masks those features, and represents stuff as compact combinations of multiple segment features.
Results
The method achieves state-of-the-art PASCAL VOC object-segmentation results and substantially outperforms previous state of the art on PASCAL-CONTEXT, with approximately 150× faster processing than SDS in one setting.
Takeaways & Limitations
Computing feature maps once enables effective segment features without masking raw-pixel images and supports object and stuff segmentation in one framework.
Takeaways & Limitations
The authors identify improving object detection with convolutional feature masking and exploiting context from joint object and stuff segmentation as future work.
Abstract
from arXiv · showhide
The topic of semantic segmentation has witnessed considerable progress due to the powerful features learned by convolutional neural networks (CNNs). The current leading approaches for semantic segmentation exploit shape information by extracting CNN features from masked image regions. This strategy introduces artificial boundaries on the images and may impact the quality of the extracted features. Besides, the operations on the raw image domain require to compute thousands of networks on a single image, which is time-consuming. In this paper, we propose to exploit shape information via masking convolutional features. The proposal segments (e.g., super-pixels) are treated as masks on the convolutional feature maps. The CNN features of segments are directly masked out from these maps and used to train classifiers for recognition. We further propose a joint method to handle objects and "stuff" (e.g., grass, sky, water) in the same framework. State-of-the-art results are demonstrated on benchmarks of PASCAL VOC and new PASCAL-CONTEXT, with a compelling computational speed.
1. Introduction
The paper replaces raw-image masking with convolutional feature masking to avoid artificial boundaries and reduce computation, while extending the framework to joint object and stuff segmentation.
- Motivation: Raw-image R-CNN methods extract segment features from masked image content, but artificial boundaries may degrade feature quality.These methods concatenate region and segment features for classifier training.
- Motivation: Thousands of masked or unmasked raw image regions must be processed, making existing segmentation pipelines time-consuming.The computational burden remains substantial even on high-end GPUs.
- Motivation: SPP-Net motivates computing convolutional feature maps once and forming cropped features for classification, prompting the question of whether semantic segmentation can use feature maps only.SPP-Net reported competitive detection accuracy with approximately 50× faster speed.
- Method: Convolutional feature masking projects proposal segments onto the last convolutional feature maps, applies binary masks, and feeds the masked features into fully connected layers.Because features come from the unmasked image and maps are computed once, the method addresses both feature-quality and efficiency issues.
- Joint segmentation: Stuff classes such as sky, grass, and water are represented as compact combinations of multiple segment features rather than a single box or segment.This training procedure places object and stuff segmentation in the same framework.
- Results: The method reports state-of-the-art PASCAL VOC 2012 object-segmentation results and substantially outperforms previous state of the art on PASCAL-CONTEXT.It is also reported as the first deep-learning-based method applied to PASCAL-CONTEXT for both object and stuff segmentation.
2. Convolutional Feature Masking
The paper develops convolutional feature masking to extract segment features from full-image convolutional maps, then evaluates two designs and their accuracy, proposal, scale, and speed trade-offs.
- Motivation: SPP-Net shows that full-image convolutional maps can provide localized features, motivating irregular segment masks instead of rectangular regions.Local regions encode semantic and spatial information, while SPP masks rectangular areas and produces fixed-length features.
- Convolutional Feature Masking: CFM projects binary segment masks onto the last convolutional feature map, averages and thresholds assignments, then multiplies the mask across channels.The resulting segment features are passed through SPP and fully connected layers for recognition.
- Network Designs: Design A applies CFM on the last convolutional layer and uses separate regional and segment pathways, whereas Design B masks the finest SPP level and uses one pathway.Design B reduces computational cost and over-fitting risk by retaining a single fully connected pathway.
- Ablation and Setup: The no-CFM baseline performs worse, while Designs A and B are comparable; the paper therefore adopts Design B for ZF SPPnet.Both designs use region proposals and associated segments, with convolutional maps extracted once from full images and then fixed.
- Accuracy: 55.4 mean IoU is achieved on the PASCAL VOC 2012 test set with ZF SPPnet and MCG, 3.8% above SDS, while VGG reaches 61.8.The method also reports mean APr values of 53.2 with ZF SPPnet and 60.7 with VGG on the validation set.
- Efficiency and Proposals: ~150× faster than SDS is achieved with one-scale ZF SPPnet, while the single-scale variant has negligible accuracy degradation.The speed gain comes from computing convolutional feature maps only once; more accurate MCG proposals also improve accuracy over Selective Search.
3. Joint Object and Stuff Segmentation
The framework extends convolutional feature masking to stuff by representing each stuff category as a compact combination of segment proposals. Segment pursuit generates training samples for joint object-and-stuff segmentation, achieving strong PASCAL-CONTEXT results and retaining efficient shared computation.
- Stuff representation: Stuff regions have arbitrary shapes and consistent colors or textures, making rectangular boxes or single segments unsuitable representations.Objects are countable and shape-consistent, whereas stuff includes categories such as grass, sky, and water.
- Joint framework: The joint framework computes convolutional feature maps once, so adding stuff segmentation incurs little extra computational cost.Object and stuff samples are handled in the same framework, with stuff samples supplied by segment pursuit.
- Stuff representation: The method represents each stuff category as a compact combination of multiple segment proposals within the masked-feature framework.Several proposals can collectively cover a stuff region, with fewer segments preferred.
- Segment pursuit: Segment pursuit selects compact proposals by starting from high-purity candidates, then sequentially selecting segments while inhibiting highly overlapping proposals.Candidate proposals have purity scores above 0.6, and the inhibition threshold is IoU=0.2.
- Segment pursuit: Because deterministic pursuit yields few samples, stochastic pursuit samples proposals with area-proportional probabilities to provide varied compact combinations for fine-tuning.Figure 4 contrasts uniform sampling, deterministic pursuit, and stochastic pursuit for fine-tuning.
- Results: 34.4 overall mean IoU is achieved on PASCAL-CONTEXT with VGG and MCG, improving over 26.6 for CFM with segment pursuit using ZF and SS.The comparison evaluates 60 categories, including a reported subset of 33 easier categories.
- Results: 26.6 overall mean IoU with CFM and segment pursuit exceeds 24.0 without segment pursuit and 20.7 for the no-CFM baseline.O2P obtains 18.1 overall, while SuperParsing reports 15.2 on the easier 33 categories and no overall score.
- Results: The method can also generate instance-wise object results, although PASCAL-CONTEXT evaluation reports only mean IoU scores.An external MIT-Adobe FiveK evaluation produces reasonably good results despite no training on that dataset.
4. Conclusion
The paper presents convolutional feature masking as a way to exploit shape information at a late network stage and applies it to joint object and stuff segmentation. It also reports cross-dataset visual results and identifies object-detection improvement as future work.
- Convolutional feature masking exploits shape information at a late stage in the network.
- The trained model produces visual results for cross-dataset joint object and stuff segmentation.The network is trained on PASCAL-CONTEXT and applied on MIT-Adobe FiveK.
- The method is applicable to joint object and stuff segmentation.
- Improving object detection with convolutional feature masking remains future work.