Source-linked AI summary
Proposal-free Network for Instance-level Object Segmentation
Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Jianchao Yang, Liang Lin, Shuicheng Yan
TL;DR
Instance-level object segmentation requires accurate masks for individual objects, while prior methods largely depend on difficult region proposals and multi-stage processing. PFN predicts category confidences, per-pixel instance-box locations, and category instance counts in an end-to-end network, then clusters these outputs to form segments. On PASCAL VOC 2012, PFN reaches 58.7% AP^r at 0.5 IoU and exceeds prior published methods, though heavy occlusion and small objects remain difficult.
Problem
Instance-level segmentation is challenging because scenes contain occlusion, varied shapes and appearances, obscured boundaries, clutter, and varying instance counts.
Method
PFN jointly predicts category-level confidences, each pixel's instance bounding-box coordinates, and instance numbers, using clustering instead of region proposals.
Results
58.7% AP^r at 0.5 IoU on PASCAL VOC 2012 surpasses the reported state of the art, improving from 46.3%.
Takeaways & Limitations
PFN produces fine-grained instance masks with simpler implementation and lower computational cost than prior methods requiring complicated pre- and post-processing.
Takeaways & Limitations
Heavy occlusion and small object instances remain difficult for PFN to identify and segment.
Abstract
from arXiv · showhide
Instance-level object segmentation is an important yet under-explored task. The few existing studies are almost all based on region proposal methods to extract candidate segments and then utilize object classification to produce final results. Nonetheless, generating accurate region proposals itself is quite challenging. In this work, we propose a Proposal-Free Network (PFN ) to address the instance-level object segmentation problem, which outputs the instance numbers of different categories and the pixel-level information on 1) the coordinates of the instance bounding box each pixel belongs to, and 2) the confidences of different categories for each pixel, based on pixel-to-pixel deep convolutional neural network. All the outputs together, by using any off-the-shelf clustering method for simple post-processing, can naturally generate the ultimate instance-level object segmentation results. The whole PFN can be easily trained in an end-to-end way without the requirement of a proposal generation stage. Extensive evaluations on the challenging PASCAL VOC 2012 semantic segmentation benchmark demonstrate that the proposed PFN solution well beats the state-of-the-arts for instance-level object segmentation. In particular, the $AP^r$ over 20 classes at 0.5 IoU reaches 58.7% by PFN, significantly higher than 43.8% and 46.3% by the state-of-the-art algorithms, SDS [9] and [16], respectively.
1 INTRODUCTION
Instance-level object segmentation seeks pixel-accurate masks for each object instance, but existing approaches rely on challenging proposals and multi-stage processing. PFN instead predicts category information, per-pixel instance locations, and instance counts for end-to-end proposal-free clustering, achieving 58.7% AP^r at 0.5 IoU on PASCAL VOC 2012.
- Motivation: Instance-level segmentation predicts a pixel-wise mask for every instance of each category, beyond detection boxes or category-only semantic labels.The task is motivated by applications requiring detailed scene parsing, including image captioning, retrieval, 3-D navigation, and driver assistance.
- Motivation: Prior methods commonly require region proposals, classification, and complex preprocessing or post-processing, preventing excellent end-to-end performance.Proposal generation and classification optimize different targets, such as region recall versus single-class accuracy.
- Proposal-Free Network: PFN directly infers instance regions from global image context without a traditional region-proposal stage.Pixels predicting the same instance locations can be clustered together, while differences in predicted locations can help infer occluded boundaries.
- Proposal-Free Network: PFN jointly addresses category-level segmentation, per-pixel instance-location prediction, and instance-number prediction.Instance locations represent the bounding-box coordinates of the instance containing each pixel; instance numbers specify the cluster count for each category.
- Results: 58.7% AP^r on PASCAL VOC 2012 improves the reported state of the art from 46.3%, while PFN uses a simpler processing pipeline.At test time, predicted outputs are combined through clustering to produce instance-level segmentation results.
2 RELATED WORK
Earlier object-recognition and instance-segmentation systems typically combine proposal generation, classification, and additional inference stages. PFN replaces this decomposition with jointly optimized pixel-wise instance locations and instance counts, followed by simple clustering to produce fine-grained masks.
- Object Detection: Object-detection pipelines generally extract box proposals and then recognize or localize objects with classifiers or localizers.These outputs provide coarse bounding-box localization rather than fine-grained instance masks.
- Object Detection: PFN groups pixels that infer similar instance locations into one region and produces fine-grained masks instead of detector-style coarse boxes.Pixel-wise locations and category instance counts are optimized simultaneously in one network.
- Semantic Segmentation: Semantic-segmentation methods assign category labels to pixels but do not identify separate object instances.Prior semantic-segmentation systems commonly fine-tune pretrained classification networks using category-level masks.
- Instance-level Object Segmentation: Most prior instance-segmentation approaches use region proposals as a requisite and may add integer-programming or probabilistic post-processing.These proposal-based pipelines combine multiple stages to determine instance segments.
- Instance-level Object Segmentation: PFN contrasts with proposal-based systems by directly predicting pixel-wise instance-location maps and applying simple clustering to generate instance segments.The paper motivates this design as a way to combine local and global context within one network rather than independent stages.
3 PROPOSAL-FREE NETWORK
PFN jointly predicts category-level segmentation, pixel-wise instance locations, and per-category instance numbers in a unified instance-level network. These outputs are trained with dedicated objectives and combined to support instance segmentation without relying on shared late-stage features between category- and instance-level predictions.
- Network Architecture: PFN uses category-level segmentation, pixel-wise instance locations, and per-category instance numbers as three training targets.The category-level network is initialized from VGG-16 and uses C + 1 confidence maps for dense segmentation.
- Network Training: PFN is optimized in two stages because category-level segmentation seeks category-insensitive predictions, whereas instance-level segmentation distinguishes individual instances.The paper states that late convolutional feature maps cannot be shared effectively between these differently motivated targets.
- Pixel-wise Instance Location Prediction: Each pixel predicts six instance-location coordinates: the box center, top-left corner, and bottom-right corner.The redundant coordinate representation is intended to improve robustness to noise and inaccurate predictions.
- Pixel-wise Instance Location Prediction: Instance-location regression is activated only for foreground pixels because only those pixels belong to a specific object instance.The loss uses a robust smooth-L1 regression formulation and is normalized by the number of foreground pixels.
- Pixel-wise Instance Location Prediction: Five multi-scale prediction streams attach to image and intermediate feature maps, use individual supervision, and preserve feature-map resolution.The streams combine local details from early layers with global context from deeper layers.
- Instance Number Prediction: The network predicts a real-valued instance-number vector for all categories and jointly optimizes it with pixel-wise instance-location prediction.The instance-number vector contains both category-presence and per-category instance-count information; λ is empirically set to 10.
4 EXPERIMENTS
PFN is evaluated on PASCAL VOC 2012 through benchmark comparisons and component ablations. The experiments show strong performance, efficient inference, and measurable contributions from its architectural and post-processing choices.
- Results and Comparisons: PFN processes a 300 × 500 image in about one second, compared with about 40 seconds for SDS.The competing methods use region-proposal pre-processing and complex post-processing steps.
- Results and Comparisons: PFN achieves 15.7% AP^r at 0.9 IoU, compared with 0.9% for SDS and 2.6% for.The result is reported for strict localization evaluation on PASCAL VOC 2012.
- Results and Comparisons: PFN obtains higher AP^r than the baselines for small or heavily occluded instances, including bird at 74.2% and sofa at 64.4%.The cited comparisons report bird against 60.1% and 61.5%, and sofa against 26.9% and 33.5%.
- Ablation Studies: PFN’s multi-scale fusion improves AP^r by 3.5% over its ablated variant, incorporating local fine details and global semantic information.Spatial coordinates also contribute: removing them produces 55.3% versus 58.7% AP^r during clustering.
- Ablation Studies: 51.4% AP^r without category-level information and 57.9% without instance-level information trail PFN’s 58.7%.The ablation supports using both category-level information and pixel-wise instance-location information for instance-number prediction.
- Ablation Studies: 58.7% AP^r for PFN exceeds 56.4% without classification and size-constrained clustering, showing that both refinement steps improve segmentation.Removing only the size constraint decreases performance by 0.9%, while removing instance-number prediction reduces it by 1.3%.
5 CONCLUSION AND FUTURE WORK
PFN provides fine-grained instance-level segmentation without requiring extra region-proposal methods, using predicted instance locations and counts with simple spectral clustering. On PASCAL VOC 2012, it achieves significant improvements while remaining simpler and less computationally costly than prior state-of-the-art methods.
- PFN directly predicts each pixel’s instance location and the instance numbers of all categories instead of requiring extra region-proposal methods.
- Pixels with the same or close predicted instance locations are clustered into object instances, with predicted category-wise counts indicating the number of clusters.
- PFN achieves significant improvements over state-of-the-art methods on the PASCAL VOC 2012 segmentation benchmark.
- PFN avoids complicated pre-processing and post-processing requirements, making it simpler to implement with lower computational cost than previous state-of-the-art methods.
- The authors plan to extend PFN to generic multiple-instance segmentation in outdoor and indoor scenes with greater clutter and occlusion.