Source-linked AI summary
Bridging Category-level and Instance-level Semantic Image Segmentation
Zifeng Wu, Chunhua Shen, Anton van den Hengel
TL;DR
The paper addresses instance segmentation through a pipeline built on semantic category segmentation rather than detect-then-segment methods. It predicts per-pixel instance bounding boxes, uses online bootstrapping and fully convolutional residual networks, and reports best semantic and state-of-the-art instance segmentation results on PASCAL VOC 2012.
Problem
Instance segmentation commonly relies on bounding-box detection, while comparatively little work fully exploits powerful semantic category segmentation methods.
Method
The method transforms semantic score maps using per-pixel bounding-box regression and trains the networks with online bootstrapping of hard pixels.
Results
The method achieves the currently best mean IoU for semantic segmentation and state-of-the-art instance-level segmentation on PASCAL VOC 2012.
Takeaways & Limitations
Semantic segmentation quality and online bootstrapping are presented as important components of the reported performance in both semantic and instance segmentation.
Takeaways & Limitations
The authors leave domain adaptation from COCO to PASCAL VOC for future work and note that additional training data are needed for difficult categories.
Abstract
from arXiv · showhide
We propose an approach to instance-level image segmentation that is built on top of category-level segmentation. Specifically, for each pixel in a semantic category mask, its corresponding instance bounding box is predicted using a deep fully convolutional regression network. Thus it follows a different pipeline to the popular detect-then-segment approaches that first predict instances' bounding boxes, which are the current state-of-the-art in instance segmentation. We show that, by leveraging the strength of our state-of-the-art semantic segmentation models, the proposed method can achieve comparable or even better results to detect-then-segment approaches. We make the following contributions. (i) First, we propose a simple yet effective approach to semantic instance segmentation. (ii) Second, we propose an online bootstrapping method during training, which is critically important for achieving good performance for both semantic category segmentation and instance-level segmentation. (iii) As the performance of semantic category segmentation has a significant impact on the instance-level segmentation, which is the second step of our approach, we train fully convolutional residual networks to achieve the best semantic category segmentation accuracy. On the PASCAL VOC 2012 dataset, we obtain the currently best mean intersection-over-union score of 79.1%. (iv) We also achieve state-of-the-art results for instance-level segmentation.
1 Introduction
The paper proposes instance segmentation built on semantic category segmentation rather than the prevailing detect-then-segment pipeline. It introduces Hough-like score-map transforms and online hard-pixel bootstrapping, supported by fully convolutional residual networks.
- Instance segmentation identifies individual objects within semantic categories, combining object detection and segmentation.
- The proposed pipeline builds instance segmentation on semantic segmentation instead of first locating objects with bounding boxes and then predicting masks.
- The method transforms semantic score maps into Hough-like maps from which instances can be detected.
- 79.1% mean intersection-over-union is achieved on PASCAL VOC 2012 using augmented training data.
- Online bootstrapping selects hard pixel samples within the current mini-batch, and the paper presents it as a first such method for instance and semantic segmentation.
2 Proposed method
The proposed method combines semantic score maps with per-pixel bounding-box regression, then detects instance hypotheses and recovers masks. Online bootstrapping focuses training on difficult pixels for both semantic and localization networks.
- The pipeline first computes category-wise semantic score maps, predicts category-wise bounding-box transform maps, and applies the transforms to the scores.
- Local maxima found by non-maximum suppression become instance hypotheses, after which suppressed pixels are traced back to recover masks and region-based NMS produces the final result.
- The semantic and localization networks are trained separately with logistic regression and smoothed ℓ1 losses, respectively.
- 2.2 Online bootstrapping of hard training pixels: Online bootstrapping drops pixels that are too easy for the current model and adjusts the threshold to retain enough pixels for stable gradients.
- 2.2 Online bootstrapping of hard training pixels: For localization, online bootstrapping thresholds ground-truth and predicted bounding-box IoU rather than regression loss.
- 2.3 Fully convolutional residual network: The FCRN uses a convolutional prediction layer, removes 7×7 pooling, and uses dilated convolutions to produce higher-resolution feature maps.
3 Experiments
Experiments evaluate semantic segmentation across PASCAL VOC 2012, Cityscapes, and PASCAL-Context, then assess instance segmentation on PASCAL VOC 2012. The results show gains from deeper, higher-resolution models and online bootstrapping, with competitive or improved instance-segmentation performance.
- Experimental setup: The experiments evaluate semantic category segmentation on PASCAL VOC 2012, Cityscapes, and PASCAL-Context, and instance segmentation on PASCAL VOC 2012.The study reports semantic and instance-level evaluation using validation or test protocols described for each dataset.
- Semantic category-level segmentation results: Increasing network depth from 50 to 101 layers significantly improves PASCAL VOC 2012 validation performance, while 152 layers provide no further improvement.The authors attribute the lack of additional improvement at 152 layers probably to over-fitting.
- Semantic category-level segmentation results: Higher feature-map resolution and larger fields of view improve semantic segmentation, but deeper networks and larger fields of view increase computational cost and memory demands.The reported field-of-view effects depend partly on image-crop size and the original image resolution.
- Semantic category-level segmentation results: The semantic model outperforms the previous best PASCAL VOC 2012 result by 3.8% and ranks first for 18 of 20 object categories.The largest remaining difficulties involve diverse, occluded categories such as bicycle and chair; with COCO pre-training, the improvement is limited, while the authors leave domain adaptation for future work.
- Semantic category-level segmentation results: 74.6% IoU is achieved on PASCAL-Context validation, compared with the previously reported best result of 68.6%.The same model used for PASCAL VOC 2012 is evaluated on PASCAL-Context, where the authors report a clear margin over prior performers across three scores.
- Instance-level segmentation results: On PASCAL VOC 2012 instance segmentation, the method exceeds the previous best by 3.1% at mAPr 0.7 and gains a further 2.0% with COCO pre-training.The reported table includes mAPr 0.5, mAPr 0.7, and mAPr vol; ground-truth semantic masks raise performance to 73.0% mAPr 0.5 and 60.6% mAPr 0.7.
- Importance of online bootstrapping of hard training pixels: Online bootstrapping improves mean IoU by 3.1% on Cityscapes and also contributes to localization-network performance.The best semantic-segmentation settings retain the 512 hardest pixels, with especially clear gains for less frequent categories.
4 Conclusions
The work combines a fully convolutional residual network for semantic category-level segmentation with a distinct pipeline for instance-level segmentation. It also introduces online bootstrapping and reports leading results on PASCAL VOC 2012.
- Its instance-level segmentation pipeline is intrinsically different from commonly used bounding-box detection-based methods.
- Online bootstrapping contributes greatly to both semantic and instance-level segmentation.
- The paper reports the currently best mean IoU for semantic segmentation and state-of-the-art performance for instance-level segmentation on PASCAL VOC 2012.