Source-linked AI summary
High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
Zifeng Wu, Chunhua Shen, Anton van den Hengel
TL;DR
Whether very deep residual networks can be configured effectively for semantic segmentation remains an open question, with GPU-memory limits complicating the use of large field-of-view and high-resolution features. The paper builds fully convolutional residual networks, simulates high-resolution processing with a lower-resolution model, and trains with online hard-pixel bootstrapping. It reports a 78.3% mean intersection-over-union score on PASCAL VOC 2012 and best results on Cityscapes.
Problem
Whether fully convolutional networks built from improved residual networks can be configured effectively for segmentation remains open, while their depth poses GPU-memory limits.
Source: §1 · p. 2Method
The paper evaluates fully convolutional residual-network configurations and combines high-resolution simulation, online hard-pixel bootstrapping, and dropout regularisation.
Source: §1 · p. 2Results
78.3% mean intersection-over-union on PASCAL VOC 2012 is reported as a new record, alongside currently best results on VOC and Cityscapes.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that large field-of-view and high-resolution feature maps are important, and that online hard-pixel mining significantly improves accuracy.
Source: §5 · p. 10
Abstract
from arXiv · showhide
We propose a method for high-performance semantic image segmentation (or semantic pixel labelling) based on very deep residual networks, which achieves the state-of-the-art performance. A few design factors are carefully considered to this end. We make the following contributions. (i) First, we evaluate different variations of a fully convolutional residual network so as to find the best configuration, including the number of layers, the resolution of feature maps, and the size of field-of-view. Our experiments show that further enlarging the field-of-view and increasing the resolution of feature maps are typically beneficial, which however inevitably leads to a higher demand for GPU memories. To walk around the limitation, we propose a new method to simulate a high resolution network with a low resolution network, which can be applied during training and/or testing. (ii) Second, we propose an online bootstrapping method for training. We demonstrate that online bootstrapping is critically important for achieving good accuracy. (iii) Third we apply the traditional dropout to some of the residual blocks, which further improves the performance. (iv) Finally, our method achieves the currently best mean intersection-over-union 78.3\% on the PASCAL VOC 2012 dataset, as well as on the recent dataset Cityscapes.
1 Introduction
The paper studies how to build effective fully convolutional segmentation networks from very deep residual networks despite GPU-memory constraints. It evaluates network configuration and proposes high-resolution simulation, online bootstrapping, and dropout; it reports a 78.3% mean intersection-over-union score on PASCAL VOC 2012.
1 Introduction
- The study evaluates residual-network depth, feature-map resolution, and field-of-view; larger field-of-view and higher resolution are generally beneficial but demand more GPU memory.The proposed workaround simulates a high-resolution network with a low-resolution one during training and/or testing.Source: §1 · p. 2
- Online bootstrapping is proposed for training and described as critically important for achieving the best performance.The paper frames this as a missing topic in adapting fully convolutional networks to improved residual networks.Source: §1 · p. 2
- 78.3% mean intersection-over-union on PASCAL VOC 2012 is reported as a new record, with currently best results also reported on Cityscapes.The authors also report that dropout regularisation further improves performance.Source: §1 · p. 2
2 Related work
Related work covers very deep convolutional networks, fully convolutional segmentation models and context methods, and training-time selection of hard examples. The paper positions online hard-pixel bootstrapping as a new application of that sampling idea to semantic segmentation.
Very deep convolutional networks
- ResNets achieved leading image-classification performance, while shortcut connections help very deep networks avoid vanishing gradients and train deeper models.A later redesign enabled a 200-layer residual network.Source: §2 · pp. 2–3
Fully convolutional networks for semantic segmentation
- FCNs were effective and efficient for semantic segmentation; Chen et al. raised feature-map resolution by removing some downsampling operations and introducing kernel dilation, and found reducing field-of-view hurt performance.Chen et al. used dense CRFs to refine predicted category score maps. Lin et al. [3] jointly trained CRFs with down-lying convolution layers to capture patch-patch and patch-background context.Source: §2 · p. 3
- Prior context methods include an RNN simulation of dense CRFs and joint CRF training to capture patch-patch and patch-background context [3].Source: §2 · p. 3
Online bootstrapping for training deep convolutional networks
- Earlier sampling methods selected hard images across a training set [15] or hard RoIs within an image; this paper instead mines hard pixels within the current mini-batch.The authors describe their work as the first online bootstrapping of hard pixel samples for semantic image segmentation.Source: §2 · p. 3
3 Our method
The method adapts ResNet into a fully convolutional residual network, then addresses field-of-view, resolution, and GPU-memory constraints and trains by focusing on hard pixels. Its high-resolution simulation uses multiple shifted passes through a lower-resolution model.
3.1 Fully convolutional residual network
- The FCRN replaces ResNet's linear classifier with a convolution layer and removes the 7×7 pooling layer; dilated kernels then support higher-resolution maps and a large field-of-view.The hole algorithm skips down-sampling and increases subsequent kernel dilation without changing weights; small dilated kernels can realize a large field-of-view.Source: §3.1 · pp. 3–4
3 Our method
- The baseline FCRN is built from FCN and ResNet ideas, but the authors report that it does not beat the best segmentation algorithm based on VGGNet [5].Source: §3 · p. 3, §3.1 · p. 4
3.2 Training of a large network with limited GPU memories
- Figure 1 illustrates simulating a high-resolution model with a low-resolution model to break the compromise between large fields of view, which require lower-resolution feature maps, and higher resolution, which requires smaller crops.During training, the method shifts the 1/4-resolution feature maps horizontally by a stride of one before down-sampling, and during testing places the four passes’ scores in corresponding locations on 1/4-resolution score maps.Source: §3.2 · pp. 4–5, Figure 1 · p. 5
3.3 Online bootstrapping of hard training pixels
- Online bootstrapping focuses training on hard pixels because easy pixels, such as those near the center of large semantic regions, contribute little to improving the segmentation objective.The loss uses a threshold t; the threshold is increased when a mini-batch is handled well to retain a reasonable number of pixels.Source: §3.3 · p. 5
4 Experiments
The experiments evaluate fully convolutional residual networks on PASCAL VOC 2012 and Cityscapes, examining configurations, feature-map resolution, online bootstrapping, and comparisons with prior methods. The VOC-only model outperforms the previous best performer, while the experiments also identify settings and training choices that affect performance.
4.1 Datasets
- The method is evaluated on PASCAL VOC 2012 and Cityscapes, with mean class-wise intersection over union (IoU) reported when possible; VOC test results provide only mean IoU.VOC has twenty semantic categories besides background and image side lengths no larger than 500 pixels; Cityscapes has nineteen semantic categories and images 1024 pixels high by 2048 pixels wide.Source: §4.1 · pp. 5–6
4.2 Implementation details
- The implementation initializes FCRNs from ResNet-50, ResNet-101, or ResNet-152 and uses random resizing and cropping for data augmentation.The method is implemented in Caffe, with SGD hyperparameters evaluated on the validation sets.Source: §4.2 · p. 6
4.3 Results of the vanilla FCRN
- On VOC, increasing depth from 50 to 101 improves performance, but depth 152 overfits; among tested settings, FoV 392 performs best.The FoV is constrained by image and training-crop sizes, and dilation 18 offers no clear advantage over dilation 12.Source: §4.3 · p. 6, Table 1 · p. 7
- Cityscapes shows lighter overfitting than VOC, and its larger images make FoV 392 smaller than the optimum; ideal FoV depends partly on crop size.The authors suggest image resolution and dataset diversity as possible reasons for the difference in overfitting.Source: §4.3 · p. 7
4.4 Impact of the feature map resolution
- At the same test resolution, networks trained at 1/8 resolution outperform those trained at 1/16; increasing prediction resolution at test time also moderately improves results.The authors propose that finer training labels help a very deep network discriminate pixels near semantic boundaries.Source: §4.4 · p. 7, Table 3 · p. 8
4.5 Impact of online bootstrapping of hard training pixels
- For the tested crop sizes, keeping the 512 hardest valid pixels is best; larger crops call for keeping more, while smaller crops call for fewer.Bootstrapping improves results for less frequent categories, including cow and horse on VOC and traffic light and train on Cityscapes; dropout also lets the 152-layer network outperform the 101-layer network.Source: §4.5 · pp. 7–8
4.6 Comparison with previous state-of-the-art
- 2.0%: On VOC 2012, the VOC-only method outperforms the previous best performer; COCO pre-training adds 1.0% and wins first place in 13 of 20 categories.The comparison also reports first place in 12 of 20 categories when trained only with VOC data.Source: §4.6 · pp. 8–9, Table 5 · p. 9
4.7 Discussions
- Larger learning rates for added convolution layers and random color noise show no obvious improvement; fewer images per mini-batch have no obvious adverse impact.The authors note that their whole-crop noise differs from the prior approach of adding different noise to images within a mini-batch.Source: §4.7 · pp. 9–10
5 Conclusions
The paper explores fully convolutional residual networks for semantic segmentation, addressing GPU-memory limits with high-resolution simulation and improving accuracy through online hard-pixel mining. It reports a state-of-the-art mean IoU on PASCAL VOC 2012.
5 Conclusions
- The method achieves a state-of-the-art mean IoU on PASCAL VOC 2012.Source: §5 · p. 10
- The authors find large fields of view and high-resolution feature maps important, and simulate a high-resolution network with a low-resolution one to address GPU-memory limits.Source: §5 · p. 10
- Online bootstrapping mines hard training pixels and significantly improves accuracy.Source: §5 · p. 10