Source-linked AI summary
Efficient piecewise training of deep structured models for semantic segmentation
Guosheng Lin, Chunhua Shen, Anton van dan Hengel, Ian Reid
TL;DR
Semantic segmentation needs contextual information beyond CNN-based pixel prediction, especially for ambiguous image regions and relationships between patches and backgrounds. The paper models patch-patch context with CNN-based CRF pairwise potentials, uses piecewise training for efficient learning, and encodes background context with multi-scale networks and sliding pyramid pooling. It reports state-of-the-art performance across several datasets, including 78.0 intersection-over-union on PASCAL VOC 2012.
Problem
CNN-based segmentation methods had not well studied explicit patch-patch contextual relations, despite contextual information providing important cues for scene understanding.
Method
The method combines CNN-based unary and general pairwise CRF potentials with piecewise training, while using multi-scale networks and sliding pyramid pooling for patch-background context.
Results
78.0 intersection-over-union is achieved on PASCAL VOC 2012, with state-of-the-art performance also reported on NYUDv2, PASCAL-Context, and SIFT-flow.
Takeaways & Limitations
CNN-CRF modeling of complex contextual information produces best performance on several popular semantic segmentation datasets.
Takeaways & Limitations
The prediction refinement stage uses bilinear upsampling and boundary refinement, and the paper identifies more sophisticated refinement methods as a possible source of further improvement.
Abstract
from arXiv · showhide
Recent advances in semantic image segmentation have mostly been achieved by training deep convolutional neural networks (CNNs). We show how to improve semantic segmentation through the use of contextual information; specifically, we explore `patch-patch' context between image regions, and `patch-background' context. For learning from the patch-patch context, we formulate Conditional Random Fields (CRFs) with CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied to avoid repeated expensive CRF inference for back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image input and sliding pyramid pooling is effective for improving performance. Our experimental results set new state-of-the-art performance on a number of popular semantic segmentation datasets, including NYUDv2, PASCAL VOC 2012, PASCAL-Context, and SIFT-flow. In particular, we achieve an intersection-over-union score of 78.0 on the challenging PASCAL VOC 2012 dataset.
1. Introduction
The paper improves semantic segmentation by modeling patch-patch and patch-background context, combining CNN-based CRF potentials with efficient training and multi-scale background features. It reports state-of-the-art results across several datasets, including an IoU of 78.0 on PASCAL VOC 2012.
- Semantic segmentation predicts a category label for every image pixel and remains an important, challenging image-understanding task.
- Patch-patch context captures semantic relations between image patches, while patch-background context captures relations between a patch and a large background region.
- CNN-based pairwise CRF potentials model semantic compatibility between neighboring patches rather than only enforcing local smoothness.
- Piecewise CRF training avoids repeated expensive inference during stochastic-gradient training, enabling more efficient CNN-CRF learning.
- The method uses multi-scale networks and sliding pyramid pooling to encode background information for segmentation.
- 78.0 intersection-over-union is achieved on PASCAL VOC 2012, alongside reported state-of-the-art performance on NYUDv2, PASCAL-Context, and SIFT-flow.
2. Modeling semantic pairwise relations
The architecture maps image regions to feature-map nodes in a CRF and connects spatially nearby nodes to model different relations. Separate CNN components produce unary and pairwise potentials for the structured prediction model.
- FeatMap-Net generates a lower-resolution feature map from the input image for constructing the CRF representation.
- Each feature-map location becomes a CRF node corresponding to a rectangular input-image region.
- Nodes connect to other nodes within a spatial range box, with different range-box types representing different spatial relations.
- The system trains three FeatMap-Net instances: one unary-potential network and two pairwise-potential networks.
3. Contextual Deep CRFs
The model uses CNN-generated unary and general pairwise potentials within a CRF to represent label configurations and semantic compatibility between connected image regions. It also combines contextual CNN potentials with dense-CRF boundary refinement.
- CRF formulation: The CRF energy uses unary and pairwise potentials over nodes and edges associated with image regions.Unary potentials score individual node labels, while pairwise potentials score connected-node label combinations.
- Network construction: FeatMap-Net generates a lower-resolution feature map that supplies node features for the potential networks.Unary-Net processes feature vectors at CRF-node locations, while Pairwise-Net receives features from connected node pairs.
- Pairwise potentials: Pairwise-Net outputs a K × K potential for every possible label pair, modeling semantic compatibility between two connected nodes.The formulation can represent asymmetric relations by preserving the order of the input node pair.
- Pairwise potentials: Unlike Potts-model pairwise terms, the CNN pairwise potentials individually parameterize every label combination rather than enforcing only neighborhood smoothness.This formulation targets semantic compatibility between image regions.
- Prediction refinement: The system applies dense CRF post-processing after contextual CNN potentials to refine object boundaries using low-level image information.The dense CRF supplies traditional smoothness potentials during prediction refinement.
4. Exploiting background context
The background-context design combines multi-scale image processing with sliding pyramid pooling. These components provide scale-dependent features and broader spatial context for segmentation.
- Effectiveness: The paper reports that multi-scale network design and pyramid pooling are effective for encoding background information and improving segmentation performance.The claim is empirical and concerns the combined background-context architecture.
- Multi-scale input: The FeatMap-Net uses three image scales, shared top convolution blocks, and scale-specific final blocks before concatenating aligned feature maps.The input scales are 1.2, 0.8, and 0.4; the two smaller feature maps are bilinearly upscaled.
- Sliding pyramid pooling: Sliding pyramid pooling captures background regions at multiple sizes and increases the feature map’s field-of-view.The design targets patch-background context and large image-region information.
- Sliding pyramid pooling: Two-level max-pooling uses 5×5 and 9×9 sliding windows, concatenating both pooled feature maps with the original feature map.This configuration is applied for each image scale.
5. Prediction
Prediction proceeds from a low-resolution structured-model output to a refined high-resolution result. Mean-field CRF inference supplies coarse predictions, followed by upsampling and boundary refinement.
- Prediction pipeline: The structured model produces a low-resolution prediction at 1/16 of the input image size because pooling and convolution layers subsample the image.The prediction pipeline therefore has separate coarse-level and refinement stages.
- Coarse-level prediction: Mean-field inference is applied to the contextual CRF to obtain node marginals for the coarse prediction.The graph is neither tree-structured nor submodular, so approximate inference is used; experiments perform 3 mean-field iterations.
- Prediction refinement: The coarse score map is bilinearly upsampled to the input-image size before dense-CRF boundary refinement produces the final prediction.The refinement uses low-level pixel-intensity or color-contrast information.
- Prediction refinement: The authors identify more sophisticated refinement methods as a source of potential further performance improvement.Suggested alternatives include deconvolution networks, coarse-to-fine learning networks, and middle-layer features.
6. CRF training
Direct CRF maximum-likelihood training is expensive because the partition function requires inference on a large loopy graph during many CNN updates. Piecewise training replaces this global objective with independently normalized potential terms.
- Direct CRF training: The CRF negative log-likelihood combines the observed energy with the log partition function, whose gradient complicates optimization.CNN parameters can be differentiated through the energy, but the partition function introduces the difficult term.
- Training challenge: The output space is exponential in the number of nodes, and repeated inference on the loopy segmentation graph is computationally expensive during tens or hundreds of thousands of SGD iterations.These factors prohibit direct calculation of the partition function and make per-iteration inference impractical.
- Piecewise training: Piecewise training formulates the CRF conditional likelihood as independent likelihoods defined on unary and pairwise potentials.The method is developed specifically for CRFs with CNN potentials.
- Piecewise training: The piecewise objective removes the global partition function and requires only local gradients for unary and pairwise likelihoods.Unary normalization is a Softmax over K classes, and the potential functions can be trained in parallel.
- Practical training: The approach provides a practical way to train CRFs with CNN potentials on large-scale data without expensive inference at every gradient update.This is the stated motivation for applying piecewise training in the deep structured model.
7. Experiments
Experiments across four semantic-segmentation datasets evaluate the proposed contextual model and its components. The method achieves leading performance, including a 78.0 IoU score on PASCAL VOC 2012.
- The method is evaluated on PASCAL VOC 2012, NYUDv2, PASCAL-Context, and SIFT-flow using IoU, pixel accuracy, and mean accuracy.
- NYUDv2: The NYUDv2 evaluation uses RGB images only, and the contextual model with CNN pairwise potentials achieves the dataset’s best performance without depth information.
- NYUDv2: Multi-scale input and sliding pyramid pooling significantly improve the fully convolutional baseline, while dense-CRF refinement and CNN pairwise potentials provide further gains on NYUDv2.
- PASCAL VOC 2012: 75.3 IoU is the best result among methods trained only on VOC images, while adding COCO images raises the score to 77.2.
- The method is reported as the best-performing approach on PASCAL-Context and SIFT-flow, and it outperforms competing methods in most PASCAL VOC categories.
8. Conclusions
The paper combines CNNs and CRFs to model complex contextual information for semantic image segmentation. CNN-based pairwise potentials support strong performance across several popular datasets, including PASCAL VOC 2012.
- The proposed method combines CNNs and CRFs, using CNN-based pairwise potentials to model semantic relations between image regions.
- The method achieves the best performance on several popular datasets, including PASCAL VOC 2012.