Source-linked AI summary
Fully Connected Deep Structured Networks
Alexander G. Schwing, Raquel Urtasun
TL;DR
Semantic segmentation commonly separates convolutional unary-feature learning from graphical-model refinement. This paper jointly trains both components using deep convolutional networks and fully connected CRFs, achieving 64.060% validation mean intersection-over-union on PASCAL VOC 2012, while remaining challenged by noisy images and boundary quality.
Problem
Semantic-segmentation methods commonly train convolutional unary features and graphical-model refinement separately, despite the goal of combining local and global modeling.
Method
The paper jointly trains convolutional-network unary parameters and fully connected CRF compatibility and shape parameters using mean-field inference and back-propagation.
Results
64.060% validation mean intersection-over-union slightly outperforms the 63.74% separate-training result reported by Chen et al.
Takeaways & Limitations
Joint training of convolutional networks and fully connected CRFs produces good semantic-segmentation results on the challenging PASCAL VOC 2012 dataset.
Takeaways & Limitations
The method struggles with noisy images and varied objects, and learned parameters generally oversmooth results while producing noisy boundaries.
Abstract
from arXiv · showhide
Convolutional neural networks with many layers have recently been shown to achieve excellent results on many high-level tasks such as image classification, object detection and more recently also semantic segmentation. Particularly for semantic segmentation, a two-stage procedure is often employed. Hereby, convolutional networks are trained to provide good local pixel-wise features for the second step being traditionally a more global graphical model. In this work we unify this two-stage process into a single joint training algorithm. We demonstrate our method on the semantic image segmentation task and show encouraging results on the challenging PASCAL VOC 2012 dataset.
1 Introduction
Convolutional networks have become effective across vision tasks, including semantic segmentation, but segmentation methods commonly train local unary features and global graphical-model refinement separately. The paper introduces joint training of both components for semantic segmentation.
- CNNs achieve strong performance across image classification, detection, pose estimation, stereo, and caption generation through learned nonlinear dependencies.
- Semantic segmentation uses graphical-model refinement because pooling can reduce prediction dimensionality and weaken boundary-respecting outputs.
- Existing segmentation approaches train convolutional unary features and graphical-model components piecewise rather than jointly.
- The proposed algorithm jointly trains convolutional-network unary parameters and smoothness terms while accounting for dependencies between output variables.
- The method is evaluated on the PASCAL VOC 2012 semantic image-segmentation dataset.
2 Background
Deep structured prediction scores and normalizes exponentially large output spaces, then learns parameters through gradient-based optimization. Structured decompositions and message passing make this tractable in some cases, but densely connected segmentation models remain computationally difficult.
- A structured model assigns scores to output configurations and learns parameters by maximizing training-set likelihood, equivalently cross-entropy.
- For simple output spaces, prediction searches configurations and probability normalization sums exponentiated scores over all possibilities.
- Gradient-based training iterates forward scoring, soft-max normalization, loss-gradient back-propagation, and parameter updates.
- Local-function decompositions efficiently represent global scores when each restriction contains only a small subset of variables.
- Message passing computes the marginals needed for likelihood gradients, exactly at low tree-width and approximately otherwise.
- Dense image-segmentation models challenge this approach because many restrictions are required and every pixel may correlate with every other pixel.
3 Approach
The approach combines deep structured prediction with fully connected probabilistic models using mean-field inference, efficient Gaussian filtering, and back-propagation through iterative marginals. It permits arbitrary unary functions such as convolutional networks while restricting pairwise terms to kernel mixtures.
- The method extends fully connected log-linear models to arbitrary scoring functions, including deep convolutional neural networks.
- Pairwise functions are restricted to mixtures of feature-space kernels, trading general-method computational complexity for efficient dense-model processing.
- Inference: Mean-field inference factorizes the model distribution and computes single-variable marginals by minimizing KL divergence.
- Inference: Dense marginal updates require O(N^2) operations, while Gaussian filtering reduces simultaneous marginal updates to O(N) under the kernel-mixture restriction.
- Learning: Learning uses a surrogate log-likelihood loss, computes marginal gradients through efficient back-tracking, and back-propagates through the scoring-function structure.
- Learning: Unlike prior settings, unary terms need not be logistic regressors or remain fixed during CRF parameter updates.
- Learning: The resulting algorithm performs a forward pass, mean-field filtering, loss-gradient back-tracking, chain-rule back-propagation, and parameter updates.
4 Experiments
The approach jointly trains a convolutional network and dense CRF on PASCAL VOC 2012, combining upsampled class maps with mean-field inference. It achieves 64.060% peak validation performance, slightly above a separately trained baseline.
- Dataset and evaluation: The experiments use 21-class semantic segmentation on PASCAL VOC 2012, with performance measured by intersection-over-union on an unused validation set.Additional annotations yield 10582 training instances; the validation set contains 1449 images.
- Model and training: The model uses a 16-layer DeepNet converted to convolutions, producing 40×40×21 class maps that are bilinearly upsampled to image resolution.The network input is 306 × 306, and the intermediate maps provide class probabilities for subsequent CRF processing.
- Model and training: Five mean-field iterations compute fully connected CRF marginals, which are compared with ground truth using a sum of cross-entropy terms.Gradients are back-propagated through the marginals and unary terms, while CRF shape and compatibility parameters are updated directly.
- Results: 61.476% mean intersection over union is reached by the unary model, outperforming the reported 59.80% unary baseline by 1.5%.This result corresponds to fine-tuning the 16-layer DeepNet on Pascal data before adding pairwise connections.
- Results: The results peak after roughly 20 joint-training iterations, while bike and chair remain particularly difficult classes.For both classes, validation performance is roughly half the training accuracy; visual failures also occur on noisy or highly variable objects.
- Results: 64.060% peak validation performance slightly exceeds the 63.74% separate-training result reported by Chen et al.Joint training combines convolutional-network parameters with dense-CRF compatibility and shape parameters.
5 Discussion
The paper presents joint training of convolutional networks and fully connected CRFs for semantic segmentation, extending earlier structured-prediction approaches. It situates the method among prior work and notes an independent method with a different network architecture.
- Contribution: The method jointly trains convolutional-network parameters and fully connected conditional-random-field parameters for semantic image segmentation.The approach generalizes earlier work to joint training and modifies the 16-layer DeepNet architecture.
- Relation to prior work: Earlier approaches incorporated nonlinear unary potentials but generally assumed tractable exact inference, while other work kept pairwise potentials fixed.The paper frames joint training as part of a longer line of structured-prediction research.
6 Conclusion
The paper concludes that jointly training convolutional neural networks and conditional random fields yields good results for semantic image segmentation on PASCAL VOC 2012. Future work will assess larger datasets and weakly labeled data.
- Conclusion: The proposed method combines deep convolutional neural networks with variational mean-field approximations to train convolutional networks and conditional random fields jointly.The reported evaluation is on the challenging Pascal VOC 2012 dataset.
- Future work: Future work will train the method on larger datasets and investigate weakly labeled data.These directions define the stated scope of the paper’s planned extensions.