PaperlayerView exact revision on arXiv

Source-linked AI summary

Higher Order Conditional Random Fields in Deep Neural Networks

Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, Philip Torr

arXiv:1511.08119v4cs.CV

TL;DR

Semantic segmentation needs both accurate pixel classification and structured outputs, but deep-network CRFs had generally omitted higher order potentials. The paper embeds detection- and superpixel-based higher order potentials using differentiable mean field inference, enabling end-to-end training and state-of-the-art segmentation performance. Its mean field procedure empirically converges in most cases, but parallel updates lack general convergence guarantees.

  • Problem

    Deep-learning CRFs for semantic segmentation had largely used only unary and pairwise potentials, although higher order potentials had improved segmentation in earlier work.

  • Method

    The paper adds object-detection and superpixel higher order potentials to a CRF embedded in a CNN, with differentiable mean field inference for joint end-to-end learning.

  • Results

    3.2%: the best method outperforms the baseline on the PASCAL VOC 2012 test set, a 12.6% reduction in error rate and a new state-of-the-art result.

  • Takeaways & Limitations

    Higher order detection and superpixel potentials can coexist with unary and pairwise terms in a trainable deep CRF and substantially improve semantic segmentation accuracy.

  • Takeaways & Limitations

    Parallel mean field updates have no general convergence guarantees, although the authors usually observed convergence and good final segmentations empirically.

Abstract

from arXiv · show

We address the problem of semantic segmentation using deep learning. Most segmentation systems include a Conditional Random Field (CRF) to produce a structured output that is consistent with the image's visual features. Recent deep learning approaches have incorporated CRFs into Convolutional Neural Networks (CNNs), with some even training the CRF end-to-end with the rest of the network. However, these approaches have not employed higher order potentials, which have previously been shown to significantly improve segmentation performance. In this paper, we demonstrate that two types of higher order potential, based on object detections and superpixels, can be included in a CRF embedded within a deep network. We design these higher order potentials to allow inference with the differentiable mean field algorithm. As a result, all the parameters of our richer CRF model can be learned end-to-end with our pixelwise CNN classifier. We achieve state-of-the-art segmentation performance on the PASCAL VOC benchmark with these trainable higher order potentials.

1 Introduction

Semantic segmentation combines strong CNN pixel classifiers with structured CRF outputs, but embedded deep-learning CRFs had largely remained limited to unary and pairwise potentials. This paper adds trainable object-detection and superpixel higher order potentials using differentiable mean field inference and reports state-of-the-art benchmark performance.

  • Semantic segmentation assigns an object-class label to every pixel while encouraging boundaries and similarly appearing regions to remain visually consistent.
  • CNNs provide learned pixel-level classifiers, while CRFs impose structured label consistency in segmentation systems.
  • The higher order potentials improve over the pairwise-only baseline and achieve significant improvements on PASCAL VOC 2012 and PASCAL Context, including state-of-the-art results.
  • Existing CRFs embedded in deep networks generally used only unary and pairwise potentials, despite prior benefits from higher order cliques.
  • The paper introduces object-detection and superpixel potentials that can be represented with mean field inference as CNN layers and trained jointly with the classifier.

2 Related Work

Earlier segmentation systems used CRFs to impose structure, while later deep networks learned stronger features and embedded CRFs. This work makes detection- and superpixel-based higher order potentials compatible with differentiable mean field inference and end-to-end learning.

  • Pre-deep-learning segmentation used hand-crafted features and CRF post-processing to reduce noisy predictions and enforce contextual label consistency.
  • Conventional CRFs evolved from local unary-pairwise models toward dense pairwise, higher order, co-occurrence, and object-detector-based formulations.
  • Deep learning replaced hand-crafted features with learned representations, and subsequent methods incorporated CRFs into CNN-based segmentation pipelines.
  • The detection potential is formulated for mean field inference, preserving dense pairwise connections and differentiability for end-to-end parameter learning.
  • The authors position their work as the first thorough investigation of detection- and superpixel-based higher order potentials learned end-to-end in a deep CRF network.

3 Conditional Random Fields

The paper models pixel labels with a CRF whose energy sums clique costs conditioned on the image. MAP inference is exact in principle, but dense pairwise interactions require approximate mean field inference that can be expressed within deep networks.

  • Semantic segmentation represents each image pixel with a random variable assigned a label from a predefined label set.
  • A CRF defines the conditional distribution of pixel labels through an energy function and its partition-function normalization.
  • The energy is the sum of clique cost functions over variable subsets, using image information and prior knowledge about good segmentations.
  • Minimizing the energy yields the MAP labeling, but dense pairwise potentials make exact inference impracticable and motivate approximate mean field inference.
  • Mean field inference is attractive for deep learning because its updates can be formulated as recurrent neural network operations.

4 CRF with Higher Order Potentials

The richer CRF adds object-detection and superpixel higher order energies to standard unary and pairwise terms. Detection potentials use latent validity variables to tolerate false detections, while superpixel terms encourage regional consistency without hard constraints.

  • CRF with Higher Order Potentials: The complete model combines usual unary and densely connected pairwise energies with newly introduced detection and superpixel higher order energies.
  • Object Detection Based Potentials: Recognition errors from poor unary classification motivate object detectors as additional cues because they can recognize and localize objects.
  • Object Detection Based Potentials: Detection potentials softly encourage detected foreground pixels to take the detection label while allowing other CRF energies to invalidate false detections.
  • Object Detection Based Potentials: Latent binary detection variables indicate whether detections remain valid after inference, allowing false positives to be rejected probabilistically.
  • Superpixel Based Potentials: Superpixel potentials encourage pixels within oversegmented regions to share labels, while learnable P^n-Potts costs avoid enforcing consistency as a hard constraint.
  • Superpixel Based Potentials: Superpixel higher order energy removes small spurious regions of incorrect labels, and detection inference can ignore erroneous masks when other energy terms disagree.

5 Mean Field Updates and Their Differentials

This section describes differentiable mean field updates for detection- and superpixel-based higher order CRF potentials, enabling end-to-end learning of the embedded CRF. It also notes important optimization and convergence caveats.

  • Differentials and Training: Differentiable mean field updates allow the higher order CRF parameters and earlier network components to be trained end-to-end.The updates are differentiable with respect to the distribution inputs and higher order potential parameters.
  • Mean Field Inference: Mean field inference approximates the CRF joint distribution with a product of marginal distributions and iteratively updates each variable's marginal.The update uses clique costs, assignments to the remaining clique variables, and a normalization constant.
  • Detection-Based Potentials: Detection-based potentials contribute updates to both pixel-label distributions Q(X(i)) and latent detection-variable distributions Q(Yd = b).The detection contribution is derived from the higher order detection energy and assignments with one pixel removed.
  • Detection-Based Potentials: Class-specific detection weights increase flexibility by allowing detector evidence to matter differently across object classes.The assumption is that detector outputs help some classes more than others, especially when detector performance or foreground segmentation varies by class.
  • Superpixel-Based Potentials: Superpixel-based potentials encourage label consistency within superpixel cliques, and their mean field updates are differentiable with respect to both potential parameters and incoming marginals.This permits learning the superpixel weights and optimizing preceding network layers through backpropagation.
  • Convergence: Parallel mean field updates lack general convergence guarantees, although the authors usually observed convergence and good final segmentations empirically.Good initialization from the pixelwise network may explain the observed behavior; nonconvergent cases still produced empirically good segmentations.

6 Experiments

Experiments evaluate the higher-order CRF on VOC 2012 and PASCAL Context, including ablations of detection, superpixel, and training strategies. The combined model improves interior accuracy and achieves state-of-the-art benchmark results, while superpixel effects near boundaries are mixed.

  • Experimental set-up and results: The model is evaluated on VOC 2012 and PASCAL Context, using CRF-RNN as the main baseline and additional ablation studies.The experiments also examine error patterns, end-to-end versus piecewise training, and compatibility with different unary CNN architectures.
  • Experimental set-up and results: 3.2% higher mean IoU than the baseline on the VOC test set corresponds to a 12.6% reduction in error rate and establishes a new state-of-the-art result.The higher-order potentials were evaluated against the baseline on the reduced validation set, while test-set results were reported only for the best method.
  • Experimental set-up and results: 0.8% improvement over the previous state of the art and 2% over the baseline is reported on the PASCAL Context validation set.The dataset contains 59 classes, while the object detectors cover only 20 labels.
  • Ablation studies: Detection potentials improve both Interior IoU and Boundary IoU across tested trimap widths relative to pairwise-only inference.The paper attributes the boundary improvement to stronger pairwise influences from correctly detected interior pixels.
  • Ablation studies: Superpixel potentials improve Interior IoU by removing spurious regions, but their Boundary IoU is slightly lower than the baseline.The authors suggest that imperfect alignment between superpixels and object edges may explain the boundary decrease.
  • Ablation studies: Combining detection and superpixel potentials substantially improves Interior IoU, which is identified as the primary reason overall VOC IoU increases.End-to-end CRF training also yields considerable improvements over piecewise training with either FCN-8s or DCN unaries.

7 Conclusion

The paper presents a CRF with detection- and superpixel-based higher order potentials for semantic segmentation. Differentiable mean field inference makes the model trainable end-to-end with a pixelwise CNN, improving accuracy to state-of-the-art performance.

  • The CRF uses object-detection and superpixel-based higher order potentials alongside standard unary and pairwise potentials.Detection cues can support segmentation, while superpixel potentials encourage consistent regional labels.
  • The detection potential can automatically reject false object detections that disagree with the semantic segmentation.
  • Efficient mean field inference remains possible with the new potentials, including explicit updates and differentials.
  • The CRF becomes a stack of CNN layers that can be trained end-to-end with a pixelwise CNN classifier.
  • Adding higher order potentials significantly increases semantic segmentation accuracy and reaches state-of-the-art performance.

Appendix

The appendix documents derivative calculations for mean field inference and provides additional qualitative experimental results.

  • Appendix A presents derivatives of the mean field updates used for CRF inference.Appendix B provides detailed qualitative results for the main-paper experiments.

A Derivatives of Mean Field Updates

The appendix describes mean field inference updates and their derivatives for the higher order CRF. It frames backpropagation through recurrent mean field iterations using standard chain-rule calculations.

  • Algorithm 1 gives pseudocode for mean field inference with latent detection variables.
  • Each iteration combines unary, detection, and superpixel updates for the CRF variables.The displayed algorithm includes DetectionUpdate and SuperpixelUpdate terms alongside UnaryUpdate.
  • The derivation assumes one object detection and one superpixel clique because multiple instances are handled by summing their updates and differentials.
  • DetectionUpdate and SuperpixelUpdate apply when a pixel belongs to the corresponding detection foreground or superpixel clique.
  • Backpropagation starts from the loss gradient at mean field output and computes derivatives with respect to CNN unaries and CRF parameters.Differentials for multiple iterations follow from the recurrent behavior of the mean field iterations.
  • The required differentials can be calculated using the chain rule, analogous to backpropagation through a softmax operation.
  • The effect of superpixel potentials on derivatives of the output beliefs was negligible and was ignored in the calculations.

B Additional Experimental Results

Additional experiments report per-class test-set accuracy, qualitative comparisons, ablations, and failure cases. The method improves over CRF-as-RNN across all 20 VOC classes, while its potentials contribute differently across examples.

  • Per-class accuracy improves over the CRF-RNN baseline for all 20 PASCAL VOC classes.
  • Qualitative figures compare the method with CRF-as-RNN, show failure cases, examine individual potentials, and compare against other current methods.
  • Table 5 compares mean IoU accuracy between the proposed approach and other state-of-the-art methods on the PASCAL VOC 2012 test set.
  • Examples of improvements over CRF-as-RNN use detection bounding boxes and attribute gains to both detection and superpixel potentials.These examples come from the reduced VOC 2012 validation set and were not used for training.
  • Failure cases include sofa misclassification from person detections and removal of correct labels when superpixel potentials suppress apparent noise.
  • Superpixel potentials can remove spurious output regions, while detection potentials can substantially improve recognition of objects such as trains; combining all potentials can improve results further.
  • The method is qualitatively compared with Deeplab, FCN-8s, and CRF-as-RNN on VOC 2012 test examples.
Loading 1511.08119v4…