PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Markov Random Field for Semantic Segmentation

Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, Xiaoou Tang

arXiv:1606.07230v2cs.CVcs.LG

TL;DR

Semantic segmentation requires models that capture contextual dependencies without the optimization and inference costs of directly combining CNNs with complex MRFs. The paper proposes DPN, a CNN that jointly models unary and rich pairwise terms while approximating mean-field inference in one forward pass, and reports state-of-the-art performance on three standard benchmarks.

  • Problem

    Directly combining CNNs with MRFs is inefficient for jointly optimizing millions of CNN parameters, latent variables, and complex pairwise terms.

  • Method

    DPN jointly learns CNN-based unary terms and high-order, mixed-context pairwise terms, reformulating mean-field updates as deterministic convolutional operations.

  • Results

    DPN achieves state-of-the-art performance on VOC12, Cityscapes, and CamVid semantic segmentation benchmarks.

  • Takeaways & Limitations

    DPN provides a unified, parallelizable framework for incorporating rich contextual information into semantic image and video segmentation.

  • Takeaways & Limitations

    Performance remains sensitive to atypical object poses, scale, and illumination, with future work targeting larger class counts and greater appearance or scale variation.

Abstract

from arXiv · show

Semantic segmentation tasks can be well modeled by Markov Random Field (MRF). This paper addresses semantic segmentation by incorporating high-order relations and mixture of label contexts into MRF. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic end-to-end computation in a single forward pass. Specifically, DPN extends a contemporary CNN to model unary terms and additional layers are devised to approximate the mean field (MF) algorithm for pairwise terms. It has several appealing properties. First, different from the recent works that required many iterations of MF during back-propagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing models as its special cases. Furthermore, pairwise terms in DPN provide a unified framework to encode rich contextual information in high-dimensional data, such as images and videos. Third, DPN makes MF easier to be parallelized and speeded up, thus enabling efficient inference. DPN is thoroughly evaluated on standard semantic image/video segmentation benchmarks, where a single DPN model yields state-of-the-art segmentation accuracies on PASCAL VOC 2012, Cityscapes dataset and CamVid dataset.

1 INTRODUCTION

Semantic segmentation assigns category labels to pixels, and MRFs capture contextual dependencies through unary and pairwise terms. DPN addresses inefficient CNN–MRF integration by jointly learning these terms in a deterministic CNN that approximates mean-field inference with one iteration.

  • Semantic segmentation assigns each pixel a category label in images or video sequences, supporting applications such as scene understanding and automated driving.
  • MRFs model joint label distributions using unary terms for per-pixel confidence and pairwise terms for inter-pixel constraints.
  • Existing approaches combine expressive pairwise models with relatively shallow unary models, while CNN-based methods often use simple pairwise functions or separate CNN and MRF components.
  • DPN approximates mean-field inference with one iteration, replacing recurrent or iterative computation with deterministic end-to-end CNN computation and reducing computational cost.The approximation uses convolutional and pooling operations that can be parallelized on GPUs and accelerated through low-rank approximation.
  • DPN supports multiple pairwise-term types and represents several previous models as special cases while achieving state-of-the-art results on PASCAL VOC 2012, CityScapes, and CamVid.The paper presents DPN as a general framework for jointly modeling unary and pairwise terms.
  • DPN jointly trains a VGG16-based unary model with rich pairwise information, including mixtures of label contexts and high-order relations.

2 RELATED WORK

Prior semantic segmentation research developed either graph structures for contextual dependencies or deep architectures for pixel classification. Related work spans MRF/CRF models, CNN unary classifiers, and emerging attempts to incorporate temporal information for video segmentation.

  • Existing studies primarily pursue richer graph structures for contextual information and long-term dependencies or network architectures that exploit deep learning.
  • MRF and CRF methods improve image segmentation by modeling long-range dependencies, high-order potentials, and semantic label contexts.
  • Video segmentation extends MRF or CRF graphs into the spatio-temporal domain using mechanisms such as tracking, occlusion reasoning, optical-flow trajectories, and object potentials.
  • CNNs became strong unary classifiers for semantic segmentation, often producing encouraging results with simple or omitted pairwise functions.
  • Few unified deep-learning frameworks addressed semantic video segmentation; SegNet used an encoder–decoder architecture but did not model temporal relationships.The paper positions its extension of DPN as incorporating temporal voxels into joint learning and inference with rich spatio-temporal information.

3 OUR APPROACH

DPN is a unified CNN framework that models unary terms and rich high-order pairwise relations in MRFs, then approximates mean-field inference through convolutional layers. It supports dynamic node linking across spatial-temporal dimensions and can represent multiple contextual and smoothness structures.

  • DPN framework: DPN extends VGG16 for unary modeling and adds layers that model pairwise terms in high-order MRFs across 2-D, 3-D, or N-D data.The framework can be adapted by removing or adding relationships between nodes.
  • Dynamic node linking: Dynamic node linking defines temporal neighbors along optical-flow trajectories rather than a fixed rectangular grid, preserving spatio-temporal context under large motion.The spatial domain retains a 2-D structure, while temporal edges connect voxels on the same estimated trajectory.
  • Pairwise modeling: The baseline pairwise formulation encourages compatible labels for nearby, visually similar voxels but omits spatial mixtures of label contexts and high-order interactions.Examples include different person-table configurations and the inability to model interactions beyond pixel pairs.
  • Pairwise modeling: DPN enriches smoothness modeling with mixtures of local label contexts, triple penalties, and learned position-dependent costs over neighboring voxels.The mixture selects among K local-context components, while the triple term involves a voxel, a neighbor, and that neighbor’s surrounding voxels.
  • Mean-field inference: DPN approximates one mean-field iteration by filtering predicted label maps for distance-weighted smoothing and then convolving them with learned labeling-context filters.The initial probabilities come from VGG16 unary predictions, and the resulting update incorporates neighboring smoothness penalties.

4 DEEP PARSING NETWORK

DPN extends VGG16 with layers that model unary terms and approximate one mean-field iteration for complex pairwise terms. Its convolutional formulation supports diverse pairwise structures and parallel message passing.

  • Architecture: DPN extends VGG16 to model unary terms and adds layers that approximate one iteration of mean-field inference for pairwise terms.The added layers transform mean-field computation into deterministic feed-forward operations.
  • Pairwise terms: The b12 layer uses position-specific 3D local filters shared across 21 label channels to capture distance-based relations between nearby pixels.Each b12 filter represents a distance metric for a specific spatial region.
  • Pairwise terms: The b13 layer learns category-specific 3D filters for local label contexts, while b14 selects the contextual pattern with the smallest penalty.With K = 5 and n = 9, b13 generates 105 feature maps and b14 reduces them to 21 output channels.
  • Inference: The b15 layer combines unary and smoothness outputs by element-wise summation, producing the final labeling scores before label normalization.This combines the outputs of b11 and b14 in the manner of the mean-field update.
  • Model flexibility: DPN represents multiple pairwise-term types, making several previous CNN-MRF models special cases while allowing complexity changes through convolutional receptive fields.Changing the range of pixels or contexts does not require changing the back-propagation procedure.
  • Efficiency: DPN transforms mean-field operations into convolutions and poolings, enabling parallel filtering across training images and easier speedup.For ten 512×512 images, a fast Gaussian-filtering implementation requires 3.5×10^11 operations and 114 ms for one iteration, whereas DPN enables image-level parallelization.

5 EXPERIMENTS

The experiments evaluate DPN and spatial-temporal DPN on established image and video segmentation benchmarks using several complementary metrics. The study compares them with state-of-the-art CNN- and MRF-based methods and examines component effectiveness.

  • Datasets: Experiments evaluate 2-D DPN on PASCAL VOC 2012 and Cityscapes, and spatial-temporal DPN on the CamVid video dataset.The datasets cover generic image segmentation, urban street-scene parsing, and video sequences.
  • Evaluation metrics: Performance is assessed with mean pixelwise intersection-over-union, tagging accuracy, localization accuracy, and boundary accuracy.These metrics examine segmentation overlap, image-level tags, object localization, and boundaries.
  • Comparisons: DPN is benchmarked against CNN- and MRF-based methods, including jointly trained and separately trained CNN-MRF systems.Compared methods include FCN, DeepLab, WSSL, BoxSup, Piecewise, RNN, SegNet, and others.
  • Study design: The experimental program includes component ablations, spatial-temporal evaluation, system analysis, and comparisons with state-of-the-art methods.These analyses are organized across Sections 5.1 through 5.4.

5.1 Effectiveness of DPN

Ablations show that DPN benefits from an appropriately sized pixel neighborhood and mixtures of local label contexts. The reported improvements support richer receptive fields and contextual modeling over simpler pairwise configurations.

  • Triple Penalty: A 50×50 receptive field achieves the best mIoU among 10×10, 50×50, and 100×100 settings, while all triple-relation models outperform VGG16+denseCRF.The authors interpret smaller and larger regions as tending to under-fit or over-fit the training data.
  • Label Contexts: ‘9×9 mixtures’ improve preceding label-context settings by 1.7, 0.5, and 0.2 percent, respectively.A 1×1 receptive field represents global label co-occurrence without local spatial context.
  • Label Contexts: DPN pairwise terms are more effective than those of DSN and DeepLab4 in the reported comparison.The comparison appears in Table 3(c).
  • Limitations: The reported ablation evidence does not include exact improvements after combining unary and pairwise terms for RNN and Piecewise.Those methods did not report the exact combined improvements.
  • Label Contexts: Across categories, the last three label-context settings improve mIoU over the first by 1.2±0.2, 1.5±0.2, and 1.7±0.3, respectively.The study also visualizes learned label compatibilities and contextual patterns, including asymmetric object co-occurrences and spatial relations.

5.2 Effectiveness of Spatial-temporal DPN

Spatial-temporal DPN uses 3D pairwise terms to capture relationships between successive video frames. These terms slightly outperform 2D pairwise terms and encode interpretable cross-frame label contexts.

  • Temporal Regularization: CamVid’s moving-vehicle sequences provide temporal regularization for evaluating spatial-temporal DPN.
  • 3D Convolution: 3D pairwise terms capture information between successive frames and perform slightly better than 2D pairwise terms.The evaluated 3D receptive fields are 50×50×3 for b12 and 7×7×3 for b13.
  • Visualization: The visualizations illustrate how learned spatial-temporal pairwise terms represent moving semantic regions across neighboring CamVid frames.
  • Temporal Regularization: Cross-frame label compatibilities encode directional relationships, such as pavement favoring nearby road in the next frame while sky penalizes that association.
  • Temporal Regularization: Spatial-temporal contextual patterns model both same-frame neighborhood consistency and positional shifts across adjacent frames.For example, sky in the previous frame is associated with sky appearing above the corresponding position in the current frame.

5.3 Further Analysis

Further analysis shows that DPN benefits from incremental training, reaches strong accuracy with one mean-field iteration, and improves different segmentation properties at different stages. The metrics also expose category-specific weaknesses involving small objects, localization, and boundaries.

  • Per-stage Analysis: The four stage-wise mIoU rows are accompanied by tagging, localization, and boundary accuracy metrics for diagnosing different performance factors.
  • Incremental Learning: Each incremental DPN stage improves performance, with triple penalty adding 2.3 percent, label contexts adding 1.8 percent, and joint fine-tuning adding 1.3 percent.
  • Incremental Learning: Incremental learning produces higher and more stable accuracies than joint learning, which may become stuck at local minima.
  • One-iteration MF: DPN reaches good accuracy with one MF iteration, whereas dense-pairwise CRF needs more than 5 iterations and existing deep models require 5∼10 iterations to converge.
  • Per-class Analysis: Joint training improves most categories but can discard extremely small animals, while object localization helps and complex boundaries can still depress mIoU.Bike illustrates the latter pattern through high localization accuracy but low boundary accuracy and mIoU.

5.4 Benchmarks

DPN performs strongly across VOC12, Cityscapes, and CamVid, including competitive image segmentation and improved video segmentation with spatial-temporal pairwise terms. Its reported results also highlight benefits for arbitrary-shaped, narrow, and small objects.

  • VOC12: 74.1% accuracy is achieved on VOC12 without outside training data by averaging two DPNs, with at least 10× lower runtime than previous works.DPN uses one MF iteration, compared with 10 iterations for RNN, DeepLab, and Piecewise in the cited comparison.
  • VOC12: 77.5% mIoU is achieved by a single COCO-pretrained DPN† on the VOC12 test set.
  • Cityscapes: 66.8% is achieved on Cityscapes, placing DPN second and close to the first-place result of 67.1%.
  • Object-level Results: DPN is especially effective on Cityscapes objects with arbitrary shapes and CamVid’s narrow or small objects such as poles and signs.The paper attributes these gains to high-order, long-range, and spatial-temporal pairwise terms.
  • CamVid: 60.25% performance is achieved on CamVid after adding the spatial-temporal pairwise term, outperforming all existing methods in the comparison.

5.5 Visual Quality Comparisons

Visual comparisons show that DPN generally produces more accurate image-level and instance-level predictions than the compared models. The paper also documents failures linked to atypical pose, scale, and illumination.

  • Visual Comparisons: DPN generally makes more accurate image-level and instance-level predictions than FCN and DeepLab in the visual comparisons.The examples include discovering all persons while recovering sharp aeroplane boundaries.
  • Failure Modes: Atypical object poses can make objects difficult to detect in DPN’s visual failure cases.
  • Failure Modes: Scale and illumination also influence performance, with data augmentation or increased training-data variance identified as partial mitigations.

6 CONCLUSION

DPN unifies unary and pairwise inference and learning in one convolutional network while incorporating richer label relations. It also supports efficient parallel computation and achieves state-of-the-art performance across three segmentation benchmarks.

  • DPN unifies unary-term and pairwise-term inference and learning within a single convolutional network, eliminating iterative inference during back-propagation.
  • DPN incorporates high-order relations and mixtures of label contexts into pairwise modeling, with existing approaches represented as special cases.
  • DPN uses conventional CNN operations, making its computation easier to parallelize and speed up.
  • DPN achieves state-of-the-art performance on VOC12, Cityscapes, and CamVid datasets.The experiments also examine interactions among tagging, localization, and boundary accuracies across processing stages.
Loading 1606.07230v2…