PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Image Segmentation via Deep Parsing Network

Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, Xiaoou Tang

arXiv:1509.02634v2cs.CV

TL;DR

Semantic segmentation needs both strong per-pixel predictions and rich spatial relations, but existing MRF approaches face limited unary capacity or expensive inference. The paper introduces DPN, which jointly learns CNN unary terms and complex pairwise terms while approximating mean-field inference in one feed-forward pass. DPN achieves state-of-the-art performance on VOC12, while its evaluation is scoped to VOC12 and future generalizability remains open.

  • Problem

    Existing segmentation formulations either use limited pairwise relations or insufficiently powerful unary models, while rich pairwise inference can be expensive.

  • Method

    DPN extends VGG16 for unary terms and adds convolutional layers that jointly learn rich pairwise terms, including local label-context mixtures and high-order relations.

  • Results

    77.5% accuracy on the PASCAL VOC 2012 test set is reported for a single DPN model.

  • Takeaways & Limitations

    DPN unifies unary and pairwise learning and inference in one convolutional network, requiring no iterative inference during back-propagation.

  • Takeaways & Limitations

    The experiments are conducted on VOC12, and the authors identify generalizability to larger class counts and stronger appearance or scale variation as future work.

Abstract

from arXiv · show

This paper addresses semantic image segmentation by incorporating rich information into Markov Random Field (MRF), including high-order relations and mixture of label contexts. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic end-to-end computation in a single forward pass. Specifically, DPN extends a contemporary CNN architecture to model unary terms and additional layers are carefully devised to approximate the mean field algorithm (MF) for pairwise terms. It has several appealing properties. First, different from the recent works that combined CNN and MRF, where many iterations of MF were required for each training image during back-propagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing works as its special cases. Third, DPN makes MF easier to be parallelized and speeded up in Graphical Processing Unit (GPU). DPN is thoroughly evaluated on the PASCAL VOC 2012 dataset, where a single DPN model yields a new state-of-the-art segmentation accuracy.

1. Introduction

DPN combines CNN-based unary modeling with rich MRF pairwise terms for semantic segmentation. It approximates mean-field inference in one feed-forward computation while retaining high performance.

  • Motivation: Existing MRF segmentation methods either enrich pairwise terms or strengthen unary classifiers with deep CNNs.The former use long-range, high-order, or semantic-context relations; the latter often use simpler pairwise functions or omit them.
  • Motivation: Earlier rich-pairwise approaches used weaker unary models and faced expensive learning and inference for complex pairwise terms.Their unary terms were modeled with SVM or Adaboost, limiting learning capacity.
  • DPN: DPN jointly trains CNN unary terms and complex pairwise terms through deterministic end-to-end computation.It extends VGG16 for unary modeling and designs additional layers for pairwise modeling.
  • DPN: DPN approximates mean-field inference with one iteration, reducing computational cost while maintaining high performance.The approach avoids recurrently computing many mean-field forward passes during back-propagation.
  • Results: A single DPN model achieves 77.5% accuracy on the PASCAL VOC 2012 test set.The paper reports this as a new state-of-the-art segmentation accuracy.

2. Our Approach

The approach enriches MRF smoothness terms with local label-context mixtures and high-order relations. DPN implements one mean-field update through convolutional operations over CNN predictions.

  • MRF formulation: An MRF represents pixels as graph nodes and pixel relations as edges, with unary and pairwise terms defining its energy.Unary terms measure per-pixel label costs, while pairwise terms impose smoothness constraints between labels.
  • Limitations of standard smoothness: The standard pairwise term captures label co-occurrence and pixel distance but misses spatial object context and high-order interactions.For example, it cannot distinguish different relative configurations such as an object appearing beside versus below another.
  • Rich pairwise terms: DPN replaces this limited smoothness modeling with a mixture of local label contexts that penalizes assignments in local regions.Each mixture component selects a local context pattern, with K components and binary activation indicators λ_k.
  • Rich pairwise terms: The pairwise design also models high-order relations involving a center pixel, a neighboring pixel, and that neighbor’s surrounding pixels.The learned compatibility depends on relative positions and can impose penalties on particular spatial configurations.
  • Mean-field inference: Mean-field inference estimates a factorized label distribution by repeatedly updating each pixel’s predicted label probabilities from neighboring probabilities and smoothness penalties.The unary predictions from VGG16 initialize the probabilities, while the updates enforce smoothness constraints.
  • DPN implementation: DPN approximates one mean-field iteration with two convolutions: neighborhood distance smoothing followed by local label-context filtering.The first uses an m × m filter to aggregate nearby probabilities; the second uses an n × n filter encoding label-context penalties.

3. Deep Parsing Network

DPN extends VGG16 to model unary terms and adds layers that approximate one mean field iteration for rich pairwise terms. Its architecture converts MRF inference into parallel convolutions and pooling while preserving high-resolution labeling information.

  • Architecture: DPN extends VGG16 to produce unary labeling results and adds layers that approximate one iteration of mean field inference for pairwise terms.The unary output consists of 21 probabilistic 512×512 label maps for VOC12.
  • Modeling Unary Terms: Removing selected max-pooling layers keeps DPN feature maps at a minimum size of 64×64 instead of reducing them to 7×7.The removed layers are a8 and a10; feature maps in b11 are later up-sampled to 512×512 by bilinear interpolation.
  • Modeling Unary Terms: The unary pathway transforms VGG16 fully connected layers into convolutional layers, using padded filters to preserve the intended receptive fields.The first transformed layer uses 4096 filters of size 25×25×512, and the second uses 4096 filters of size 1×1×4096.
  • Modeling Smoothness Terms: DPN models smoothness through locally convolutional, convolutional, block-min-pooling, and summation layers that encode local label contexts and combine unary and smoothness terms.b12 uses position-specific filters, b13 learns category-specific penalties over local label maps, b14 selects the contextual pattern with the smallest penalty, and b15 sums outputs.
  • Modeling Smoothness Terms: The b12 filters are initialized with Euclidean distances because VOC12 regions contain heterogeneous object-shape patterns, although the formulation can learn filters for homogeneous patterns.The broader formulation is described as more general than the triple penalty and may learn from data when regional patterns are regular.

4. Experiments

Experiments on VOC12 evaluate DPN’s components, training strategy, MF iteration efficiency, and performance against existing CNN-MRF methods. DPN benefits from multi-scale triple relations, mixture-based label contexts, incremental learning, and achieves strong VOC12 test accuracy.

  • Ablation studies: 50×50 triple-relation neighborhoods achieve the best mIoU, while all triple-relation models outperform the denseCRF baseline.Smaller or larger receptive fields tend to under-fit or over-fit the training data.
  • Ablation studies: 9×9 label-context mixtures improve preceding settings by 1.7, 0.5, and 0.2 percent, while DPN pairwise terms outperform DSN and DeepLab4.Increasing receptive-field size and learning mixtures improves mIoU across all categories.
  • Training strategy: Incremental learning produces higher and more stable accuracies than joint learning, which may become stuck at local minima.Each successive DPN stage improves performance, and joint fine-tuning adds another 1.3 percent gain.
  • MF iterations: One MF iteration is sufficient for DPN to reach good accuracy, whereas dense-CRF and existing deep models require more iterations to converge.The cited comparisons report more than 5 iterations for dense CRF and 5–10 iterations for existing deep models.
  • Overall performance: DPN outperforms existing VOC12-trained methods and reaches 74.1% accuracy when averaging two DPNs without outside training data.The paper reports at least a 10× runtime reduction relative to methods using 10 MF iterations.
  • Overall performance: A single COCO-pretrained DPN achieves 77.5% mIoU on the VOC12 test set and performs best on more than half of the object classes.The COCO pretraining uses the 20 categories shared by COCO and VOC12.

5. Conclusion

DPN unifies unary and pairwise modeling within a single convolutional network, incorporating high-order relations and mixtures of label contexts. It achieves state-of-the-art performance on VOC12 while remaining amenable to GPU parallelization and speedup.

  • DPN unifies inference and learning of unary and pairwise terms in a single convolutional network without iterative inference during back-propagation.
  • High-order relations and mixtures of label contexts enrich DPN’s pairwise-term modeling, with existing methods represented as special cases.
  • Conventional CNN operations make DPN easier to parallelize and speed up.
  • DPN achieves state-of-the-art performance on VOC12, while future work targets larger class counts and stronger appearance or scale variation.

A. Fast Implementation of Locally Convolution

DPN accelerates locally convolutional filtering by using a lookup table for distances between pixel intensities. The table stores all pairwise distances among 8-bit intensity values.

  • DPN’s locally convolutional layer b12 computes local filters from distances between pixels’ RGB values.XY coordinates are omitted because they can be precomputed.
  • A 256 × 256 lookup table stores distances between every pair of pixel intensities from 0 to 255.The lookup-table filtering approach accelerates locally convolutional computation.

B. Visual Quality Comparisons

Visual comparisons evaluate DPN against FCN and DeepLab using input images, ground truth, and predicted label maps. DPN generally produces more accurate image-level and instance-level predictions, while its components help resolve confusing objects and boundaries.

  • DPN generally makes more accurate predictions than FCN and DeepLab at both image and instance levels.The comparison uses a publicly released FCN model and DeepLab results extracted from its published paper.
  • Local label contexts help DPN differentiate confusing objects, while the triple penalty facilitates capturing intrinsic object boundaries.
  • Figure 8 compares visual segmentation quality across input images, ground truth, FCN, DeepLab, and DPN.
  • Figure 9 shows DPN label maps alongside inputs and ground truth, with white labels marking ambiguous regions.
Loading 1509.02634v2…