PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Context with Deep Structured models for Semantic Segmentation

Guosheng Lin, Chunhua Shen, Anton van den Hengel, Ian Reid

arXiv:1603.03183v3cs.CV

TL;DR

The paper addresses limited explicit modeling of patch-patch context in CNN segmentation. It combines CNNs and CRFs to learn contextual pairwise potentials, reporting state-of-the-art results across eight datasets, including 77.8 intersection-over-union on PASCAL VOC 2012.

  • Problem

    Explicit modeling of patch-patch contextual relations had not been well studied in recent CNN-based segmentation, despite context providing cues for scene understanding.

    Source: §1 · p. 2

  • Method

    The method combines CNNs and CRFs, learning CNN-based pairwise potentials for semantic relations between image regions and using piecewise training to avoid repeated inference.

    Source: §1 · p. 2

  • Results

    The method reports state-of-the-art performance on all eight evaluated datasets, including an intersection-over-union score of 77.8 on PASCAL VOC 2012.

    Source: §1 · p. 2, §10 · p. 14

  • Takeaways & Limitations

    The authors suggest the proposed method may be applicable to other tasks.

    Source: §10 · p. 14

    Final high-resolution prediction uses bilinear upsampling and boundary-refinement post-processing, which the authors say could be improved with more sophisticated refinement.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

State-of-the-art semantic image segmentation methods are mostly based on training deep convolutional neural networks (CNNs). In this work, we proffer to improve semantic segmentation with the use of contextual information. In particular, we explore `patch-patch' context and `patch-background' context in deep CNNs. We formulate deep structured models by combining CNNs and Conditional Random Fields (CRFs) for learning the patch-patch context between image regions. Specifically, we formulate CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied in order to avoid repeated expensive CRF inference during the course of back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image inputs and sliding pyramid pooling is very effective for improving performance. We perform comprehensive evaluation of the proposed method. We achieve new state-of-the-art performance on a number of challenging semantic segmentation datasets including $NYUDv2$, $PASCAL$-$VOC2012$, $Cityscapes$, $PASCAL$-$Context$, $SUN$-$RGBD$, $SIFT$-$flow$, and $KITTI$ datasets. Particularly, we report an intersection-over-union score of $77.8$ on the $PASCAL$-$VOC2012$ dataset.

1 INTRODUCTION

The paper explores patch-patch and patch-background context to improve CNN-based semantic segmentation. It combines CNNs and CRFs for contextual modeling, uses piecewise training, and reports state-of-the-art results across challenging datasets.

1 INTRODUCTION

  • Contextual relations provide scene-understanding cues, especially when an object's visual appearance is ambiguous.Source: §1 · p. 2
  • CNN-based pairwise potentials explicitly model patch-patch semantic relations, targeting coarse predictions rather than merely smoothing boundaries.Source: §1 · p. 2
  • Piecewise CRF training avoids repeated inference during backpropagation, addressing the computational expense of learning general pairwise potentials.Source: §1 · p. 2
  • Multi-scale image inputs and sliding pyramid pooling are used to capture patch-background context.Source: §1 · p. 2
  • 77.8 intersection-over-union on PASCAL VOC 2012 accompanies reported state-of-the-art performance across several challenging segmentation datasets.Source: §1 · p. 2

1.1 Related work

  • Unlike methods focused on refining low-resolution predictions, this method improves coarse predictions with semantic pairwise relations; the approaches are complementary.Source: §1.1 · p. 3

2 MODELING SEMANTIC PAIRWISE RELATIONS

The CRF graph represents feature-map positions as nodes and connects spatially nearby nodes through relation-specific pairwise potentials.

2 MODELING SEMANTIC PAIRWISE RELATIONS

  • Each feature-map spatial position becomes a node in the CRF graph.Source: §2 · p. 4
  • Edges connect each node to other nodes inside a spatial range box, with separate potential functions for different spatial relations.Source: §2 · p. 4
  • The experiments use range boxes sized 0.4a × 0.4a, where a is the feature map's short-edge length; other sizes and positions can define more potentials.Source: §2 · p. 4

3 CONTEXTUAL DEEP CRFS

The section formulates a CNN-based CRF with unary and pairwise potentials; pairwise CNN potentials express label compatibility between connected image regions.

3 CONTEXTUAL DEEP CRFS

  • The CRF energy combines unary and pairwise potentials, with image regions associated with nodes and edges.The model uses a feature-generating network and a shallow network to produce potential outputs.Source: §3 · p. 4, Figure 4 · p. 5

3.1 Unary potential functions

  • Unary-Net maps each node’s feature vector to class-specific potential outputs, with one output per class.The unary potential stacks FeatMap-Net with a shallower fully connected Unary-Net.Source: §3.1 · p. 4

3.2 Pairwise potential functions

  • Pairwise-Net produces a compatibility score for each possible label pair, using concatenated features from the two connected nodes.Its K² outputs correspond to the possible label combinations.Source: §3.2 · p. 4
  • Unlike Potts smoothness potentials, CNN pairwise potentials individually model semantic compatibility for each label pair at the coarse prediction stage.The method still applies dense CRF refinement after coarse prediction to produce the final high-resolution prediction.Source: §3.2 · pp. 4–5

3.2.1 Asymmetric pairwise potentials

  • For asymmetric relations such as above/below, node order indicates spatial configuration, and the potential must distinguish input orders.Concatenating node features in order and parameterizing each label pair separately makes the edge response order-dependent.Source: §3.2.1 · p. 5

4 EXPLORING BACKGROUND CONTEXT

The section encodes patch-background context using multi-scale CNN inputs and sliding pyramid pooling on feature maps.

4 EXPLORING BACKGROUND CONTEXT

  • Three resized image scales use shared top five convolution blocks and a scale-specific sixth block before their feature maps are combined.The experimental scales are 1.2, 0.8, and 0.4; smaller feature maps are bilinearly upscaled before concatenation.Source: §4 · p. 6
  • Sliding pyramid pooling captures background information at multiple sizes and increases feature-map field-of-view.The paper states that increasing field-of-view generally helps segmentation performance.Source: §4 · p. 6
  • For each image scale, two-level spatial pyramid pooling uses 5 × 5 and 9 × 9 sliding max-pooling windows; their pooled maps are concatenated with the original feature map, yielding dimension 512 × 3 per scale.Source: §4 · p. 6

5 NETWORK CONFIGURATIONS

The section adapts a VGG-16-based feature network to balance feature-map resolution, field-of-view, and computation.

5 NETWORK CONFIGURATIONS

  • FeatMap-Net follows VGG-16’s top five convolution blocks but transfers only its first fully connected layer, which the paper describes as more efficient.That layer is converted into a convolution layer and merged into the fifth block.Source: §5 · p. 6
  • Reducing only the last max-pooling stride yields feature maps 16 times smaller than the input, or about 30 × 30 for a 500 × 500 image.The paper contrasts this with reducing the stride of two pooling layers, which yields a down-sampling factor of eight.Source: §5 · p. 7
  • Two added 3 × 3 convolution layers enlarge the field-of-view and compensate for the side effect of reducing the pooling stride.These layers form the new Conv Block 6.Source: §5 · pp. 6–7

6 PREDICTION

Prediction first uses CRF inference to produce a coarse score map, then upsamples and refines it to input resolution. The authors also explore middle-layer features as a way to improve refinement.

6.1 Coarse-level prediction stage

  • CRF inference produces a coarse score map at 1/16 input resolution; mean-field inference approximates the non-tree CRF, using three iterations in experiments.The coarse map is bilinearly upsampled before the refinement stage.Source: §6 · p. 7, §6.1 · p. 8

6.2 Prediction refinement stage

  • Bilinear upsampling followed by dense-CRF boundary refinement uses low-level color contrast to produce a prediction at input resolution.The dense CRF sharpens object boundaries after the coarse score map is upsampled.Source: §6 · p. 7, §6.2 · p. 8
  • They bilinearly upsample the coarse score map and apply boundary-refinement post-processing, while suggesting more sophisticated refinement—including middle-layer features for high-resolution prediction—may improve performance; they demonstrate feature-based refinement on PASCAL VOC 2012.Source: §6.2 · p. 8

7 CRF TRAINING

Direct CRF learning is difficult because the global partition function is intractable for the large loopy segmentation graph, and inference at every CNN update is expensive. Piecewise training avoids that repeated global inference by optimizing separate likelihoods for the model's potentials.

7 CRF TRAINING

  • The exponentially large label space makes the partition function and its gradient intractable to calculate directly, while inference on large loopy graphs is expensive.The segmentation graph has more than 1,000 nodes and 2 × 10^4 pairwise connections per image, and CNN training requires tens or hundreds of thousands of SGD iterations.Source: §7 · p. 8

7.1 Piecewise training of CRFs

  • Piecewise training replaces the global CRF objective with independent likelihoods defined on unary and pairwise potentials.The objective minimizes negative log likelihood with regularization.Source: §7.1 · pp. 8–9
  • Because piecewise training omits the global partition function, each potential's gradients can be calculated without expensive inference and the potentials can be trained in parallel.The unary likelihood uses a softmax over the K classes, with analogous treatment for pairwise potentials.Source: §7.1 · p. 9
  • The authors present piecewise training as a practical way to learn CNN-based CRFs on large-scale data without repeated inference during CNN training.The approach addresses the computational cost of inference at every SGD iteration.Source: §7.1 · p. 9

8 IMPLEMENTATION DETAILS

The implementation uses separate feature-map and potential networks, samples pairwise connections to reduce computation, and updates the Pairwise-Net asynchronously to limit memory use and batch size.

8 IMPLEMENTATION DETAILS

  • Each potential type has its own FeatMap-Net and shallow potential network, allowing specialized features and parallel training across potential types.The model uses one unary and two pairwise potential types.Source: §8 · p. 9

8.1 Efficient learning

  • Sampling 24 neighboring nodes on a regular 5 × 5 grid reduces pairwise connections by an order of magnitude without degrading observed performance.The sampling excludes self-connections and is intended to remove redundant local relations.Source: §8.1 · p. 9

8.2 Asynchronous gradient update

  • Pairwise-Net asynchronous updates address GPU-memory demands and concerns that very large batches may harm convergence or performance.The authors note that processing all connections in one gradient iteration could create an excessively large batch.Source: §8.2 · p. 9
  • Within each SGD iteration, Pairwise-Net is updated over subsets of connections before the accumulated gradients update FeatMap-Net.The example subset contains 2,000 connections, reducing memory use and Pairwise-Net batch size.Source: §8.2 · pp. 9–10

9 EXPERIMENTS

The experiments report strong results across segmentation datasets and evaluate both contextual components and VOC training settings. On PASCAL VOC 2012, the reported result reaches 77.8 IoU with middle-layer refinement.

9 EXPERIMENTS

  • The method reports new state-of-the-art performance across eight segmentation datasets, including indoor, outdoor, and street scenes.The evaluation includes PASCAL VOC 2012, NYUDv2, PASCAL-Context, SIFT-flow, SUN-RGBD, KITTI, COCO, and Cityscapes.Source: §9 · p. 10

9.1 Results on the NYUDv2 dataset

  • On NYUDv2, the contextual model with CNN pairwise potentials achieves the best performance, setting a new state-of-the-art result; the model uses no depth information.Source: §9.1 · pp. 10–11

9.1.1 Component evaluation

  • On NYUDv2, multi-scale inputs and sliding pyramid pooling improve the baseline, dense-CRF refinement adds further gains, and CNN pairwise potentials improve performance again.The component evaluation reports the best performance after adding contextual CNN pairwise potentials.Source: Table 2 · p. 10, §9.1.1 · p. 11

9.1.2 Comparison with multi-unary ensemble

  • On NYUDv2, one pairwise potential with one unary potential outperforms unary-only ensembles, and adding a second unary network improves the CRF model further.The table caption identifies the comparison as involving unary-only ensembles of up to four networks.Source: Table 3 · p. 10, §9.1.2 · pp. 11–12

9.2 Results on the PASCAL VOC 2012 dataset

  • Training only on VOC images yields IoU 75.3, the best among methods using only VOC training data; adding extra COCO training images raises our model’s IoU to 77.2.Source: §9.2 · p. 12
  • 77.8 IoU is reported as the best result on PASCAL VOC 2012 after adding middle-layer feature refinement to the coarse prediction.The refinement changes the prediction from 1/16 to 1/2 input resolution before final boundary refinement.Source: §9.2 · p. 12
  • On VOC, the method outperforms DPN on 18 of 20 categories with VOC-only training and on 15 of 20 with VOC+COCO training.Table 4 reports the individual category IoU scores.Source: Table 4 · p. 11, §9.2 · p. 12

9.3 Results on the Cityscapes dataset

Across six benchmark sections, the method is reported to outperform competing methods or achieve the best performance, including on PASCAL-Context, SUN-RGBD, SIFT-flow, and KITTI.

9.3 Results on the Cityscapes dataset

  • On Cityscapes, the method outperforms other competing methods.The refinement network outputs predictions at 1/4 of the input image size.Source: §9.3 · p. 12

9.4 Results on the PASCAL-Context dataset

  • The method significantly outperforms competitors on PASCAL-Context, achieving the best reported result on this dataset.The dataset evaluation uses 60 classes, including a background class.Source: §9.4 · p. 13, Table 6 · p. 13

9.5 Results on the SUN-RGBD dataset

  • On SUN-RGBD, the method outperforms existing methods by a large margin without using depth information for training.The dataset provides pixel labels for 37 classes.Source: §9.5 · p. 13, Table 7 · p. 13

9.6 Results on the COCO dataset

  • On COCO, the method significantly outperforms the fully convolutional network baselines.The evaluation treats regions smaller than 200 pixels as void.Source: §9.6 · p. 13, Table 8 · p. 13

9.7 Results on the SIFT-flow dataset

9.8 Results on the KITTI dataset

  • On KITTI road segmentation, the method performs best among the compared methods.The paper also reports a COCO-pretrained variant, denoted “ours+”.Source: §9.8 · p. 14, Table 10 · p. 14

10 CONCLUSIONS

The paper combines CNNs and CRFs to model contextual information for semantic segmentation, reporting state-of-the-art performance across eight datasets.

10 CONCLUSIONS

  • The proposed segmentation method combines CNNs and CRFs to exploit complex contextual information.CNN-based pairwise potentials model semantic relations between image regions.Source: §10 · p. 14
  • The method achieves state-of-the-art performance on all eight evaluated segmentation datasets, including PASCAL VOC 2012.Source: §10 · p. 14
  • The authors state that the proposed method is potentially applicable to other tasks.Source: §10 · p. 14

(a) Testing

The figure presents prediction examples on the PASCAL-Context dataset.

(a) Testing

  • The figure shows prediction examples on the PASCAL-Context dataset.Source: Figure 14 · p. 15
Loading 1603.03183v3…