PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional CRFs for Semantic Segmentation

Marvin T. T. Teichmann, Roberto Cipolla

arXiv:1805.04777v2cs.CV

TL;DR

CNNs provide powerful local features for semantic segmentation, but lack global context and direct interactions between predictions, while CRFs are slow and hard to optimize. The paper introduces convolutional CRFs, adding conditional independence to fully-connected CRFs so inference can use GPU-efficient convolutions. The authors report a two-orders-of-magnitude speed increase and a modest accuracy improvement with exact message passing.

  • Problem

    CNNs lack global context and direct prediction interactions, while CRFs offer structured modeling but are slow and difficult to optimize.

    Source: §1 · p. 1

  • Method

    ConvCRFs add conditional independence to fully-connected CRFs, reformulating much of inference as GPU-efficient convolutions and enabling backpropagation to train CRF parameters.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    Training and inference speed increase by two orders of magnitude, with a modest accuracy improvement when message passing is computed exactly.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors present faster training and inference as a way to benefit future CRF research, and report that Gaussian features can be trained by backpropagation.

    Source: §1 · p. 2, §6 · p. 9

    The locality assumption sets pairwise potentials to zero for pixel pairs farther apart than k.

    Source: §4 · p. 4

Abstract

from arXiv · show

For the challenging semantic image segmentation task the most efficient models have traditionally combined the structured modelling capabilities of Conditional Random Fields (CRFs) with the feature extraction power of CNNs. In more recent works however, CRF post-processing has fallen out of favour. We argue that this is mainly due to the slow training and inference speeds of CRFs, as well as the difficulty of learning the internal CRF parameters. To overcome both issues we propose to add the assumption of conditional independence to the framework of fully-connected CRFs. This allows us to reformulate the inference in terms of convolutions, which can be implemented highly efficiently on GPUs. Doing so speeds up inference and training by a factor of more then 100. All parameters of the convolutional CRFs can easily be optimized using backpropagation. To facilitating further CRF research we make our implementation publicly available. Please visit: https://github.com/MarvinTeichmann/ConvCRF

1 Introduction

CNNs provide strong local features for semantic segmentation, while CRFs add structured modeling; the authors attribute CRFs’ declining use chiefly to slow inference and difficult optimization. They propose convolutional CRFs to make inference efficient and parameters trainable by backpropagation.

1 Introduction

  • CNNs extract strong local features but lack global context and direct interactions between predictions, motivating structured models such as CRFs.Source: §1 · p. 1
  • CRFs are slow to infer and hard to optimize, limiting practical use and making extended training and experimentation impractical.The passage reports CRF inference as two orders of magnitude slower than CNN inference.Source: §1 · p. 1
  • The method adds conditional independence to fully-connected CRFs, allowing much of inference to be reformulated as GPU-efficient convolutions.The authors name the resulting method convolutional CRFs (ConvCRFs).Source: §1 · pp. 1–2
  • ConvCRF inference takes less than 10 ms, a two-orders-of-magnitude speed increase over FullCRFs, and all parameters can be trained by backpropagation.Source: §1 · p. 2

2 Related Work

Prior segmentation pipelines largely rely on CNN feature extraction, while CRFs and related approaches address structured prediction and context. The passage highlights challenges in learning CRF features and improving inference speed.

2 Related Work

  • CNN segmentation architectures primarily rely on feature extraction and make pixel-wise predictions, leaving structured knowledge and background context unmodeled.Source: §2 · p. 2
  • Fully-connected CRFs are commonly applied to CNN predictions, whereas pyramid pooling expands the field of view without providing true structured reasoning.Source: §2 · p. 2
  • Earlier CRF learning methods retain hand-crafted Gaussian features or use approaches described as approximate, difficult to integrate, or slow at inference.CRFasRNN [41] and the gradient-based approach require hand-crafted pairwise features; piecewise training [20] is approximate and inference remains slow.Source: §2 · p. 2
  • Some CRF pipelines downsample outputs by 8 × 8 to speed inference, but this harms predictive capability; the authors report no significant speed progress since FullCRFs [17].Source: §2 · p. 2

3 Fully Connected CRFs

Fully-connected CRFs model segmentation as a conditional distribution over pixel labels, with unary terms for individual pixels and pairwise terms for pixel interactions. Mean-field inference is parallelizable except for message passing, which is the computational bottleneck.

3 Fully Connected CRFs

  • A fully-connected CRF represents an image segmentation as a random field of pixel labels and seeks the labeling that maximizes the conditional probability given the image.Source: §3 · p. 2
  • Unary potentials provide per-pixel segmentation predictions, while pairwise potentials model interactions between pixel labels using image-dependent features.The passage notes that modern pipelines commonly use CNNs to predict unaries and that similar-color pixels may be likely to share a class.Source: §3 · p. 3
  • FullCRFs use weighted Gaussian kernels with hand-crafted appearance and spatial features; their learnable parameters include kernel weights and scales.The Potts compatibility function encourages pixels with similar features to receive the same prediction.Source: §3 · p. 3

3.1 Mean Field Inference

  • Mean-field inference parallelizes most steps, but message passing is a bottleneck: exact computation is quadratic in pixel count, so FullCRFs use a permutohedral-lattice approximation.The approximation’s complex data structure does not follow the SIMD paradigm of efficient GPU computation, and efficient gradient computation is non-trivial.Source: §3.1 · p. 3

4 Convolutional CRFs

ConvCRFs impose a locality-based conditional-independence assumption on fully connected CRFs, then express message passing as convolutions. The section describes the implementation and design choices used to make that computation efficient and trainable.

4 Convolutional CRFs

  • The conditional-independence assumption sets pairwise potentials to zero for pixels more than k Manhattan-distance units apart.The paper calls k the filter-size and describes this locality assumption as strong.Source: §4 · p. 4

4.1 Efficient Message Passing in ConvCRFs

  • Exact message passing is reformulated as convolution with a truncated Gaussian kernel, enabling efficient GPU computation without the permutohedral-lattice approximation.The operation is implemented similarly to CNN convolutions, with filters that vary spatially but are constant across channels.Source: §4.1 · p. 4
  • 90% of GPU time went to data reorganization when using standard CNN operations, motivating a native low-level implementation that yielded an additional 10-fold speedup.The implementation tiles the input and replaces the spatial batched matrix multiplication used by 2D convolutions with a batched dot product over channels.Source: §4.1 · p. 4

4.2 Additional implementation details

  • For baseline comparability, ConvCRFs use softmax normalization, the Potts model, and the same hand-crafted Gaussian features as prior approaches.Gaussian blur increases the effective filter size by a factor of 4.Source: §4.2 · p. 4
  • Table 1 compares CRFs on synthetic data using full-resolution Pascal VOC images; speed tests use a GTX 1080 Ti, and Conv7 denotes filter-size 7.Source: Table 1 · p. 5
  • Additional experiments make smoothness-kernel features learnable and use a 1 × 1 convolution for the compatibility transformation.The learnable feature variables start from the hand-crafted feature values.Source: §4.2 · p. 5

5 Experimental Evaluation

Experiments evaluate ConvCRFs on synthetic denoising and PASCAL VOC, using both decoupled and end-to-end training. The reported results cover CRF comparisons, training behavior, and computational cost.

5 Experimental Evaluation

  • On PASCAL VOC 2012, the unary model achieves 71.23% validation mIoU and 91.84% training mIoU, with 10,582 labeled training images and 1,464 validation images.The 200-image held-out subset is used to tune internal CRF parameters, leaving 10,382 images to train the unary CNN.Source: §5 · p. 5

5.1 ConvCRFs on synthetic data

  • On synthetic denoising, ConvCRFs outperform FullCRFs with identical parameters and run two orders of magnitude faster.The authors attribute the performance difference to exact message passing, which avoids permutohedral-lattice approximation errors; Figure 1 shows boundary artifacts in the approximation.Source: §5.1 · p. 6, Figure 1 · p. 6

5.2 Decoupled training of ConvCRFs

  • With decoupled training on the same unary predictions, CRFs boost performance, ConvCRFs slightly outperform FullCRFs, and learnable compatibility plus Gaussian features performs best.In Figure 3, examples 2 and 4 are cases where CRFs do not improve the unary predictions.Source: §5.2 · p. 7, Table 2 · p. 7, Figure 3 · p. 8

5.3 End-to-End learning with ConvCRFs

  • End-to-end training propagates gradients through five mean-field iterations, uses an auxiliary unary loss to counter vanishing gradients, and takes about 30 hours on four 1080Ti GPUs.The Table 3 comparison is described as not entirely fair.Source: §5.3 · p. 7, Table 3 · p. 8

Supplementary Material.

The training protocols use held-out training data to tune CRF parameters and begin joint training before unary learning converges. The authors note constraints on comparing their end-to-end training with CRFasRNN.

Decoupled training protocol

  • A held-out training subset is used for CRF tuning because unary mIoU differs between training and validation data: 91.84% versus 71.23%.The authors state that the CRF must represent how trustworthy the unary predictions are.Source: Decoupled training protocol · pp. 8–9

End-to-end training protocol

  • Starting joint training at epoch 100, before unary learning converges, is crucial because a later learning-rate restart drops unary validation mIoU to around 69% without recovery.The training mIoU jumps when end-to-end training begins.Source: End-to-end training protocol · p. 9, Figure 4 · p. 9
  • The CRFasRNN comparison is not entirely fair: its five-hour epochs and batch-size-one constraint limit protocol matching.The authors report better models from their approach but qualify the quantitative comparison.Source: End-to-end training protocol · p. 9

6 Conclusion

The conclusion presents ConvCRFs as a CRF design that uses conditional independence to make message passing convolutional and efficient on GPUs. The authors identify further Gaussian-feature learning, richer CRF architectures, and applications beyond semantic segmentation as future directions.

6 Conclusion

  • Conditional independence removes the permutohedral lattice approximation and enables GPU-efficient message passing as convolution operations.The authors report that this design increases CRF training and inference speed by two orders of magnitude.Source: §6 · p. 9
  • Exact message passing yields a modest accuracy improvement, and ConvCRFs allow Gaussian features to be trained using backpropagation.Source: §6 · p. 9
  • Future work will investigate Gaussian-feature learning, more sophisticated CRF architectures, and ConvCRFs for instance segmentation and landmark recognition.Source: §6 · p. 9

Related papers

Loading 1805.04777v2…