PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Segmentation with Boundary Neural Fields

Gedas Bertasius, Jianbo Shi, Lorenzo Torresani

arXiv:1511.02674v2cs.CV

TL;DR

FCN-based segmentation suffers from blurred, poorly localized boundaries, and common CRF remedies add tuning difficulties while relying on unsuitable color affinities. The paper introduces BNF, which derives semantic boundaries from FCN features and uses them as pairwise terms in a relaxed global energy. Minimizing this energy yields better localized and spatially smoother segmentations than prior methods.

  • Problem

    FCNs produce blurred, low-resolution segmentations with poorly localized object boundaries, while CRF remedies add difficult-to-tune parameters and low-level affinities.

  • Method

    BNF uses a semantic segmentation FCN to predict boundaries and unaries, incorporates boundary-based pairwise potentials, and relaxes a global energy decomposed into binary problems.

  • Results

    Minimizing the global boundary-based energy produces segmentations better localized around object boundaries and spatially smoother than prior methods.

  • Takeaways & Limitations

    Semantic boundaries from FCN features are better suited to semantic segmentation than commonly used color-based affinities.

  • Takeaways & Limitations

    The entire BNF architecture is not jointly optimized end-to-end; the authors leave joint optimization for future work.

Abstract

from arXiv · show

The state-of-the-art in semantic segmentation is currently represented by fully convolutional networks (FCNs). However, FCNs use large receptive fields and many pooling layers, both of which cause blurring and low spatial resolution in the deep layers. As a result FCNs tend to produce segmentations that are poorly localized around object boundaries. Prior work has attempted to address this issue in post-processing steps, for example using a color-based CRF on top of the FCN predictions. However, these approaches require additional parameters and low-level features that are difficult to tune and integrate into the original network architecture. Additionally, most CRFs use color-based pixel affinities, which are not well suited for semantic segmentation and lead to spatially disjoint predictions. To overcome these problems, we introduce a Boundary Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues. The boundary information is used to enhance semantic segment coherence and to improve object localization. Specifically, we first show that the convolutional filters of semantic FCNs provide good features for boundary detection. We then employ the predicted boundaries to define pairwise potentials in our energy. Finally, we show that our energy decomposes semantic segmentation into multiple binary problems, which can be relaxed for efficient global optimization. We report extensive experiments demonstrating that minimization of our global boundary-based energy yields results superior to prior globalization methods, both quantitatively as well as qualitatively.

1. Introduction

FCNs improve semantic segmentation but produce blurry, low-resolution predictions with poorly localized boundaries, while Dense-CRF post-processing introduces tuning issues and spatially disjoint segments. The proposed BNF uses FCN features to predict semantic boundaries and integrates them into a global energy for improved segmentation.

  • Motivation: Large receptive fields and pooling layers blur FCN predictions and reduce their spatial resolution.These effects make predicted segments blobby and lacking fine object-boundary details.
  • Motivation: Dense-CRF post-processing adds difficult-to-tune parameters and relies on low-level affinities that can miss semantic relationships.Color-based affinities can produce spatially disjoint object segments.
  • Boundary Neural Field: BNF uses one semantic segmentation FCN to predict semantic boundaries and generate segmentation maps through global optimization.The FCN supplies both semantic unaries and boundary cues for the global energy.
  • Boundary Neural Field: Semantic boundaries are predicted as a learned linear combination of interpolated convolutional feature maps.A sigmoid is applied to the weighted sum to produce the boundary prediction.
  • Global Optimization: The proposed global energy decomposes semantic segmentation into multiple binary problems and relaxes the integrality constraint.The authors report better qualitative and quantitative results than traditional globalization models such as MRFs and CRFs.

2. Related Work

Prior segmentation and boundary-detection methods either use staged or independent predictions, hand-constructed features, or global inference with limitations. BNF instead derives semantic boundary affinities from FCN features and uses them for globally optimized segmentation.

  • Boundary Detection: Spectral boundary methods perform global inference over entire images and tend to be slow.They formulate a generalized eigenvalue system whose eigenvectors predict boundaries.
  • Boundary Detection: Local boundary classifiers improve efficiency but often depend on hand-constructed features that are difficult to tune.Sketch tokens and structured edges are cited as prominent examples using fast random forests.
  • Boundary Detection: Deep boundary methods use CNNs, with some optimizing networks from scratch and others using pretrained models to compute boundaries.The related approaches include N4 fields, DeepNet, DeepContour, DeepEdge, HFL, and HED.
  • BNF: BNF uses semantic boundaries from FCN features as pairwise affinities and predicts segmentation globally without selecting candidate contour points.The method reports state-of-the-art boundary detection by F-score and Average Precision, while improving localization and fine structural details.
  • Semantic Segmentation: Independent per-pixel predictions fail to exploit relationships between pixels and may produce spatially disjoint segmentations.This is identified as a weakness of deep per-pixel CNN segmentation approaches.

3. Boundary Neural Fields

Boundary Neural Fields use FCN features to predict semantic boundaries, then incorporate those boundaries into pairwise affinities and a globally optimized segmentation energy. The resulting boundary-aware approach improves boundary detection and produces smoother, better-localized segmentations.

  • Boundary Neural Fields: BNFs combine FCN-derived unary potentials with boundary-based pairwise affinities in a global energy model.The energy is minimized to improve the initial FCN segmentation.
  • Learning to Predict Semantic Boundaries: Semantic boundaries are predicted as a nonlinear function of a linear combination of interpolated FCN feature maps.Resizing feature maps to image dimensions enables pixel-level boundary probabilities without selecting candidate boundary points beforehand.
  • Learning to Predict Semantic Boundaries: The boundary detector learns feature-map weights from BSDS500 boundary annotations after FCN training on semantic segmentation.Training balances samples across four quartiles of annotator agreement.
  • Learning to Predict Semantic Boundaries: FCN feature maps with the largest learned weight magnitudes contain highly localized boundary information.Figure 3 visualizes the feature maps contributing most heavily to boundary detection.
  • Boundary Detection Results: The boundary detector outperforms prior methods on BSDS500 according to both F-score measures and average precision.The final boundary maps apply non-maximum suppression to pixel-level softmax outputs.
  • Constructing Pairwise Pixel Affinities: Boundary-based affinities treat pixels as similar when no boundary crosses their connecting path and as increasingly dissimilar when stronger boundaries cross it.The affinity uses the maximum boundary value along the straight path between two pixels.
  • Constructing Pairwise Pixel Affinities: The final affinity also incorporates softmax-based object information, setting similarity to zero for pixels with different hard segmentation labels.For matching labels, similarity depends on the difference between their most likely-class softmax outputs.
  • Global Inference: The global inference formulation relaxes constraints and decomposes semantic segmentation into multiple binary problems.This design yields better qualitative and quantitative segmentation than traditional MRF and CRF globalization models.

4. Experimental Results

Experiments on SBD compare boundary-based affinities and global inference against common alternatives using shared FCN unary potentials. BNF improves semantic segmentation quantitatively and produces better-localized, smoother boundaries qualitatively.

  • Evaluation Setup: The evaluation uses SBD annotations for 20 Pascal VOC classes and reports PP-IOU and PI-IOU metrics.PP-IOU measures per-pixel accuracy, while PI-IOU measures per-image accuracy.
  • Evaluation Setup: All compared global inference methods use the same FCN unary potentials, isolating the effects of affinity functions and inference strategies.The comparisons include Belief Propagation, ICM, Graph Cuts, and Dense-CRF.
  • Affinity Functions: Boundary-based affinities improve results for all evaluated global inference methods on both metrics compared with color-based affinities.The two global models differ in their pairwise affinities while sharing FCN-derived unary potentials.
  • Inference Methods: BNF outperforms other globalization methods by 0.3% on PP-IOU and 1.3% on PI-IOU.The comparison uses the BNF variants and shared unary potentials described for Table 3.
  • Inference Methods: Most of the observed benefit comes from the semantic boundary affinity term rather than the softmax affinity term.BNF-SB uses boundary affinities alone, whereas BNF-SB-SM combines boundary and softmax affinities.
  • Qualitative Results: BNF segmentations are better localized around object boundaries and spatially smoother than FCN and Dense-CRF predictions.These qualitative comparisons are reported in Fig. 5.
  • Semantic Boundary Classification: Semantic boundary classification achieves a mean Max F-Score of 54.5%, compared with 51.7% for HFL.The scores are averaged across all 20 classes on the SBD dataset.

5. Conclusions

The Boundary Neural Field combines FCN predictions with predicted semantic boundaries in a global optimization that improves localization and spatial smoothness. Its differentiable energy also leaves open joint end-to-end learning of additional parameters.

  • The Boundary Neural Field uses an FCN to predict semantic boundaries, then combines those boundaries with FCN outputs in global optimization.
  • Boundary-based affinities are better suited to semantic segmentation than commonly used low-level color-based affinities.
  • The global energy decomposes semantic segmentation into multiple binary problems and relaxes an integrality constraint.
  • Minimizing the global energy produces segmentations better localized around object boundaries and spatially smoother than those from prior methods.
  • Because the global energy is differentiable, future work could add parameters to BNFs and optimize the full architecture jointly.
Loading 1511.02674v2…