PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Contrast Learning for Salient Object Detection

Guanbin Li, Yizhou Yu

arXiv:1603.01976v1cs.CV

TL;DR

Existing CNN-based salient object detectors operate on patches, causing blurry boundaries and redundant computation. The paper proposes an end-to-end deep contrast network with complementary pixel-level and segment-wise streams, and reports significant improvement over the state of the art.

  • Problem

    Patch-level CNN saliency methods produce blurry maps near object boundaries and redundantly process overlapping image patches.

  • Method

    An end-to-end deep contrast network combines a multi-scale fully convolutional stream with a segment-wise spatial pooling stream, fusing their saliency maps.

  • Results

    3.5%, 5.0%, 7.7%, 7.6% and 6.0% maximum F-measure improvements are reported over the best existing algorithms on MSRA-B, HKU-IS, DUT-OMRON, PASCAL-S and SOD, respectively.

  • Takeaways & Limitations

    The fused network significantly improves the state of the art in salient object detection.

Abstract

from arXiv · show

Salient object detection has recently witnessed substantial progress due to powerful features extracted using deep convolutional neural networks (CNNs). However, existing CNN-based methods operate at the patch level instead of the pixel level. Resulting saliency maps are typically blurry, especially near the boundary of salient objects. Furthermore, image patches are treated as independent samples even when they are overlapping, giving rise to significant redundancy in computation and storage. In this CVPR 2016 paper, we propose an end-to-end deep contrast network to overcome the aforementioned limitations. Our deep network consists of two complementary components, a pixel-level fully convolutional stream and a segment-wise spatial pooling stream. The first stream directly produces a saliency map with pixel-level accuracy from an input image. The second stream extracts segment-wise features very efficiently, and better models saliency discontinuities along object boundaries. Finally, a fully connected CRF model can be optionally incorporated to improve spatial coherence and contour localization in the fused result from these two streams. Experimental results demonstrate that our deep model significantly improves the state of the art.

1. Introduction

Salient object detection remains challenging because patch-based CNN methods produce blurry boundaries and redundant computation. The paper addresses these limitations with an end-to-end network combining pixel-level and segment-wise streams.

  • Salient object detection supports tasks including content-aware image editing, object detection, image classification, person re-identification, and video summarization.
  • Handcrafted local contrast can miss homogeneous salient regions, while global contrast can struggle with complex backgrounds.
  • Patch-level CNN methods assign each pixel its enclosing patch’s saliency value, producing blurry maps near salient-object boundaries.
  • The proposed end-to-end network runs once on an input image and produces a complete saliency map at the input resolution.
  • The two streams are trained together, fused into a saliency map, and optionally refined with a fully connected CRF for spatial coherence and contour localization.
  • Its multi-scale fully convolutional stream predicts pixel-level saliency while capturing semantic properties and visual contrast across feature scales.
  • A segment-wise spatial pooling stream efficiently extracts segment-wise features and models contrast and saliency discontinuities along region boundaries.

2. Related Work

Deep learning improved salient object detection, but earlier methods relied on independent local patches. The paper’s approach uses an end-to-end architecture designed to avoid redundant computation from overlapping patches.

  • Deep CNNs have improved salient object detection by providing more robust and semantically informative features than handcrafted representations.
  • Earlier deep saliency methods treated local image patches as independent training and testing samples.
  • Ignoring shared computation among overlapping patches creates redundant feature computation and high training and testing costs.
  • The paper organizes its deep contrast network around a fully convolutional stream and a segment-wise spatial pooling stream.

3. Deep Contrast Network

The deep contrast network combines a pixel-level multi-scale fully convolutional stream with a segment-wise spatial pooling stream, then fuses their saliency maps. The design addresses coarse pixel predictions and boundary discontinuities while avoiding repeated image-wide convolution.

  • Architecture: The network has complementary fully convolutional and segment-wise spatial pooling streams whose saliency maps are fused through a learned 1 × 1 convolutional layer.The fully convolutional stream produces S1, while the segment-wise stream produces S2.
  • Multi-Scale Fully Convolutional Network: Skipping subsampling in VGG16’s last two pooling layers maintains an 8-pixel stride, while the hole algorithm preserves receptive fields for denser prediction maps.The original convolutional evaluation would produce a 32-pixel stride; the hole algorithm avoids approximation while supporting dense feature maps.
  • Multi-Scale Fully Convolutional Network: The multi-scale fully convolutional stream maps the raw image to a pixel-level saliency map using contrast across feature maps at different receptive-field scales.It modifies VGG16 and combines five output maps generated from receptive fields at four different scales plus the final output map.
  • Segment-Level Saliency Inference: The segment-wise stream uses superpixel masks and fixed-grid spatial pooling to create fixed-length features for variable-size segments.Each segment feature has h × w × C dimensions after pooling over the segment bounding box.
  • Segment-Level Saliency Inference: Three nested windows provide segment, neighborhood, and whole-map context for visual-contrast estimation before segment saliency prediction.The windows are the segment bounding box, neighboring-segment bounding box, and the entire Conv5 3 map with the segment masked out.
  • Segment-Level Saliency Inference: The segment-wise stream is more efficient than the related approach because convolutional feature maps are computed once and reused across thousands of segments.Its segment features are extracted from the fine-tuned multi-scale network rather than the original VGG16 classification model.
  • Training: Training initializes the second stream separately, then alternates optimization of the two streams and fusion weights using cross-entropy and squared prediction errors.The first stream and fusion weights use cross entropy, while the second stream minimizes squared errors over segments.

4. The Complete Algorithm

The complete algorithm decomposes images into connected superpixels, fuses the two stream outputs, and optionally refines them with a fully connected CRF. The refinement promotes spatial coherence and contour preservation in the final saliency map.

  • Superpixel Decomposition: The algorithm uses modified SLIC with geodesic image distance in CIELab space to decompose each input image into connected, edge-preserving superpixels.The paper reports little performance variation when the number of superpixels is varied within an experimental range.
  • Spatial Coherence: The fully connected CRF refines fused pixelwise saliency scores by solving a binary salient-versus-nonsalient labeling problem.Initial unary probabilities come from the fused saliency map, with P(1) = S_i and P(0) = 1 − S_i.
  • Spatial Coherence: The CRF pairwise potential encourages nearby similarly colored pixels to receive similar labels and removes small isolated regions.Its kernels depend on pixel positions and intensities, with parameters controlling spatial and color influence.
  • Spatial Coherence: Less than 0.5 second is required to minimize the CRF energy on a 300×400-pixel image using mean-field inference and high-dimensional filtering.The final saliency map is generated from each pixel’s posterior probability of being salient.
  • Spatial Coherence: Without CRF, the proposed saliency maps can be coarse and lose object contours; CRF refinement produces smoother maps with pixelwise accuracy and better contour preservation.The comparison is presented in Figure 3 and accompanied by a quantitative study in Section 5.3.2.

5.1. Experimental Setup

The evaluation uses five public datasets, standard saliency metrics, and a cross-dataset testing protocol. Training and implementation details specify the data split, thresholds, network input, and runtime.

  • Datasets: Performance is evaluated on five public datasets: MSRA-B, PASCAL-S, DUT-OMRON, HKU-IS, and SOD.MSRA-B contains 5,000 images, PASCAL-S 850 images, and DUT-OMRON 5,168 challenging images.
  • Datasets: MSRA-B is split into 2,500 training, 500 validation, and 2,000 testing images, with models tested on the other datasets.The cross-dataset tests use models trained on MSRA-B.
  • Evaluation Criteria: Evaluation uses precision-recall curves, F-measure, and mean absolute error to compare saliency maps with ground truth.Precision and recall use thresholded binary masks; MAE measures average absolute per-pixel error.
  • Evaluation Criteria: The study compares 11 saliency detection methods across three datasets using precision, recall, and adaptive-threshold F-measure curves.Figure 5 compares PR curves, while Figure 6 reports precision, recall, and F-measure using a per-image adaptive threshold.
  • Implementation: Images are resized to 321 × 321 for training, and the segment-level stream uses 400 superpixels at three scales.The implementation uses Caffe with specified learning-rate, momentum, and weight-decay settings.
  • Implementation: DCL detects a 400 × 300 image in 1.5 seconds, while CRF refinement adds 0.8 second per image.Training the network on MSRA-B takes around 25 hours on the reported hardware.

5.2. Comparison with the State of the Art

DCL and DCL+ outperform existing methods across public saliency datasets, with DCL+ achieving the strongest quantitative results. The visual comparison also highlights performance in challenging object and contrast configurations.

  • Visual Comparison: DCL+ produces saliency maps closest to ground truth in visual comparisons, including boundary-touching, disconnected, and low-contrast objects.The comparison also notes that MC and LEGS are overrated on datasets containing training samples.
  • Quantitative Comparison: DCL and DCL+ outperform the evaluated state-of-the-art methods across public datasets on PR curves and average precision, recall, and F-measure.The comparison includes eight recent methods and an FCN adapted from semantic segmentation.
  • Quantitative Comparison: DCL+ improves maximum F-measure over the best existing algorithm by 3.5%, 5.0%, 7.7%, 7.6%, and 6.0% across MSRA-B, HKU-IS, DUT-OMRON, PASCAL-S, and SOD.MC and LEGS are skipped for MSRA-B, and LEGS is skipped for PASCAL-S, because of training-test overlap concerns.
  • Quantitative Comparison: DCL+ lowers MAE by 28.8%, 35.5%, 9.1%, 25.5%, and 18.7% on MSRA-B, HKU-IS, DUT-OMRON, PASCAL-S, and SOD.The reported reductions use the same dataset-specific exclusions for MC and LEGS.
  • Analysis: Figure 7 assesses the proposed network components and CRF refinement as part of the state-of-the-art comparison.The componentwise and CRF analyses are reported separately from the method comparisons.

5.3. Ablation Studies

Ablation studies show that the two streams complement each other, with the fused DCL map performing best, while CRF refinement further improves accuracy.

  • Effectiveness of Deep Contrast Network: The fused DCL saliency map achieves the best average precision, recall, and F-measure among the two individual streams and their fusion.The fully convolutional stream contributes more to the fused result, while the streams remain complementary.
  • Effectiveness of Deep Contrast Network: The complete multi-scale fully convolutional stream outperforms its best single-scale version on PR curves, average precision, recall, and F-measure.The single-scale comparison uses the fully convolutional version of the original VGG16 network.
  • Effectiveness of CRF: Adding the fully connected CRF improves the accuracy of the final saliency model on MSRA-B testing images.The CRF is incorporated to improve spatial coherence of the saliency maps.

6. Conclusions

The paper introduces an end-to-end deep contrast network for salient object detection, combining pixel-level and segment-level processing with optional CRF refinement. Experiments show significant improvement over the state of the art.

  • The network combines a pixel-level fully convolutional stream with a segment-level spatial pooling stream.The two components are complementary and their fused result can be refined with a fully connected CRF for spatial coherence and contour localization.
  • A fully connected CRF can optionally improve spatial coherence and contour localization in the fused saliency result.
  • Experimental results demonstrate that the deep model significantly improves the state of the art in salient object detection.
Loading 1603.01976v1…