Source-linked AI summary
Deeply supervised salient object detection with short connections
Qibin Hou, Ming-Ming Cheng, Xiao-Wei Hu, Ali Borji, Zhuowen Tu, Philip Torr
TL;DR
Saliency detection still struggles with scale-space handling, and generic FCN or HED-style feature fusion does not fully resolve segmentation needs. The paper introduces short connections within a deeply supervised HED architecture to combine multi-scale features, reporting state-of-the-art results across five benchmarks with efficient prediction.
Problem
Generic CNN and FCN models leave substantial room for improvement in scale-space handling, while learned saliency representations can fail in cluttered, low-contrast scenes.
Method
The method adds short connections between deeper and shallower side outputs in an enhanced HED network, combining semantic localization with spatial detail under deep supervision.
Results
The approach significantly advances state-of-the-art salient object detection and produces a 300 × 400 pixel prediction map in 0.08s.
Takeaways & Limitations
Rich multi-scale feature maps at each layer support accurate salient-region and boundary capture across simple and difficult images.
Takeaways & Limitations
Failures remain when salient objects are incompletely segmented, backgrounds are complex or low-contrast, or objects are transparent.
Abstract
from arXiv · showhide
Recent progress on saliency detection is substantial, benefiting mostly from the explosive development of Convolutional Neural Networks (CNNs). Semantic segmentation and saliency detection algorithms developed lately have been mostly based on Fully Convolutional Neural Networks (FCNs). There is still a large room for improvement over the generic FCN models that do not explicitly deal with the scale-space problem. Holistically-Nested Edge Detector (HED) provides a skip-layer structure with deep supervision for edge and boundary detection, but the performance gain of HED on salience detection is not obvious. In this paper, we propose a new method for saliency detection by introducing short connections to the skip-layer structures within the HED architecture. Our framework provides rich multi-scale feature maps at each layer, a property that is critically needed to perform segment detection. Our method produces state-of-the-art results on 5 widely tested salient object detection benchmarks, with advantages in terms of efficiency (0.15 seconds per image), effectiveness, and simplicity over the existing algorithms.
1 INTRODUCTION
Salient object detection moved from hand-crafted cues toward CNNs, but generic FCN and HED-style approaches still leave challenges in scale handling and segmentation quality. The paper addresses these challenges by using short connections to combine deeper semantic and shallower spatial features.
- Hand-crafted saliency features often fail to generalize across complex and cluttered scenes.
- Learning-based methods integrate feature types better, yet manually designed fusion can produce low-contrast representations and missed salient objects.
- CNNs and FCNs improved salient object detection, while generic CNN models still leave substantial room for scale-space improvement.
- Short connections transfer high-level localization cues to shallower outputs and low-level detail to deeper-output refinements.
- 0.08s produces a 300 × 400 pixel prediction map while the architecture supplies rich multi-scale feature maps at each layer.
2 RELATED WORKS
Related work progresses from hand-crafted and CNN-based saliency models to architectures that fuse multi-level features. The paper positions short connections as a more flexible way to combine deeper semantic and shallower spatial information while addressing scale-space limitations.
- Earlier saliency methods relied on hand-crafted local, global, or combined features, with detailed surveys covering this extensive literature.
- CNN-based models replaced these traditional representations with learned hierarchical or contrast-based features for salient object detection.
- The proposed architecture introduces short connections to combine the advantages of deeper and shallower layers, with its most general pattern subsuming earlier forms.
- Hypercolumn, FCN-like, and HED structures fuse features across levels or scales, but generic CNN models still leave room for scale-space improvement.
3 DEEP SUPERVISION WITH SHORT CONNEC-
The proposed network extends HED with six supervised side outputs and a fusion loss so features from multiple VGGNet scales contribute to salient object detection.
- Six side outputs represent the six VGGNet scales, and each side output receives deep supervision.
- A fusion loss combines features from different levels to capture multi-scale information in the final prediction.
3.1 Observations
The authors observe complementary strengths in HED side outputs: deeper layers locate salient objects, while shallower layers preserve spatial and boundary information. They therefore propose a top-down combination of both feature types.
- 3.1 Observations: Deeper side outputs provide high-level semantic knowledge that better locates salient objects but can produce irregular maps after down-sampling.
- 3.1 Observations: The proposed top-down method combines low-level and high-level features to improve saliency detection accuracy.
3.2 HED-based saliency detection
The method extends HED for salient object detection by adding a deeper side output, richer side-output processing, and deep supervision with weighted fusion. Enhanced HED improves over original HED, but deeper outputs still lose detail and shallower outputs remain messy.
- HED architecture: The HED architecture uses five side outputs connected to the last convolutional layer of each VGGNet stage.Each side output is paired with a classifier and trained using a weighted side loss.
- HED architecture: A weighted-fusion layer combines side activations, with fusion trained alongside the side-output losses.The side losses use image-level class-balanced cross-entropy, while the fusion loss measures distance between the ground-truth map and fused predictions.
- Enhanced HED: Enhanced HED adds a side output connected to VGGNet’s pool5 layer to better locate salient regions.Each side output also gains two convolutional layers with different filter channels and spatial sizes, followed by bilinear up-sampling.
- Enhanced HED: About 3% improvement is achieved by enhanced HED over original HED in salient object detection.Despite this gain, shallower side outputs remain messy, deeper outputs remain irregular, and deeper outputs lose detail.
3.3 Short connections
Short connections combine deeper semantic information with shallower spatial detail through a top-down refinement pathway. The resulting architecture supplies multi-scale features at each layer for dense and accurate saliency maps.
- Motivation: Deeper side outputs locate salient regions but lose details, whereas shallower outputs retain low-level detail but lack global information.This complementarity motivates combining side outputs rather than relying on a single depth.
- Training: Short-connection activations and their losses extend the side-output and fusion objectives, while selected connections can be dropped by setting their weights to zero.The new side loss remains a standard cross-entropy loss.
- Construction: For side output 2, score maps from side outputs 3 and 4 are upsampled, concatenated with its original map, reweighted, and transformed by a 1 × 1 convolution.The same construction extends to side outputs receiving more than one short connection.
- Architecture: The method adds short connections that pass information from deeper side outputs to shallower side outputs.These connections form a top-down details refinement stage following a saliency locating stage.
3.4 Implementation Details
Inference combines selected side-output maps and optionally applies a fully connected CRF to improve spatial coherence and boundary quality. The implementation uses Caffe with VGGNet and specified training and CRF settings.
- Implementation: The network uses Caffe, an FCN implementation, and VGGNet as its pretrained backbone.These choices support comparison with other methods.
- Inference: At inference, the method combines side-output maps because the deepest and shallowest predictions are individually unsatisfactory.Maps from selected intermediate side outputs help fill lost details.
- Inference: The final output map is chosen as the mean of the fused map and selected side-output maps, which improves results by a small margin.The combination compensates for detailed information still missing from the fusion output.
- Smoothing: A fully connected CRF is used during inference to improve spatial coherence and saliency-map boundary quality in complex scenes.The unary term uses normalized saliency values, a sigmoid function, and a scale parameter.
- Training: Training uses full-resolution images with mini-batches of 10, learning rate 1e-8, weight decay 0.0005, momentum 0.9, and side-output loss weight 1.Fusion weights are initialized to 0.1667, and CRF parameters are selected by cross-validation.
4 EXPERIMENTS AND ANALYSES
The experiments evaluate the method across five datasets, standard metrics, architecture patterns, side-output settings, upsampling choices, data augmentation, backbones, and CRF refinement. Ablations show that short connections and the selected side-output design improve performance, while ResNet-101 and the modulating factor provide further gains.
- Evaluation setup: The method is evaluated on five representative datasets using precision-recall curves, F-measure, and MAE under the same training and validation protocol as prior work.The datasets are MSRA-B, ECSSD, HKU-IS, PASCALS, and SOD.
- Architecture ablations: 2.5 points of F-measure gain come from adding one side output and two convolutional layers per side output, while successive short-connection patterns add 0.8 and 0.6 points.Pattern 1 matches the enhanced HED structure; Pattern 2 adds 0.8 points and Pattern 3 adds another 0.6 points.
- Side-output settings: 1.5 points of F-measure are lost when one convolutional layer is removed from each side output, whereas increasing channels provides no additional gain.Reducing kernel size in deeper side outputs also slightly decreases F-measure.
- Backbones: A ResNet-101 backbone yields a further one-point average F-measure improvement on each dataset with the same training set.The backbone replacement uses five side outputs while keeping other settings unchanged.
- CRF refinement: The modulating factor reduces MAE by around 0.3 points but contributes little to F-measure improvement.It is introduced to give positive predictions more confidence and reduce wrong predictions.
4.4 Comparison with the State-of-the-art
Compared with recent CNN-based and classical methods, the proposed approach performs strongly across diverse datasets and difficult visual conditions. It achieves the best reported F-measure and MAE results in the comparison, with especially strong behavior on multi-object datasets.
- Visual comparison: The visual comparisons report correct salient-region highlighting and coherent boundaries across complex scenes, center-biased objects, varied object sizes, and low-contrast cases.The selected images are grouped by scene properties for comparison.
- PR curve: The proposed PR curve is especially strong in the upper-left region and maintains higher precision when recall approaches 1.The authors associate this with fewer false positives than competing methods.
- F-measure and MAE: The proposed approach achieves the best score on all datasets in terms of maximum F-measure and MAE.It improves maximum F-measure by 1 point on ECSSD and SOD, and reduces MAE by more than 1 point on MSRA-B and PASCALS.
- F-measure and MAE: 1 point of maximum F-measure improvement is reported on both ECSSD and SOD, where existing values were already close to ideal value 1.The paper describes this as a large margin in that setting.
- Difficult datasets: The method performs especially well on HKU-IS, PASCALS, and SOD, which contain many images with multiple salient objects.The authors state that the method can detect and segment the most salient object, whereas other methods often fail at one of these stages.
4.5 The Existence of Saliency
The paper extends the network to determine whether an input image contains any salient object, addressing scenes excluded by methods that assume saliency is always present. The added branch uses global average pooling and an MLP, while the system remains fast at inference.
- Existence prediction: The saliency-existence branch predicts whether an input image contains salient objects, addressing the assumption underlying most existing methods.It consists of global average pooling followed by a three-layer MLP trained with softmax loss.
- Training setup: The existence experiments use 5,000 background images and 5,000 MSRA10K images, while salient-object gradients are blocked for background images.The authors report that blocking those gradients is essential to avoid interference with the prediction maps.
- Visual analysis: The selected visual results are grouped by scene properties and separated by solid lines to compare processing across different image conditions.The caption emphasizes the capability to process different scenes for each approach.
- Scope: The authors expect more challenging datasets that better reflect real-world difficulties to be developed in the future.This marks a scope boundary in the current evaluation context.
- Efficiency: 0.08s is required to process a 300 × 400 image before CRF refinement, and total processing remains below 0.5s with CRF.The reported timing is substantially faster than the cited DCL baseline, which requires more than 1s per image.
5 DISCUSSION
The discussion analyzes failure modes, training-set effects, and benchmark-comparison fairness. It finds that dataset composition and annotation quality matter more than training-set size alone, while complex scenes remain difficult.
- Failure Case Analysis: Complex backgrounds and very low foreground–background contrast cause failures in which salient objects are missed or non-salient regions are falsely highlighted.These cases are identified as a major failure circumstance in the analysis.
- Failure Case Analysis: Transparent objects remain difficult to segment completely, although the model can detect some of their parts.The paper identifies transparent objects as a distinct failure category.
- Dataset Quality Measuring: Training on each dataset’s corresponding training set consistently gives that dataset’s best result, making direct comparisons across models trained on different datasets inappropriate.The paper attributes this pattern possibly to dataset-specific image characteristics and favored features.
- Dataset Quality Measuring: More training images do not necessarily improve performance: ECSSD training achieves the best SOD result despite using only 1,000 training images.The analysis also notes that dataset size and quality can undermine fair comparisons.
- Beyond Training on Individual Datasets: A composite training set can underperform an individual dataset when its combinations are inappropriate, with schemes 4 and 0 trading gains on some datasets for decreases on MSRA-B and DUT-OMRON.The comparison shows that broader composition does not guarantee uniformly better results.
- Beyond Training on Individual Datasets: The paper recommends Scheme 11 for fair comparison and calls for more challenging datasets with complex scenes and high consistency.It argues that existing datasets often produce closely clustered model performance.
6 CONCLUSION
The paper concludes that short connections improve deeply supervised saliency detection by combining object-level localization with boundary refinement. Together with a fully connected CRF, the architecture produces more accurate saliency maps across simple and difficult images.
- 6 CONCLUSION: Short connections between shallower and deeper side-output layers let each layer highlight entire salient objects while accurately locating their boundaries.The design replaces direct connections from loss layers to only the last layer of each stage.
- 6 CONCLUSION: A fully connected CRF corrects wrong predictions and further improves the spatial coherence of saliency maps.
- 6 CONCLUSION: The experiments report more accurate saliency maps and state-of-the-art performance across simple and difficult cases.