PaperlayerView exact revision on arXiv

Source-linked AI summary

EDN: Salient Object Detection via Extremely-Downsampled Network

Yu-Huan Wu, Yun Liu, Le Zhang, Ming-Ming Cheng, Bo Ren

arXiv:2012.13093v3cs.CV

TL;DR

SOD methods have largely focused on low-level feature learning, while high-level features for salient-object localization remain less investigated. EDN learns a global image view through extreme downsampling and uses SCPC for multi-level feature fusion and detail recovery, achieving reported state-of-the-art real-time performance; EDN-Lite reaches 316fps.

  • Problem

    Most SOD efforts improve low-level feature fusion or boundaries, while high-level feature learning for salient-object localization remains less investigated.

  • Method

    EDN stacks an Extremely-Downsampled Block on the backbone to learn a global image view and uses Scale-Correlated Pyramid Convolution for decoder feature fusion.

  • Results

    EDN achieves state-of-the-art performance with real-time speed, while EDN-Lite achieves competitive performance at 316fps.

  • Takeaways & Limitations

    The paper presents high-level feature enhancement through extreme downsampling as a new direction for salient object detection.

Abstract

from arXiv · show

Recent progress on salient object detection (SOD) mainly benefits from multi-scale learning, where the high-level and low-level features collaborate in locating salient objects and discovering fine details, respectively. However, most efforts are devoted to low-level feature learning by fusing multi-scale features or enhancing boundary representations. High-level features, which although have long proven effective for many other tasks, yet have been barely studied for SOD. In this paper, we tap into this gap and show that enhancing high- level features is essential for SOD as well. To this end, we introduce an Extremely-Downsampled Network (EDN), which employs an extreme downsampling technique to effectively learn a global view of the whole image, leading to accurate salient object localization. To accomplish better multi-level feature fusion, we construct the Scale-Correlated Pyramid Convolution (SCPC) to build an elegant decoder for recovering object details from the above extreme downsampling. Extensive experiments demonstrate that EDN achieves state-of-the-art performance with real-time speed. Our efficient EDN-Lite also achieves competitive performance with a speed of 316fps. Hence, this work is expected to spark some new thinking in SOD. Code is available at https://github.com/yuhuan-wu/EDN.

I. INTRODUCTION

SOD remains challenging in complicated scenarios, while recent multi-scale methods have emphasized low-level details and boundaries over high-level feature learning. EDN addresses this gap with extreme downsampling for global salient-object localization and SCPC-based multi-level feature fusion for detail recovery.

  • SOD detects salient objects or regions for applications including visual tracking, scene classification, image retrieval, and weakly supervised learning.
  • Recent SOD methods use high-level semantics for localization and low-level representations for object details and boundaries through multi-scale learning.
  • Existing SOD research mainly improves low-level features through boundary representations, boundary supervision, and multi-level feature fusion.
  • High-level feature learning is less investigated, although SOD requires an overall image understanding to locate salient objects before recovering details with a decoder.
  • EDB gradually downsamples feature maps to a 1 × 1 feature vector while continuing deep feature learning, producing a global view with tiny computational overhead.
  • SCPC correlates multiple decoder branches and scales to integrate multi-level features and recover complete salient objects from extreme downsampling.

II. RELATED WORK

Related SOD methods evolved from handcrafted priors to CNN-based image-to-image prediction, with multi-level fusion and boundary modeling dominating recent progress. EDN instead strengthens high-level representations through extreme downsampling and uses SCPC for decoder fusion.

  • SOD research includes handcrafted features, shallow learning, and heuristic priors such as color contrast, center, and background priors.
  • CNN-based SOD became dominant, progressing from region classification that discards spatial layout to end-to-end image-to-image saliency prediction.
  • Multi-level fusion architectures combine high-level semantics with low-level details, commonly drawing on HED, Hypercolumns, or U-Net designs.
  • Boundary-aware methods directly use boundary information or supervision to improve saliency accuracy at object boundaries.
  • Existing high-level feature methods often adapt semantic-segmentation modules such as ASPP and PSP, but their mismatch with SOD limits salient-object localization accuracy.
  • EDN stacks EDB above a backbone for global-view learning and uses SCPC for multi-level decoder integration; its downsampling halves feature-map resolution except for GAP.

1) High-level Feature Learning:

EDN enhances high-level feature learning by extremely downsampling feature maps to obtain a global image view for salient-object localization, then uses top-down integration to recover fine details.

  • 1) High-level Feature Learning:: EDB progressively downsamples features to a 1 × 1 vector, learning increasingly global representations of the whole image.The resulting global view is intended to improve salient-object localization.
  • 1) High-level Feature Learning:: EDB is stacked on top of VGG16, producing an output feature map with scale 1/32.The paper argues that extreme downsampling benefits SOD by learning a global image view.
  • 1) High-level Feature Learning:: After EDB, a five-stage decoder uses two SCPCs per stage for top-down multi-level integration and fine-detail saliency prediction.The decoder fuses multi-level features and produces outputs D1 through D6.
  • 1) High-level Feature Learning:: The EDB downsamples an input feature map by factors of 2, repeats the block, and applies global average pooling to obtain a feature vector.The intermediate small-scale feature has a very large receptive field before global pooling.
  • 1) High-level Feature Learning:: The sigmoid-compressed global representation recalibrates intermediate features through element-wise multiplication and is also used to enhance earlier features.The global representation is replicated to match feature-map dimensions before multiplication.

C. Scale-Correlated Pyramid Convolution

SCPC performs scale-correlated multi-scale feature extraction rather than using separate branches, combining atrous convolutions, concatenation, and residual connections for multi-level fusion.

  • C. Scale-Correlated Pyramid Convolution: SCPC correlates feature extraction across scales, addressing the separate-branch design used by modules such as ASPP and PSP.Its motivation is that feature extraction at different scales should be correlated.
  • C. Scale-Correlated Pyramid Convolution: M1 is evenly split into four feature maps along the channel dimension before scale-correlated multi-scale processing.The split creates the inputs for subsequent scale-specific operations.
  • C. Scale-Correlated Pyramid Convolution: The scale-correlated pathway combines features across neighboring scales using atrous convolutions with scale-specific atrous rates.Small-scale features help fill the holes of large-scale features with larger atrous rates.
  • C. Scale-Correlated Pyramid Convolution: SCPC concatenates the multi-scale features and adds a residual connection, producing output O = H(M).Its connections resemble a basic Res2Net block, while atrous convolutions strengthen multi-scale representation learning.

D. Loss Function

EDN is optimized with a combined binary cross-entropy and Dice loss, using predicted and ground-truth saliency maps to train the network.

  • D. Loss Function: The EDN loss combines binary cross-entropy and Dice loss: L(P, G) = Lbce + Ldice.The combined loss is used to optimize the proposed EDN.
  • D. Loss Function: Dice loss uses the predicted and ground-truth saliency maps and is described as effective for alleviating foreground-background class imbalance.P denotes the prediction and G denotes the ground truth.
  • D. Loss Function: During testing, the upsampled prediction P1 is used as EDN’s final output, while D6 is excluded from the loss because of its small size.The prediction is upsampled to the input-image size.

IV. EFFECT OF EXTREME DOWNSAMPLING

The section examines extreme downsampling as a complementary direction to low-level multi-scale learning, focusing on high-level features and global salient-object views. EDB improves localization most strongly in center regions while also helping boundary and other regions.

  • Extreme downsampling explores high-level feature learning by developing a global view of the whole image for SOD.This complements existing efforts focused mainly on low-level fine-grained features.
  • The foreground saliency map is divided into boundary, center, and other regions using Euclidean-distance criteria.Boundaries lie within 5 pixels of background; center regions are the foreground pixels in the top 20% of distance from background.
  • Adding EDB improves center-region MAE relatively more than boundary and other-region MAE, indicating stronger gains in salient-object localization.MAE is computed separately while ignoring the other two region types.
  • EDB also improves boundaries, potentially because stronger high-level features make decoding easier and improve low-level feature utilization.Visualizations show that EDB helps detect salient objects that are otherwise lost completely or partially.

V. EXPERIMENTS

The experiments evaluate EDN and EDN-Lite across five SOD datasets using standard accuracy metrics and comparisons with recent methods. The setup also examines design alternatives, efficiency, and qualitative predictions.

  • A. Experimental Setup: EDN is evaluated on DUTS, ECSSD, HKU-IS, PASCAL-S, and DUT-OMRON using pixel-level labels.Training uses DUTS training data, with evaluation on DUTS-TE and the other four datasets.
  • A. Experimental Setup: The study includes quantitative comparisons, qualitative visualizations, and evaluations of alternative EDB and global-attention design choices.Tables II–VI and Figure 6 cover these comparisons.
  • A. Experimental Setup: Performance is measured with F-measure, weighted F-measure, MAE, S-measure, maximum E-measure, and mean E-measure.Higher F-measure, weighted F-measure, S-measure, and E-measure indicate better performance, while lower MAE is better.

B. Comparison with State-of-the-art Methods

EDN is compared with recent state-of-the-art SOD methods across datasets, metrics, speed, and qualitative segmentation quality. EDN generally achieves the strongest performance while retaining real-time speed, and EDN-Lite trades some capacity for higher efficiency.

  • B. Comparison with State-of-the-art Methods: EDN consistently achieves the best performance in most cases and remains close to the best in the remaining cases.The comparisons cover the three largest datasets: DUTS-TE, DUT-OMRON, and HKU-IS.
  • B. Comparison with State-of-the-art Methods: EDN provides real-time speed with a relatively small number of parameters.Speed and parameter counts are reported alongside comparisons with regular SOD methods.
  • B. Comparison with State-of-the-art Methods: EDN-Lite achieves competitive performance with recent state-of-the-art methods at 10× average speed.CSNet retains a large performance gap compared with recent state-of-the-art methods in the reported comparison.
  • B. Comparison with State-of-the-art Methods: In difficult scenarios, EDN segments salient objects with clear boundaries when other competitors miss whole objects or parts of them.The qualitative comparison is presented in Figure 6.

C. Ablation Study

The ablations show that EDN’s extreme downsampling, global attention, and SCPC decoder each contribute to performance, while the hybrid loss preserves the primary SOD metric.

  • Effect of various design choices for EDB: EDB with global attention and extreme downsampling substantially outperforms variants without either component and the baseline.Removing either downsampling or global attention significantly affects performance.
  • Comparison of EDB with other alternatives: EDB outperforms ASPP, PSP, Non-local, DenseASPP, and the baseline by a large margin.The alternatives achieve only slightly better or even worse performance than the baseline.
  • Choices of global attention: Channel-wise element-wise multiplication performs better than spatial attention and matrix multiplication for global attention.This strategy is therefore used as the default global-attention design.
  • Atrous rate configurations of SCPC: SCPC is robust to different atrous-rate settings, with only slight result fluctuations; the seventh setting is selected as default.EDN uses seven SCPC modules whose atrous rates correspond to feature-map sizes.
  • Comparing SCPC with other alternatives: SCPC improves substantially over ASPP and vanilla convolution in feature fusion.The comparison uses five datasets, and SCPC further improves the performance obtained with ASPP.
  • Discussion of the loss function: The hybrid BCE and Dice loss is retained because Dice loss improves F_wβ and MAE but decreases Fβ, the primary SOD metric.The loss-function comparison evaluates BCE-only, Dice-only, and the hybrid design.

D. Discussions about Failure Cases

EDN still fails in several confusing scenarios, despite improving quantitative and qualitative SOD performance overall.

  • Failure cases: EDN may predict wrong salient regions, select the largest rather than most discriminative salient object, or treat discriminative lanes as non-salient.These representative failure cases are shown in Fig. 7.

VI. CONCLUSION

The paper addresses the limited study of high-level feature learning in SOD with EDB and SCPC, positioning EDN as a baseline for further work.

  • VI. CONCLUSION: EDB learns a better global view of the whole image to localize salient objects, while SCPC recovers object details after extreme downsampling.The conclusion frames these components as the paper’s response to the gap in high-level feature learning.
  • VI. CONCLUSION: The work is presented as a strong SOD baseline that may encourage further efforts toward enhancing high-level features.The stated scope is future development of salient object detection.
Loading 2012.13093v3…