Source-linked AI summary
Multi-scale Interactive Network for Salient Object Detection
Youwei Pang, Xiaoqi Zhao, Lihe Zhang, Huchuan Lu
TL;DR
Variable object scale challenges salient object detection by limiting multi-level and multi-scale feature use and creating foreground-background imbalance. MINet integrates adjacent-level features with AIMs, extracts multi-scale information with SIMs, and uses CEL for spatially consistent predictions. On five datasets, it performs favorably against 23 state-of-the-art methods without post-processing.
Problem
Variable salient-object scales challenge feature representation and create foreground-background imbalance that weakens pixel-level supervision and spatial consistency.
Method
MINet integrates adjacent-level features with AIMs, extracts multi-scale information using SIMs, and applies CEL to improve consistency between foreground and background predictions.
Results
The method performs favorably against 23 state-of-the-art SOD approaches on five datasets without post-processing.
Takeaways & Limitations
MINet provides an effective and efficient network for salient object detection under scale variation, with a reported forward reasoning speed of 35 FPS on GPU.
Abstract
from arXiv · showhide
Deep-learning based salient object detection methods achieve great progress. However, the variable scale and unknown category of salient objects are great challenges all the time. These are closely related to the utilization of multi-level and multi-scale features. In this paper, we propose the aggregate interaction modules to integrate the features from adjacent levels, in which less noise is introduced because of only using small up-/down-sampling rates. To obtain more efficient multi-scale features from the integrated features, the self-interaction modules are embedded in each decoder unit. Besides, the class imbalance issue caused by the scale variation weakens the effect of the binary cross entropy loss and results in the spatial inconsistency of the predictions. Therefore, we exploit the consistency-enhanced loss to highlight the fore-/back-ground difference and preserve the intra-class consistency. Experimental results on five benchmark datasets demonstrate that the proposed method without any post-processing performs favorably against 23 state-of-the-art approaches. The source code will be publicly available at https://github.com/lartpang/MINet.
1. Introduction
Salient object detection remains challenged by objects with varying scales, which expose limitations in multi-level feature use and spatially coherent prediction. MINet addresses these issues with interactive feature modules and consistency-enhanced supervision, achieving strong benchmark performance without post-processing.
- Scale variation makes CNN-based methods struggle to segment salient objects consistently and accurately because repeated subsampling removes necessary detail.
- Existing approaches integrate encoder and decoder features or use multi-scale modules, but single-level representations and deep low-resolution features limit scale-specific detail.
- AIM collaboratively integrates adjacent-resolution features with smaller resolution differences, reducing fusion interference while using multi-level contextual information.
- SIM uses two interactive resolution branches within a convolutional block to learn multi-scale features and improve handling of size-varying objects.
- CEL preserves intra-class consistency and enlarges inter-class differences, uniformly highlighting salient regions without additional parameters.
- 35 FPS forward reasoning speed and comparisons with 23 state-of-the-art methods on five datasets support MINet’s effectiveness without post-processing.
2. Related Work
Related work addresses salient-object scale variation through multi-level fusion, multi-scale context extraction, and prediction refinement. MINet combines interactive multi-scale processing with consistency-enhanced supervision to handle scale-dependent representation and foreground-background imbalance.
- Early handcrafted-prior and multilayer-perceptron methods have limited generalization, effectiveness, computational efficiency, or feature-structure preservation.
- Existing encoder-decoder methods often use individual-resolution decoder features, which inadequately represent salient objects across varying scales.
- 2.2. Scale Variation: Scale variation changes object information across resolutions, while localized convolutions and subsampling make it difficult to process multiple scales effectively.
- 2.2. Scale Variation: Coarse multi-level fusion can introduce redundancy and noise, whereas repeated gating can attenuate information from other layers and limit learning.
- 2.2. Scale Variation: ASPP and PPM typically operate at deep, low-resolution levels, where reduced small-object information limits their ability to handle large scale variation.
- 2.2. Scale Variation: MINet uses interactively trained branches for multi-scale representation and CEL to address foreground-background imbalance without sensitivity to object size.
- 2.3. Spatial Coherence: CEL targets overall prediction quality to obtain more uniform saliency results while offering a tradeoff between effect and inference speed.
3. Proposed Method
MINet combines adjacent-level and intra-layer interactions in an FCN decoder, then uses consistency-enhanced supervision to address scale variation and spatial coherence in salient object detection.
- 3.1. Network Overview: MINet uses pretrained VGG-16 or ResNet-50 backbones to extract multi-level features for an FCN-based network.VGG-16 inputs are subsampled by 16, while ResNet-50 inputs are subsampled by 32.
- 3.2. Aggregate Interaction Module: AIMs integrate adjacent-layer features through adjusted resolutions, mutual branch interactions, fusion, and residual learning.Pooling, neighbor interpolation, convolution, element-wise addition, and residual learning support the integration while keeping other branches supplementary.
- 3.3. Self-Interaction Module: SIMs generate multi-scale representations from intra-layer features by interacting high- and low-resolution branches before fusion in the decoder.Residual connections and fusion units combine the SIM and residual paths, allowing adaptive handling of scale variation.
- 3.4. Consistency-Enhanced Loss: CEL supplements binary cross entropy by considering foreground-background imbalance and relationships among predicted and ground-truth regions.Its formulation uses true-positive, false-positive, and false-negative areas to emphasize foreground structure and reduce sensitivity to object scale.
- 3.4. Consistency-Enhanced Loss: CEL is differentiable, reaches its maximum when predicted and ground-truth foreground regions are disjoint, and imposes a global prediction constraint.Unlike binary cross entropy, its gradient depends on all pixels in the prediction and ground truth.
- 3.5. Experiments: Table 1 evaluates maximum, mean, and weighted F-measures, E-measure, S-measure, and MAE across five benchmark datasets.Higher values are better for the F-measures, E-measure, and S-measure, while lower values are better for MAE.
4. Experiments
Experiments evaluate MINet on five benchmark datasets using six metrics, visual comparisons, and ablations. MINet generally outperforms competing methods, while AIMs, SIMs, and CEL each improve performance.
- Datasets and metrics: MINet is evaluated on DUTS, DUT-OMRON, ECSSD, HKU-IS, and PASCAL-S using six measurements.The evaluation includes PR curves, F-measure, MAE, and S-measure among the reported criteria.
- Comparison with state-of-the-art: 8.11% and 7.30% average MAE improvements are reported over CPD with VGG-16 and EGNet with ResNet-50, respectively.The comparison covers 23 state-of-the-art saliency detection methods, including methods using post-processing.
- Comparison with state-of-the-art: MINet achieves the best PR and F-measure curve results on DUTS-TE, ECSSD, PASCAL-S, and HKU-IS, while remaining competitive on DUT-OMRON.The curves are shown in Figure 6 across the five datasets.
- Ablation study: AIMs and SIMs each significantly improve the baseline, and their combination further improves performance.SIMs outperform PPM and ASPP, with reported MAE improvements of 6.21% and 1.45%.
- Ablation study: 2.14%, 2.77%, and 8.26% additional improvements are reported in Favg, Fωβ, and MAE over the Amulet-like model.The comparison evaluates the fusion strategy against the Amulet-like strategy.
- Ablation study: 4.75%, 3.75%, and 13.16% improvements in Favg, Fωβ, and MAE are reported when CEL is added to AIMs and SIMs.Because Favg relates to spatial consistency, the study reports more uniformly highlighted salient regions with CEL.
5. Conclusion
MINet addresses scale variation in salient object detection through transformation-interaction-fusion. Its AIMs integrate adjacent-level features, SIMs extract multi-scale information, and CEL addresses foreground-background imbalance and spatial highlighting.
- 5. Conclusion: MINet uses AIMs for adjacent-level feature integration and SIMs for multi-scale information extraction in decoder features.Both modules interactively learn contextual knowledge from branches with different resolutions.
- 5. Conclusion: CEL is used to alleviate foreground-background imbalance and uniformly highlight salient object regions.The conclusion describes CEL as an assistant to the model's training objective.
- 5. Conclusion: Each proposed module achieves significant performance improvement, and experiments on five datasets show MINet outperforming 23 state-of-the-art methods.The reported evaluation uses different metrics and does not mention post-processing in this conclusion passage.