Source-linked AI summary
F3Net: Fusion, Feedback and Focus for Salient Object Detection
Jun Wei, Shuhui Wang, Qingming Huang
TL;DR
Salient object detectors often combine multi-level features despite mismatched distributions and treat all pixels equally, limiting attention to difficult details. F3Net addresses these issues with selective cross-feature fusion, cascaded feedback refinement, and pixel position aware loss. It reports superior performance across five datasets and six evaluation metrics.
Problem
Multi-level features have different characteristics, while binary cross entropy treats pixels equally despite boundaries and error-prone regions needing more attention.
Method
F3Net combines a cross feature module for selective fusion, a cascaded feedback decoder for iterative refinement, and pixel position aware loss for unequal pixel weighting.
Results
F3Net outperforms state-of-the-art methods on five datasets under six evaluation metrics.
Takeaways & Limitations
F3Net produces accurate salient-object regions with clear local details in challenging scenarios.
Abstract
from arXiv · showhide
Most of existing salient object detection models have achieved great progress by aggregating multi-level features extracted from convolutional neural networks. However, because of the different receptive fields of different convolutional layers, there exists big differences between features generated by these layers. Common feature fusion strategies (addition or concatenation) ignore these differences and may cause suboptimal solutions. In this paper, we propose the F3Net to solve above problem, which mainly consists of cross feature module (CFM) and cascaded feedback decoder (CFD) trained by minimizing a new pixel position aware loss (PPA). Specifically, CFM aims to selectively aggregate multi-level features. Different from addition and concatenation, CFM adaptively selects complementary components from input features before fusion, which can effectively avoid introducing too much redundant information that may destroy the original features. Besides, CFD adopts a multi-stage feedback mechanism, where features closed to supervision will be introduced to the output of previous layers to supplement them and eliminate the differences between features. These refined features will go through multiple similar iterations before generating the final saliency maps. Furthermore, different from binary cross entropy, the proposed PPA loss doesn't treat pixels equally, which can synthesize the local structure information of a pixel to guide the network to focus more on local details. Hard pixels from boundaries or error-prone parts will be given more attention to emphasize their importance. F3Net is able to segment salient object regions accurately and provide clear local details. Comprehensive experiments on five benchmark datasets demonstrate that F3Net outperforms state-of-the-art approaches on six evaluation metrics.
Introduction
F3Net addresses mismatches among multi-level features and unequal pixel difficulty in salient object detection with selective fusion, feedback refinement, and position-aware weighting. Experiments report state-of-the-art performance across five datasets and six metrics.
- Multi-level features differ in distribution: high-level features provide semantics but coarse localization, while low-level features preserve details but contain background noise.
- F3Net’s cross feature module selectively fuses levels by suppressing redundant information and complementing missing parts.The module uses element-wise multiplication rather than simple addition or concatenation.
- The cascaded feedback decoder iteratively refines multi-level features by feeding aggregated features back into previous layers.Its sub-decoders combine bottom-up aggregation with top-down feedback.
- Pixel position aware loss assigns different weights to positions, emphasizing structure and detail regions instead of treating all pixels equally.This targets difficult pixels, including boundaries and error-prone areas.
- F3Net achieves state-of-the-art performance on five datasets across six evaluation metrics.
Related Work
Related SOD methods progressed from handcrafted cues to CNN-based multi-level feature aggregation, but feature discrepancy and hard-pixel handling remain insufficiently addressed. F3Net is designed around these two unresolved problems.
- Early SOD methods used color, texture, and center-prior cues, emphasizing low-level information while overlooking contextual semantics.
- CNN-based approaches extract and aggregate multi-level features, with prior work also exploring short connections, iterative refinement, context attention, and attention-guided selection.
- The discrepancy between feature levels remained insufficiently studied, motivating more effective fusion strategies for accurate SOD.
- Hard pixels beyond boundaries can receive increased loss weights to improve discriminating ability.
Proposed Method
F3Net combines selective cross-level feature fusion, cascaded feedback refinement, and pixel-position-aware supervision to improve salient object detection. Its components address feature discrepancies, iterative feature restoration, and the need to emphasize difficult local structures.
- Cross Feature Module: CFM selectively integrates multi-level features by extracting shared components through element-wise multiplication before refinement.It combines the fused representation with the original features rather than directly adding or concatenating heterogeneous inputs.
- Cross Feature Module: CFM uses two symmetric branches so low-level details enrich high-level features while high-level semantics suppress low-level background noise.Each branch applies convolution, batch normalization, and ReLU before feature crossing and residual refinement.
- Cascaded Feedback Decoder: CFD iteratively refines multi-level features through multiple sub-decoders that alternate bottom-up aggregation with top-down feedback.Each decoder produces a supervised coarse saliency map, then feeds downsampled aggregated features back into earlier representations for the next iteration.
- Cascaded Feedback Decoder: CFD uses ResNet-50 features from the last four of five levels to reduce computation while retaining the features used by its cascaded decoding process.The lowest-level feature is omitted because it adds computational cost with little performance improvement according to the cited design choice.
- Pixel Position Aware Loss: Unlike standard BCE and unweighted IoU, the proposed weighting addresses unequal pixel difficulty and the effects of background-dominated images.Multi-level supervision additionally supplies auxiliary losses for the sub-decoders to facilitate training.
- Pixel Position Aware Loss: PPA combines weighted BCE and weighted IoU restrictions, assigning larger weights to hard pixels based on differences from their surroundings.The weighting emphasizes boundaries, edges, holes, cluttered regions, and elongated structures while incorporating local and global structure information.
Experiments
F3Net is evaluated on five SOD datasets using six metrics, controlled ablations, and comparisons with state-of-the-art methods. It ranks first across the reported dataset-metric comparisons and produces clearer saliency maps in visual evaluations.
- Datasets and evaluation: F3Net is evaluated on five human-labeled SOD datasets using six metrics, including MAE, mF, Sα, Eξ, precision-recall curves, and F-measure curves.DUTS-TR trains F3Net, while the other listed datasets evaluate it.
- Ablation study: Ablation studies evaluate the effects of BCE, IoU, PPA, multi-level supervision, CFM, and CFD on DUTS.The experiments report that PPA performs best on three evaluation metrics and that adding the proposed modules boosts performance.
- Quantitative comparison: F3Net ranks first on all datasets and metrics in the reported comparison with 12 state-of-the-art methods.The comparison uses MAE, mF, Sα, and Eξ, with smaller MAE and larger values preferred for the other metrics.
- Quantitative comparison: F3Net consistently outperforms the compared models across precision-recall and F-measure curves on five datasets under different thresholds.The curves assess holistic performance across thresholds.
- Visual comparison: Visual comparisons show clearer and more accurate saliency maps that suppress background noise across cluttered backgrounds, small objects, reflections, and occlusions.The reported visual results require no post-processing.
Conclusion
F3Net combines selective feature fusion, iterative feedback refinement, and hard-pixel-aware optimization to improve salient object detection. Experiments show that it outperforms state-of-the-art methods under six evaluation metrics.
- Conclusion: F3Net selectively integrates features with CFM and iteratively refines multi-level features through CFD feedback mechanisms.CFM limits redundant-feature influence, while CFD targets finer details through repeated refinement.
- Conclusion: PPA loss emphasizes hard pixels and error-prone parts to guide attention toward local details.The loss is designed to focus optimization on pixels requiring greater attention.
- Conclusion: When γ = 5, the model achieves the best results in the corresponding hyperparameter comparison.The comparison evaluates the effect of γ in the PPA loss.
- Conclusion: When N = 2, the model achieves the best results in the sub-decoder-number comparison.N denotes the number of CFD sub-decoders.
- Conclusion: F3Net outperforms state-of-the-art methods under six evaluation metrics.The reported evaluation covers five datasets and six metrics.