Source-linked AI summary
Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection
Youwei Pang, Lihe Zhang, Xiaoqi Zhao, Huchuan Lu
TL;DR
RGB-based salient object detection still faces severe challenges in cluttered or low-contrast scenes. HDFNet combines RGB-D features into region-aware dynamic filters and uses hybrid enhanced loss; it achieves best performance across six metrics on eight datasets, with 52 FPS inference and a 170 MB model.
Problem
Existing deep SOD methods are mainly based on RGB data and face severe challenges in cluttered or low-contrast scenes.
Method
HDFNet combines depth and RGB features to generate region-aware dynamic filters, while HEL supervises edges and fore-/background regions for sharper, more consistent predictions.
Results
The method achieves the best performance under six evaluation metrics on eight datasets, with 52 FPS inference on an NVIDIA GTX 1080 Ti and an approximately 170 MB model.
Takeaways & Limitations
The experiments support multi-scale cross-modal guidance and HEL as effective components for RGB-D SOD, with HEL also effective in RGB SOD.
Abstract
from arXiv · showhide
The main purpose of RGB-D salient object detection (SOD) is how to better integrate and utilize cross-modal fusion information. In this paper, we explore these issues from a new perspective. We integrate the features of different modalities through densely connected structures and use their mixed features to generate dynamic filters with receptive fields of different sizes. In the end, we implement a kind of more flexible and efficient multi-scale cross-modal feature processing, i.e. dynamic dilated pyramid module. In order to make the predictions have sharper edges and consistent saliency regions, we design a hybrid enhanced loss function to further optimize the results. This loss function is also validated to be effective in the single-modal RGB SOD task. In terms of six metrics, the proposed method outperforms the existing twelve methods on eight challenging benchmark datasets. A large number of experiments verify the effectiveness of the proposed module and loss function. Our code, model and results are available at \url{https://github.com/lartpang/HDFNet}.
1 Introduction
RGB-D SOD supplements RGB appearance cues with depth structure, but effective cross-modal fusion remains challenging. HDFNet addresses this with dynamic multi-scale filtering and hybrid loss optimization, achieving strong accuracy, speed, and compactness.
- Motivation: RGB-based SOD struggles in cluttered or low-contrast scenes, motivating depth-assisted RGB-D approaches.Depth provides spatial structure that complements RGB appearance and detail information.
- Motivation: Existing RGB-D methods commonly fuse modalities through addition, concatenation, convolution, or maps computed from depth.These approaches use fixed parameters for different test samples, motivating adaptive filtering.
- Method: The hybrid enhanced loss improves prediction details and salient-region consistency without additional parameters.It targets sharper boundaries and consistency between edge-adjacent and interior regions.
- Method: HDFNet combines RGB and depth features to generate region-aware dynamic filters that guide decoding in the RGB stream.The network uses a hierarchical dynamic filtering design for RGB-D SOD.
- Results: 52 FPS and approximately 170 MB characterize the VGG16-based model on an NVIDIA GTX 1080 Ti GPU.The model achieves this speed while the paper reports the best performance under six metrics across eight datasets.
2 Related Word
Prior RGB-D SOD work progressively combines, concatenates, predicts, or enhances features across RGB and depth streams. HDFNet instead uses multimodal information to generate multi-scale dynamic filters for cross-modal complementarity.
- RGB-D Salient Object Detection: RGB-D SOD methods evolved from handcrafted contrast and shape features to FCN-based cross-modal fusion.Reported fusion mechanisms include convolution, element-wise addition, concatenation, prediction fusion, and depth-derived enhancement.
- RGB-D Salient Object Detection: HDFNet uses RGB and depth inputs in a two-stream architecture whose encoders share the same network design.Examples include VGG-16, VGG-19, and ResNet-50.
- Dynamic Filters: Unlike prior dynamic filters, HDFNet uses multimodal information to generate multi-scale filters that strengthen cross-modal complementarity and suppress incompatibility.The paper contrasts this with a prior kernel computation that introduces many parameters and is difficult to extend across scales.
3 Proposed Method
The proposed method uses a two-stream RGB-D network with dense cross-modal transport and a dynamic dilated pyramid module for adaptive multi-scale decoding. A hybrid enhanced loss separately emphasizes object edges and foreground/background region consistency.
- Two Stream Structure: The network receives RGB and depth images through separate streams and extracts intermediate features at progressively smaller resolutions.Features f1–f5 are recorded from large to small; the method uses deeper features to balance efficiency and effectiveness.
- Two Stream Structure: Dense transport combines RGB and depth features to produce mixed representations containing spatial structure and appearance details for subsequent decoding.The mixed features are fed into DDPMs, whose outputs are merged in the decoder's top-down pathway.
- Dynamic Dilated Pyramid Module: DDPM generates adaptive kernels from mixed features and decoder features, using KGUs for local weight tensors and KTUs for dilated convolution kernels.KGUs cover 3 × 3, 7 × 7, or 11 × 11 neighborhoods; KTUs construct regular kernels with different dilation rates.
- Dynamic Dilated Pyramid Module: The DDPM's densely connected KGU layers strengthen feature propagation and reuse while improving parameter efficiency.Each of the four layers connects to all others in a feed-forward fashion.
- Hybrid Enhanced Loss: HEL combines BCE, edge-enhanced loss, and region-enhanced loss to optimize predictions at both edge and region levels.The edge term targets contours, while the region term separately optimizes foreground and background prediction errors.
4 Experiments
The experiments evaluate HDFNet on eight RGB-D SOD datasets using six metrics, compare it with twelve existing methods, and analyze its components through ablation and visual studies. HDFNet achieves strong benchmark performance while maintaining efficient inference and a compact model size.
- Datasets and metrics: HDFNet is evaluated on eight RGB-D SOD benchmark datasets using six metrics, including PR curves, F-measure variants, MAE, S-measure, and E-measure.The datasets include LFSD, NJUD, NLPR, RGBD135, SIP, SSD, STEREO, and DUTRGBD.
- Comparisons: HDFNet is compared with twelve RGB-D SOD models, with results reported across eight datasets and six metrics.AFNet and D3Net results on DUTRGBD are omitted because their codes and results were unavailable there.
- Quantitative comparisons: 2.02%, 2.85%, and 2.45% Fmax gains are reported over DMRA on DUTRGBD for VGG-16, VGG-19, and ResNet-50 models.The corresponding MAE improvements are 16.09%, 17.88%, and 13.56%.
5 Conclusions
The method combines RGB appearance and depth spatial information to generate adaptive multi-scale filters for cross-modal guidance, while edge-focused region supervision supports clearer predictions. Experiments on eight datasets and six metrics report state-of-the-art performance with a small model and high running speed.
- Contributions: RGB and depth features are combined to generate adaptive filters with different receptive field sizes for multi-scale cross-modal guidance.The dynamic dilated pyramid module uses mixed multi-modal features to enhance decoder representations.
- Contributions: Additional region-level supervision around object edges produces clearer predictions and improves consistency in salient and fore-/background regions.The conclusion attributes sharper prediction structure to supervision near edges.
- Results: Experiments on eight datasets and six metrics demonstrate component effectiveness and state-of-the-art performance.The approach is also described as having small model size and high running speed.
A Appendix
The appendix provides additional analyses of component contributions, performance comparisons, and the hybrid enhanced loss in RGB saliency models.
- Appendix contents: Weighted average results across six metrics are shown in Table 2, with dataset-specific results listed separately in Sections A.1 and A.2.The appendix separates aggregate evaluation from per-dataset reporting.
- Appendix contents: The appendix reports more detailed performance contributions from different components in HDFNet.These analyses supplement the main paper’s component evaluation.
- Appendix contents: It also presents detailed comparisons of RGB SOD models trained with and without the hybrid enhanced loss.The comparison examines the loss beyond the RGB-D setting.
A.1 Ablation Study
The ablation study separates the effects of the dynamic dilated pyramid module under different modality inputs and evaluates the individual and combined components of the hybrid enhanced loss.
- Experimental setup: Without DDPM, Models 2, 4, and 6 directly add dense-transport-layer features to the decoder instead of applying the DDPM convolution operation.This defines the non-DDPM comparison used in the ablation groups.
- DDPM ablations: Model 2 versus Model 3 evaluates DDPM dynamic filters computed from depth features alone.The comparison replaces direct feature addition with DDPM-based processing.
- DDPM ablations: Model 4 versus Model 5 evaluates DDPM dynamic filters computed from RGB features alone.This isolates RGB-only filter generation.
- DDPM ablations: Models 6–8 compare dynamic filters generated from two-modality features.This group tests the role of mixed RGB-depth information in DDPM.
- HEL ablations: Models 9–12 evaluate the three HEL components, Le, Lf, and Lb, alongside the overall hybrid enhanced loss.The ablation distinguishes individual loss terms from the combined formulation.
A.2 Effectiveness of the HEL
The appendix evaluates the hybrid enhanced loss in recent RGB saliency models, comparing versions with and without HEL under model-specific experimental settings.
- RGB SOD evaluation: Table 4 compares recent RGB SOD models trained with HEL against corresponding models without HEL.The best result in each comparison group is highlighted in red.
- Experimental settings: R3Net uses ResNeXt-101 and applies HEL only to the final prediction.The backbone matches the original paper’s setting.
- Experimental settings: CPD uses ResNet-50 and applies HEL to the prediction from each branch.This differs from the single-final-prediction supervision used for some other models.
- Experimental settings: PoolNet uses ResNet-50, omits joint edge training, and applies HEL to the final prediction.The appendix specifies these settings for the comparison.
- Experimental settings: GCPANet uses ResNet-50 and applies HEL to the final result at the input resolution.Its supervision setting is reported separately from the other RGB models.