Source-linked AI summary
Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection
Shuhan Chen, Yun Fu
TL;DR
RGB-D salient object detection remains challenging in complex scenes, and RGB-depth fusion faces issues including modality incompatibility, redundancy, and variable depth quality. The paper uses a lightweight depth stream and progressively guided alternate refinement. Across seven benchmark datasets, it reports state-of-the-art performance, 71 FPS, and a 64.9 MB model.
Problem
RGB-D SOD remains challenging in complex scenes, while RGB-depth fusion faces modality incompatibility, redundancy, and variable depth quality.
Source: §1 · pp. 1–2Method
A lightweight depth stream learned from scratch supplies complementary features, while RGB and depth features are alternately refined through progressively guided residual blocks.
Source: §1 · pp. 2–3Results
Across seven benchmark datasets, the model significantly outperforms state-of-the-art approaches, with 71 FPS and a 64.9 MB model.
Source: §1 · p. 3Takeaways & Limitations
The authors report that alternate refinement alleviates mutual degradation, especially with low-quality depth, while progressive guidance refines missing parts and false detections.
Source: §5 · p. 14
Abstract
from arXiv · showhide
In this paper, we aim to develop an efficient and compact deep network for RGB-D salient object detection, where the depth image provides complementary information to boost performance in complex scenarios. Starting from a coarse initial prediction by a multi-scale residual block, we propose a progressively guided alternate refinement network to refine it. Instead of using ImageNet pre-trained backbone network, we first construct a lightweight depth stream by learning from scratch, which can extract complementary features more efficiently with less redundancy. Then, different from the existing fusion based methods, RGB and depth features are fed into proposed guided residual (GR) blocks alternately to reduce their mutual degradation. By assigning progressive guidance in the stacked GR blocks within each side-output, the false detection and missing parts can be well remedied. Extensive experiments on seven benchmark datasets demonstrate that our model outperforms existing state-of-the-art approaches by a large margin, and also shows superiority in efficiency (71 FPS) and model size (64.9 MB).
1 Introduction
RGB-D salient object detection faces challenging scenes and unresolved fusion issues involving modality incompatibility, redundancy, and variable depth quality. The paper proposes a lightweight depth stream and progressively guided alternate refinement, reporting strong benchmark performance alongside efficiency and compactness.
1 Introduction
- RGB-D SOD remains challenging in low-contrast scenes and when objects resemble their surroundings; fusion also faces modality incompatibility, redundancy, and variable depth quality.The paper notes that low-quality depth maps may be noisy for prediction.Source: §1 · pp. 1–2
- A lightweight depth stream learned from scratch extracts complementary high-level features with four convolutional layers, avoiding a pre-trained depth backbone.The paper describes this stream as more compact and efficient and without the incompatibility problem.Source: §1 · p. 2
- RGB and depth features enter guided residual blocks alternately, where prediction guidance refines both features and predictions progressively to remedy missing parts and false detections.The network starts from a coarse prediction and refines it from deep side-outputs toward shallow ones.Source: §1 · pp. 2–3
- Across seven benchmark datasets, the model significantly outperforms state-of-the-art approaches, with 71 FPS and a 64.9 MB model.The reported comparison also emphasizes efficiency and compactness.Source: §1 · p. 3
2 Related Work
Related work covers coarse-to-fine and top-down guidance strategies for RGB salient object detection, alongside early, middle, and late fusion approaches for RGB-D detection.
2.1 RGB Salient Object Detection
- RGB SOD work includes coarse-to-fine refinement; this paper distinguishes its approach by learning multiple residuals per side-output with progressive guidance.Prior work also uses deep predictions to guide shallow layers, while this paper uses deep prediction for both feature and prediction refinement.Source: §2.1 · pp. 3–4
- Top-down guidance uses deep semantic information to filter shallow-layer distractions, with prior methods applying it through concatenation, erasure, or feature weighting.The cited approaches include,,, and.Source: §2.1 · pp. 3–4
2.2 RGB-D Salient Object Detection
- RGB-D SOD methods include hand-crafted features and CNN-based early, middle, and late fusion; this paper instead learns a lightweight depth stream whose features are used separately for refinement.Recent middle-fusion methods include complementarity-aware fusion, depth refinement, and depth cues used to weight RGB features.Source: §2.2 · p. 4
3 The Proposed Network
The network starts with a coarse prediction, then refines it through alternating RGB and depth features, using a lightweight depth stream and progressively guided residual blocks. Its components combine multi-scale context with guidance that updates both features and predictions.
3.1 The Overall Architecture
- RGB and depth features refine an MSR-generated coarse prediction alternately, while a lightweight depth stream extracts complementary features without a pretrained backbone.The refinement proceeds from deep to shallow side-outputs, and the streams are kept separate to reduce mutual degradation.Source: §3.1 · pp. 4–5
3.2 Multi-Scale Residual Block
- Three parallel MSR branches use dilation rates 1, 2, and 3, recur to reduce parameters, and fuse to produce one initial prediction.The branches use different weights because scale-aware ground truth is difficult to obtain in salient-object detection.Source: §3.2 · pp. 5–6
3.3 Guided Residual Block
- The GR block concatenates the prediction map with split feature groups, then learns residual updates for both the feature and prediction.The refined feature and prediction are passed to the subsequent GR block.Source: §3.3 · pp. 6–7
3.4 Progressive Guidance
- Each side-output stacks three GR blocks with progressively stronger prediction guidance, refining coarse predictions and helping remedy missing parts and false detections.The paper describes stacked residual units as shortcuts between side-output predictions and ground truth, with strong side-output supervision also reducing noisy depth distraction.Source: §3.4 · pp. 7–8
3.5 Difference to Other Networks
- Unlike GGM, GR convolves across concatenated groups and varies group channels for progressive guidance; unlike RAS [5] and R3Net, it progressively learns dual residuals and applies predictions as guidance.The authors say these distinctions help remedy missing object parts and false detections in initial predictions.Source: §3.5 · pp. 8–9
4 Experimental Results
The experiments test the network on seven RGB-D benchmarks and examine its design choices, benchmark performance, visual predictions, and efficiency.
4.1 Experimental Setup
- Evaluation used seven benchmarks, training on 1,485 NJUD and 700 NLPR images with augmentation, and assessing PR, Fβ, Sα, Eξ, and mean absolute error (M).The training images were augmented fourfold by flipping and rotating them.Source: §4.1 · p. 9
4.2 Ablation Analyses
- Progressive guidance outperforms uniform guidance, and alternate refinement performs better than direct RGB-depth concatenation.The best-performing guidance setting, No. 6, was selected for the final model.Source: Table 2 · p. 10, §4.2 · p. 10, Table 3 · p. 10
- Adding the constructed depth stream improves performance over RGB alone; replacing it with VGG16 increases model size and training time while reducing quantitative performance.The VGG16 replacement took four hours to train.Source: Table 3 · p. 10, §4.2 · p. 11
- Five recurrent MSR iterations were selected because performance stabilizes beyond five, while recurrence matches seven stacked blocks’ performance with a more compact model.The recurrent-versus-stacked comparison used the No. 1 guidance setting.Source: §4.2 · p. 10
- Adding side-output depth features improves performance through conv2, while adding conv1 brings no further gain.The authors conclude that three side-output depth features suffice to capture complementary cues.Source: Table 4 · p. 11, §4.2 · p. 11
4.3 Comparison with State-of-the-arts
- 2.8% and 2.1% average gains in Sα and Fβ, with 1.0% lower M than DMRA, accompany significant wins across seven datasets except Eξ.The comparison table covers four metrics across seven datasets.Source: §4.3 · p. 11, Table 5 · p. 12
- PR curves show the best performance especially at high recall; visual examples show complete highlighting with fewer false detections across challenging scenarios.Examples include low contrast, low-quality depth maps, complex scenes, multiple small objects, and large objects.Source: §4.3 · pp. 12–13, Figure 5 · p. 13
- 71 FPS inference speed and a 64.9 MB model size accompany training in about 2.5 hours for 30 epochs on the augmented NJUD+NLPR set.The comparison table reports running speed and model size for recent models.Source: Table 6 · p. 13, §4.3 · p. 13
5 Conclusions
The conclusion presents a coarse-to-fine RGB-D network with alternate refinement and progressive guidance, reports results on seven benchmarks, and notes a remaining boundary-detail limitation.
5 Conclusions
- The network starts with an MSR prediction and alternately refines RGB and depth features in GR blocks, using progressive guidance to address missing parts and false predictions.The depth stream learns from scratch to extract complementary features.Source: §5 · pp. 13–14
- The authors report state-of-the-art performance on seven benchmarks and describe the network as applicable to other cross-modal SOD tasks such as RGB-T.Source: §5 · p. 14
- Boundary details remain insufficiently accurate, especially in high-resolution images.The authors identify this as future work.Source: §5 · p. 14