PaperlayerView exact revision on arXiv

Source-linked AI summary

Bilateral Attention Network for RGB-D Salient Object Detection

Zhao Zhang, Zheng Lin, Jun Xu, Wenda Jin, Shao-Ping Lu, Deng-Ping Fan

arXiv:2004.14582v1cs.CV

TL;DR

RGB-D SOD methods often focus on foreground regions while underusing useful background cues. BiANet introduces complementary foreground-first and background-first attention in a bilateral module, with multi-scale extension, and reports better benchmark performance and real-time speed.

  • Problem

    Existing RGB-D SOD methods mainly emphasize foreground regions, while jointly refining complementary foreground and background cues remains an open problem.

  • Method

    BiANet uses BAM to combine gradual foreground-first and background-first attention, and extends BAM with multi-scale techniques for richer RGB-D refinement.

  • Results

    BiANet outperforms previous state-of-the-art RGB-D SOD methods on six benchmark datasets in quantitative and qualitative evaluations.

  • Takeaways & Limitations

    BiANet provides a fast RGB-D SOD solution that jointly exploits foreground and background information and runs at 34fps∼80fps under different settings.

Abstract

from arXiv · show

Most existing RGB-D salient object detection (SOD) methods focus on the foreground region when utilizing the depth images. However, the background also provides important information in traditional SOD methods for promising performance. To better explore salient information in both foreground and background regions, this paper proposes a Bilateral Attention Network (BiANet) for the RGB-D SOD task. Specifically, we introduce a Bilateral Attention Module (BAM) with a complementary attention mechanism: foreground-first (FF) attention and background-first (BF) attention. The FF attention focuses on the foreground region with a gradual refinement style, while the BF one recovers potentially useful salient information in the background region. Benefitted from the proposed BAM module, our BiANet can capture more meaningful foreground and background cues, and shift more attention to refining the uncertain details between foreground and background regions. Additionally, we extend our BAM by leveraging the multi-scale techniques for better SOD performance. Extensive experiments on six benchmark datasets demonstrate that our BiANet outperforms other state-of-the-art RGB-D SOD methods in terms of objective metrics and subjective visual comparison. Our BiANet can run up to 80fps on $224\times224$ RGB-D images, with an NVIDIA GeForce RTX 2080Ti GPU. Comprehensive ablation studies also validate our contributions.

I. INTRODUCTION

RGB-D SOD methods often emphasize foreground cues from depth while underusing complementary background information. BiANet addresses this gap with bilateral attention that jointly refines foreground, background, and uncertain boundary details.

  • Motivation: RGB-D SOD methods commonly exploit depth to emphasize foreground regions, while complementary background information remains underused.Background understanding can also promote SOD performance.
  • Approach: BiANet uses two RGB-D streams and concatenates their side outputs across multiple stages to learn cross-modal features.The architecture extracts multi-level features from RGB and depth streams before progressive prediction refinement.
  • Approach: BAM combines foreground-first attention, which gradually refines foreground cues, with background-first attention, which recovers potentially salient information in background regions.The complementary attention maps are used jointly to infer residuals for refining up-sampled saliency maps.
  • Results: BiANet achieves better performance than state-of-the-art methods on six RGB-D SOD datasets under nine standard metrics, with improved details and sharp edges.The paper also reports better visual effects than competing methods.
  • Results: 34fps∼80fps is the reported BiANet speed range on an NVIDIA GeForce RTX2080Ti GPU under different settings.The authors describe the network as feasible for real-world applications.

II. RELATED WORK

Related work shows that RGB-D SOD has progressed from low-level priors to deep cross-modal representations, yet jointly refining foreground and background cues remains an open problem.

  • RGB-D SOD: Early RGB-D SOD methods relied on low-level cues such as object distance, contrast, graph knowledge, and background enclosure.These methods often produced inaccurate predictions because they lacked high-level feature representation.
  • RGB-D SOD: Deep RGB-D SOD methods separately extract RGB and depth features, then fuse them at shallow, middle, deep, or multiple network stages.Multi-level fusion improves on one-off integration of cross-modal features.
  • Foreground and Background Cues: Traditional approaches jointly reason from complementary foreground and background information, whereas many current RGB-D networks emphasize foreground regions.Understanding the background can promote SOD performance.
  • Foreground and Background Cues: Bilateral-attention visualizations show foreground-first features emphasizing foreground areas and background-first features mining potentially significant objects in background regions.Both branches prioritize uncertain areas caused by up-sampling before fusion.
  • Foreground and Background Cues: Reverse-attention work gradually explores saliency from the background but ignores the contribution of foreground cues to final detection.The paper identifies joint foreground-background refinement as an open problem in deep RGB-D SOD.

III. PROPOSED BIANET FOR RGB-D SOD

BiANet extracts and fuses RGB-D features through two streams, then progressively refines coarse predictions using top-down up-sampling and bilateral attention residual compensation.

  • Bilateral attention residual compensation: BAM receives a higher-level prediction and the current-level feature to distinguish uncertain regions as foreground or background during top-down refinement.The method uses bilateral attention residual compensation after prediction up-sampling.
  • Feature extracting: BiANet extracts multi-level features from separate RGB and depth streams using VGG-16 convolutional blocks.Each stream includes five VGG-16 convolutional blocks and an additional three-layer convolution group for saliency prediction.
  • Feature extracting: The network concatenates RGB and depth side outputs at multiple stages instead of performing one-off cross-modal fusion.This produces feature tensors F_i for subsequent prediction and refinement.
  • Prediction up-sampling: High-level feature F6 produces a coarse saliency map S6, while lower-level features F1 through F5 support refinement of up-sampled predictions.Max-pooling contributes to constructing F6.

2) Prediction up-sampling:

BiANet progressively refines a coarse, low-resolution saliency prediction through higher-resolution features and bilateral attention. Deep supervision trains the RGB, depth, and top-down side outputs with binary cross-entropy.

  • Prediction up-sampling: The coarse high-level map S6 initializes foreground and background locations, while lower-level features refine it through residual prediction.The refined map is formed by adding a predicted residual to the up-sampled higher-level prediction.
  • Prediction up-sampling: BAM uses foreground-first and reversed background-first attention maps to distinguish foreground and background during residual refinement.The resulting residual emphasizes object boundaries, suppresses background regions, and strengthens foreground regions.
  • Loss function: Deep supervision is applied to the depth output, RGB output, and each top-down side output to clarify optimization goals and accelerate convergence.The total loss uses binary cross-entropy with weight coefficients set to 1 in the experiments.

B. Bilateral Attention Module (BAM)

The Bilateral Attention Module refines predictions by jointly exploiting complementary foreground and background cues. It weights cross-modal side-output features in two branches and predicts a residual focused on uncertain regions.

  • Bilateral Attention Module (BAM): BAM addresses refinement of an initial foreground-background prediction using higher-resolution cross-modal features.The module is motivated by the different distributions of foreground and background regions.
  • Bilateral Attention Module (BAM): Foreground-first attention comes from the sigmoid-activated up-sampled higher-level prediction, while background-first attention is its complement relative to an all-ones matrix.The two attention maps provide reversed views of the predicted regions.
  • Bilateral Attention Module (BAM): The FF and BF maps weight side-output features in separate branches, which jointly predict a residual component for refining the saliency map.The branches use separate parameters and concatenate their extracted features before residual prediction.
  • Bilateral Attention Module (BAM): FF features emphasize predicted foreground cues, whereas BF features search background regions for potentially salient objects.Both branches shift attention toward uncertain areas such as object boundaries during top-down refinement.
  • Evaluation: The PR curves compare BiANet with 14 state-of-the-art methods across six datasets, with each curve node marking the precision-recall pair used for max F-measure.The figure provides a visual comparison of precision-recall behavior and the max F-measure operating point.

C. Multi-Scale Extension of BAM (MBAM)

MBAM extends BAM with multi-scale context extraction for uncertain foreground and background regions. Applying it to high-level cross-modal features improves global object representation while adding computational cost.

  • Multi-Scale Extension of BAM (MBAM): MBAM uses groups of dilated convolutions to extract pyramid representations from undetermined foreground and background areas.This multi-scale context is intended to accommodate salient objects with varied location, size, and shape.
  • Multi-Scale Extension of BAM (MBAM): The dilated-convolution groups use rates 3, 5, and 7, while the initial 1×1-kernel layer has 32 channels and a ReLU activation.The dilated kernels are 3×3 with 32 channels and ReLU layers.
  • Multi-Scale Extension of BAM (MBAM): MBAM is recommended for high-level cross-modal features {F3,F4,F5}, where different receptive-field sizes help explore multi-scale context.The number of MBAM modules should balance detection performance against computational cost.
  • Multi-Scale Extension of BAM (MBAM): Using three MBAMs in the top three levels captures more complete high-level information than applying BAMs at all levels, supporting object localization.Without effective multi-scale perception, BAM can miss accurate global salient regions and produce incomplete predictions at high levels.

D. Implementation Details

BiANet is implemented with a VGG-16 ImageNet-pretrained backbone and evaluated on six RGB-D SOD datasets. It supports real-time inference while being compared quantitatively with deep-learning and traditional methods.

  • Implementation Details: The implementation applies MBAM to high-level side outputs {F3,F4,F5} and uses bilinear interpolation for all interpolation operations.The backbone is initialized from VGG-16 pretrained on ImageNet, and the network is implemented in PyTorch.
  • Implementation Details: Training uses 1485 NJU2K and 700 NLPR image pairs, Adam optimization, learning rate 0.0001, batch size 8, and 25 epochs.Training and testing images are resized to 224 × 224, with outputs restored to original size for evaluation.
  • Efficiency: BiANet runs at 34fps∼80fps on an NVIDIA GeForce RTX2080Ti GPU under different settings, supporting real-world application use.The reported speed depends on the configuration, including the number of MBAM modules.
  • Evaluation datasets: Evaluation covers six datasets, including NJU2K, NLPR, DES, STERE, SSD, and SIP, with dataset scale and acquisition conditions varying substantially.SSD contains 400 high-resolution images at 960×1080, while SIP contains 929 person images.
  • Quantitative evaluation: Table I compares BiANet with nine deep-learning methods and five traditional methods using S-measure, multiple F-measures, E-measures, and MAE.Higher values are better for the listed similarity, F-measure, and E-measure metrics, whereas lower MAE is better.

2) Evaluation metrics:

The evaluation uses nine standard metrics, compares BiANet with traditional and DNN-based RGB-D SOD methods, and includes visual and ablation comparisons of architectural mechanisms.

  • Evaluation metrics: Nine metrics evaluate precision-recall behavior, F-measure, pixel-wise error, structural similarity, and enhanced alignment.The metrics include max Fβ, mean Fβ, adaptive Fβ, MAE, S-measure, and E-measure.
  • Visual comparison: Figure 6 compares BiANet with the top five methods on scenes involving tiny objects, cluttered backgrounds, complex textures, low contrast, difficult depth, and multiple objects.The figure organizes these challenging conditions across different input columns.
  • Comparison methods: The comparison includes five traditional methods and nine DNN-based RGB-D SOD methods.The listed methods include ACSD, LBE, DCMC, MDSF, SE, DF, AFNet, CTMF, MMCI, PCF, TANet, CPFP, DMRA, and D3Net.
  • Ablation comparison: Figure 7 visualizes ablations combining deep information, foreground-first attention, background-first attention, and multi-scale extension.The configurations range from a baseline and depth-only model to the full Dep + FF + BF + ME combination.
  • Ablation comparison: Table II defines the ablation candidates and applies the multi-scale extension to the top three feature levels.Tables III and IV separately assess BAM and MBAM improvements at individual side outputs.

2) Quantitative evaluation:

BiANet is evaluated quantitatively and visually against existing RGB-D SOD methods across multiple challenging datasets and operating conditions. It reports strong benchmark performance, detailed-scene robustness, and practical speed-cost analyses.

  • Quantitative results: On NJU2K and NLPR, BiANet outperforms the second-best method by approximately 3% on max Fβ.The comparison covers large-scaled datasets.
  • Quantitative results: On DES, BiANet improves max Fβ by 3.8% over methods heavily dependent on depth information.The authors interpret this as more efficient use of depth information.
  • Quantitative results: BiANet performs best on SIP and exceeds D3Net on SSD despite SSD’s poor-quality depth maps.SIP contains complex scenes and multiple objects, while D3Net was designed for robustness to low-quality depth maps.
  • Qualitative results: Visual comparisons show BiANet detecting tiny or texture-similar objects, fine object structures, and multiple objects under confusing depth conditions.Examples include shoes and hats, chair legs, sculptures, windows, and large low-contrast objects.
  • Backbone analysis: Table VI compares BiANet performance across VGG-11, VGG-16, ResNet-50, and Res2Net-50 backbones.The table evaluates how backbone choice affects the proposed network.

C. Ablation Study

The ablation studies examine bilateral attention, feature-level placement, multi-scale extension, and their combinations on NJU2K and STERE. Results support complementary foreground and background modeling, with stronger gains from lower-level BAM placement and multi-scale extension.

  • Ablation scope: The ablation section evaluates bilateral attention benefits, BAM and MBAM placement, their combination, and backbone impact.These investigations cover both module effectiveness and architectural choices.
  • Effectiveness of bilateral attention: Ablations on NJU2K and STERE test depth information, foreground-first attention, background-first attention, and multi-scale extension.The baseline uses a VGG-16 backbone and residual refinement structure with RGB input only.
  • Effectiveness of bilateral attention: Foreground-first attention improves foreground detection but can mistake background objects or miss complete foreground objects without background cues.The cited example is a red house mistaken as salient in the absence of sufficient background understanding.
  • BAM placement: BAM at every tested feature layer improves detection performance, with lower-level placement contributing more to the results.Each experiment applies BAM to one side output while the other outputs use general convolutions.
  • Multi-scale extension: Multi-scale extension on the higher three levels {F3,F4,F5} improves the model’s performance over the corresponding non-extended configuration.Applying MBAM at different feature levels yields different degrees of improvement.

4) Cooperation between BAM and MBAM:

BiANet’s multi-scale extension improves performance most when applied to higher-level BAMs, while lower-level expansion adds cost without clear benefit. Backbone choice and adverse depth or RGB conditions define important practical trade-offs and failure boundaries.

  • Cooperation between BAM and MBAM: Expanding BAM into MBAM from higher to lower levels steadily improves detection, but lower-level additions provide little benefit while increasing computation.The authors recommend prioritizing higher-level MBAMs and balancing accuracy against speed.
  • Cooperation between BAM and MBAM: The lightweight BiANet reaches approximately 80fps, whereas higher-accuracy settings require balancing multi-scale expansion against computational cost.The paper reports that MBAM selection should reflect application speed requirements.
  • Performances under different backbones: BiANet remains state of the art with a lighter VGG-11 backbone, while stronger backbones produce larger improvements.With ResNet-50, it improves MAE on NJU2K by 1.5% versus D3Net; with Res2Net-50, max F-measure improves by 3.8% versus state-of-the-art methods.
  • Failure Case Analysis: BiANet can fail when depth identifies a non-salient nearby object or when coarse depth maps and confusing RGB information obscure the target.These conditions cause incorrect foreground selection, inaccurate spatial information, or failure to locate the target object.
Loading 2004.14582v1…