PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation

Xiaokang Chen, Kwan-Yee Lin, Jingbo Wang, Wayne Wu, Chen Qian, Hongsheng Li, Gang Zeng

arXiv:2007.09183v1cs.CV

TL;DR

RGB-D segmentation must exploit complementary depth while handling modality differences and noisy measurements. The paper proposes a cross-modality guided encoder using SA-Gate and BMP for bidirectional recalibration, selective aggregation, and specificity preservation, and reports state-of-the-art performance across challenging indoor and outdoor datasets.

  • Problem

    RGB-D segmentation requires effective fusion of substantially different RGB and depth features, while existing methods often assume accurate, well-aligned depth despite measurement uncertainty.

  • Method

    The cross-modality guided encoder uses SA-Gate to filter and recalibrate both modalities before aggregation, and BMP to propagate fused features while preserving modality specificity.

  • Results

    The model achieves state-of-the-art performance on challenging indoor and outdoor datasets and improves decoder variants by up to 3.7% mIoU over naive RGB-D modifications.

  • Takeaways & Limitations

    The encoder can be injected as a plug-and-play module into existing RGB-D or RGB segmentation frameworks to improve multimodal representations and performance.

Abstract

from arXiv · show

Depth information has proven to be a useful cue in the semantic segmentation of RGB-D images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion to obtain better feature representations to achieve more accurate segmentation. This, however, may not lead to satisfactory results as actual depth data are generally noisy, which might worsen the accuracy as the networks go deeper. In this paper, we propose a unified and efficient Cross-modality Guided Encoder to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively. The key of the proposed architecture is a novel Separation-and-Aggregation Gating operation that jointly filters and recalibrates both representations before cross-modality aggregation. Meanwhile, a Bi-direction Multi-step Propagation strategy is introduced, on the one hand, to help to propagate and fuse information between the two modalities, and on the other hand, to preserve their specificity along the long-term propagation process. Besides, our proposed encoder can be easily injected into the previous encoder-decoder structures to boost their performance on RGB-D semantic segmentation. Our model outperforms state-of-the-arts consistently on both in-door and out-door challenging datasets. Code of this work is available at https://charlescxk.github.io/

1 Introduction

RGB-D segmentation uses depth to complement RGB with geometric information, but modality differences and noisy depth measurements make straightforward fusion unreliable. The paper introduces a bi-directional guided encoder with SA-Gate and BMP to recalibrate, fuse, and preserve modality-specific features, achieving state-of-the-art results indoors and outdoors.

  • Depth complements RGB by providing 3D geometry that is robust to illumination changes and helps distinguish objects.
  • Directly adding depth to RGB frameworks or averaging modality outputs can produce inferior performance because the modalities differ substantially and depth measurements are uncertain.
  • The cross-modality guided encoder suppresses exceptional depth responses, uses filtered depth to refine RGB features, and performs recalibration in both directions.
  • SA-Gate first recalibrates modality-specific features and then selectively aggregates cross-modality information, while BMP preserves modality specificity during propagation.
  • The encoder is designed as a plug-and-play module for RGB-D segmentation backbones and incorporates complementary information while reducing the influence of noisy depth measurements.
  • The method achieves state-of-the-art performance on challenging indoor and outdoor semantic segmentation datasets.

2 Related Work

RGB-D semantic segmentation methods use depth to augment RGB features, but noisy cross-modal signals challenge effective fusion. The proposed SA-Gate addresses this challenge by filtering and recalibrating both modalities before aggregation.

  • RGB-D semantic segmentation leverages depth as geometric augmentation for RGB semantic segmentation.
  • Existing methods commonly treat depth as an additional input to recalibrate RGB responses, using score averaging, LSTM fusion, or gated fusion.
  • Multi-modal distillation transfers valid messages from depth features to RGB features.
  • Outdoor RGB-D segmentation is more challenging because depth data contain inevitable noisy signals, unlike methods focused mainly on indoor settings.
  • SA-Gate suppresses exceptional noisy depth features, recalibrates RGB responses, and then fuses the cross-modal representations in a unified operation.

3 Method

The method uses a cross-modality guided encoder built around Separation-and-Aggregation Gating and bidirectional propagation to filter, recalibrate, and fuse RGB-HHA features while preserving modality-specific information.

  • Cross-modality guided encoder: The SA-Gate combines Feature Separation, which recalibrates each modality, with Feature Aggregation, which fuses cross-modality features.The encoder applies these operations before propagating multimodal features through subsequent stages.
  • Feature Separation: Feature Separation uses cross-modality statistics and RGB activations to suppress noisy depth responses and produce a filtered HHA representation.The filtered depth features then provide a less noisy counterpart for RGB recalibration.
  • Feature Separation: RGB recalibration sums the filtered HHA representation with RGB features, treating depth information as an offset rather than a direct multiplicative coefficient.The paper also applies recalibration symmetrically so filtered RGB features can refine the depth stream.
  • Feature Aggregation: Feature Aggregation generates spatial-wise soft gates for RGB and HHA features, then weights their contributions to form a merged feature representation.The gates are produced from concatenated modality features, mapped with 1×1 convolutions, and normalized with softmax.
  • Bi-directional Multi-step Propagation: Bidirectional propagation averages fused and original features for both streams and passes the refined modality-specific results to the next encoder layer.The process propagates information in both directions while retaining the specificity of each modality.
  • Segmentation decoder: The encoder is plug-and-play with encoder-decoder segmentation architectures, and the authors use DeepLabV3+ as the decoder because it performs best among tested choices.The first and last SA-Gate fusion outputs are propagated to the segmentation decoder.

4 Experiments

Experiments evaluate the method on indoor and outdoor RGB-D segmentation datasets, including efficiency, ablations, plug-and-play integration, qualitative behavior, and state-of-the-art comparisons.

  • Datasets and metrics: Experiments cover NYU Depth V2 and CityScapes using mIoU and pixel accuracy, with SUN-RGBD also evaluated.NYU Depth V2 has 40 classes; CityScapes has 19 fine-annotated classes, and no additional coarse annotations are used.
  • Efficiency analysis: The proposed method achieves better performance with significantly less memory and computational cost than the RGB-D baseline on NYU Depth V2.The baseline averages predictions from two parallel DeepLab V3+ networks.
  • Ablation studies: Feature Separation filters noisy features before the bi-directional recalibration step, while Feature Aggregation models spatial relationships to exploit modality complementarity.The ablations compare alternative separation designs and aggregation mechanisms under consistent experimental settings.
  • Encoder design: Embedding SA-Gate at earlier ResNet-50 stages performs better, while stacking stages yields diminishing additional gains.The experiments also find that SA-Gate and BMP improve performance individually and complement each other when combined.
  • Plug-and-play property: The encoder consistently improves RGB decoders over their original versions and exceeds naive RGB-D modifications by at least 1.5% mIoU.With the DeepLab V3+ decoder, the improvement reaches 3.7% mIoU.
  • State-of-the-art comparison: On NYU Depth V2, the model achieves 51.3% mIoU with ResNet-50; on CityScapes, it reaches 81.7% validation and 82.8% test mIoU.Qualitative visualizations show stronger RGB responses at boundaries and in high-contrast areas, while HHA responds well in glare and dark regions.

5 Conclusion

The paper concludes that its cross-modality guided encoder addresses unified representation and low-quality depth robustness while serving as a plug-and-play enhancement for existing segmentation frameworks.

  • Conclusion: The cross-modality guided encoder combines SA-Gate and BMP to address modality unification and robustness to low-quality depth sources.The encoder is designed to be injected into current state-of-the-art RGB semantic segmentation frameworks.
  • Conclusion: The proposed encoder can be easily integrated into existing encoder-decoder structures to improve RGB-D semantic segmentation performance.

1 Introduction

The supplementary material provides additional implementation details, experimental analyses, and qualitative results for the proposed approach.

  • Supplementary material: Supplementary material covers implementation details and additional experiments on NYU Depth V2, CityScapes validation, and SUN-RGBD.

2 Implementation Details

Experiments use PyTorch with specified optimization, cropping, training-duration, loss, augmentation, and inference settings for three datasets.

  • Common settings: All experiments use batch size 16, mini-batch SGD with momentum 0.9, weight decay 0.0005, and a poly learning-rate policy.
  • NYU Depth V2: NYU Depth V2 training uses 480 × 480 crops, 800 epochs, base learning rate 0.02, and cross-entropy loss.
  • SUN-RGBD: SUN-RGBD training uses 480 × 480 crops, 80 epochs, base learning rate 0.02, and cross-entropy loss.
  • CityScapes: CityScapes training uses 800 × 800 crops, 240 epochs, base learning rate 0.04, and OHEM loss.Training augmentation uses multiple scales, while state-of-the-art comparisons use flipping and multi-scale inference.

3 Experimental Results

Experiments on NYU Depth V2, CityScapes, and SUN-RGBD evaluate the method across indoor and outdoor RGB-D segmentation settings. The approach remains effective under noisy depth and improves the supplied baselines across the reported datasets.

  • Additional Evaluation: The method is evaluated with additional robustness and ablation studies on NYU Depth V2 and with supplementary experiments on CityScapes and SUN-RGBD.These experiments are intended to assess effectiveness, generalization, and robustness.
  • CityScapes: 1.2% mIoU improvement over the RGB-D and RGB baselines is reported on CityScapes despite seriously noisy depth measurements.The table reports CityScapes results in terms of mIoU and includes RGB-based methods for reference.
  • CityScapes: 2.6% and 4.6% mIoU improvements are reported over competing RGB-D methods using ResNet-101 and ResNet-50, respectively, on CityScapes.The comparisons are against method with ResNet-101 and method with ResNet-50.
  • SUN-RGBD: 3.4% mIoU improvement over the RGB baseline and 1.9% over the RGB-D baseline are reported on SUN-RGBD.The method outperforms most state-of-the-art methods but is slightly lower than PAP on this dataset.

4 Robustness to Noisy Signals Existing in the Input

The robustness study tests Gaussian noise added to the input HHA map on NYU Depth V2. The proposed model degrades less than the RGB-D baseline as noise increases.

  • Noise Protocol: Gaussian noise with mean 0 and standard deviations from 10 to 120 is added to the input HHA map for the robustness comparison.The HHA input ranges from 0 to 255, like the RGB image.
  • Results: At standard deviation 10, both models drop slightly, but the proposed model shows a smaller decrease than the baseline.The passage interprets this as greater robustness to small noise.
  • Results: At standard deviation 40, the baseline decreases by 44.6‰ while the proposed model decreases by 22.7‰.The reported comparison shows a smaller performance loss for the proposed model under larger Gaussian noise.

5 Does Filtering and Recalibration Help?

Feature visualizations examine depth features on CityScapes and RGB features on NYU Depth V2 before and after Feature Separation Part. The results show modality-specific noisy responses being suppressed or complemented.

  • Visualization Setup: Feature Separation Part visualizations compare primitive single-modality feature responses before and after filtering and recalibration on indoor and outdoor datasets.The first-layer embeddings are compressed to three dimensions with PCA for visualization.
  • Depth Features: On CityScapes, Feature Separation Part restores responses for a missing streetlight, sharpens a pole outline, and reveals objects absent from the HHA image.These examples concern depth-feature responses in outdoor scenes.
  • RGB Features: On NYU Depth V2, Feature Separation Part removes unnecessary texture, smooths the ground, and eliminates strong-lighting effects from RGB features.These examples concern RGB-feature refinement in indoor scenes.
  • Interpretation: Cross-modality filtering and recalibration supplement missing signals while suppressing unnecessary noisy feature responses.This is the stated conclusion of the visualization analysis.

6 Qualitative Results and Discussion

Qualitative results on CityScapes, NYU Depth V2, and SUN-RGBD illustrate how cross-modality propagation handles noisy depth, improves object boundaries, and compensates for labeling errors.

  • CityScapes: On CityScapes, the model identifies pole shapes using RGB information even when the corresponding HHA poles are indistinct.The qualitative examples are shown on the CityScapes validation set.
  • NYU Depth V2: On NYU Depth V2, the method correctly recognizes desk and chair regions, produces smoother boundaries, and recovers a carpet missed by the RGB-D baseline.The examples attribute these outcomes to combining RGB and HHA characteristics.
  • SUN-RGBD: On SUN-RGBD, the model is reported to handle details with satisfactory intra-class consistency and inter-class distinction.The qualitative examples are presented in Figure 9.
  • Ground-Truth Discussion: The SUN-RGBD examples contain missing or incorrect ground-truth labels, including omitted sofas and books, invalid regions, and a floor region mislabeled as chairs.The model recognizes several of these mislabeled or missing objects in the scene.
Loading 2007.09183v1…