Source-linked AI summary
ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation
Xinxin Hu, Kailun Yang, Lei Fei, Kaiwei Wang
TL;DR
Indoor semantic segmentation remains challenging because RGB and depth branches may contain unequal, scene-dependent information. ACNet selectively weights and combines their features through attention and parallel branches, outperforming state-of-the-art methods on NYUDv2 and SUN-RGBD, including 48.3% mIoU on NYUDv2.
Problem
Indoor semantic segmentation remains challenging, and RGB and depth features may be inappropriate because their information and distributions differ across scenes.
Method
ACNet uses separate RGB and depth ResNet branches, attention modules that weight their features by carried information, and a fusion branch processing merged features.
Results
ACNet outperforms state-of-the-art methods on NYUDv2 and SUN-RGBD, achieving 48.3% mIoU on the NYUDv2 test set and matching CFN on SUN-RGBD with ResNet-50.
Takeaways & Limitations
The architecture gathers complementary RGB and depth features while preserving the original branches’ inference and supplementing a fusion branch.
Takeaways & Limitations
The authors identify improving real-time performance as future work for pixel-wise image segmentation.
Abstract
from arXiv · showhide
Compared to RGB semantic segmentation, RGBD semantic segmentation can achieve better performance by taking depth information into consideration. However, it is still problematic for contemporary segmenters to effectively exploit RGBD information since the feature distributions of RGB and depth (D) images vary significantly in different scenes. In this paper, we propose an Attention Complementary Network (ACNet) that selectively gathers features from RGB and depth branches. The main contributions lie in the Attention Complementary Module (ACM) and the architecture with three parallel branches. More precisely, ACM is a channel attention-based module that extracts weighted features from RGB and depth branches. The architecture preserves the inference of the original RGB and depth branches, and enables the fusion branch at the same time. Based on the above structures, ACNet is capable of exploiting more high-quality features from different channels. We evaluate our model on SUN-RGBD and NYUDv2 datasets, and prove that our model outperforms state-of-the-art methods. In particular, a mIoU score of 48.3\% on NYUDv2 test set is achieved with ResNet50. We will release our source code based on PyTorch and the trained segmentation model at https://github.com/anheidelonghu/ACNet.
1. INTRODUCTION
Indoor semantic segmentation is challenging because scenes contain uneven illumination and spatial overlap, while RGB and depth features differ across scenes. ACNet addresses these issues by selectively combining modality features while preserving their original processing branches.
- Indoor semantic segmentation faces uneven illumination and messy spatial overlapping.
- RGBD observations add real-world geometric information that can improve segmentation over RGB alone.
- RGB and depth images are not equivalent within each scene, so current branches may extract inappropriate features.
- Existing RGBD architectures either insufficiently combine modalities or replace original RGB and depth branches with fusion branches.
- ACNet uses independent RGB and depth ResNet branches, ACMs, and a third fusion branch to combine features according to input-determined proportions.
2. FRAMEWORK
ACNet uses channel attention to weight RGB and depth features selectively, then combines them through a three-branch architecture. This preserves modality-specific feature flows while supplying weighted features to a fusion branch.
- Attention Complementary Module (ACM): ACM applies channel attention to selectively gather features from RGB and depth branches according to their information content.
- Attention Complementary Module (ACM): Global average pooling converts input feature maps A ∈ R^C×H×W into channel descriptors Z ∈ R^C×1×1.
- Attention Complementary Module (ACM): A 1 × 1 convolution models channel correlations, sigmoid activation constrains weights between 0 and 1, and an outer product produces weighted feature maps.
- Architecture for Feature Fusion: Two complete ResNets separately extract RGB and depth features, preserving both modality flows before upsampling.
- Attention Complementary Network (ACNet): ACNet reorganizes stage-wise RGB and depth maps with ACM, adds them element-wise into a fusion branch, and retains the original branches.
3. EXPERIMENTS
Experiments evaluate ACNet on NYUDv2 and SUN-RGBD using mIoU, visual and weight analyses of ACM, and ablations of its attention and multi-branch design. ACNet achieves state-of-the-art results while selectively balancing RGB and depth information across network levels.
- Experimental setup: The evaluation uses mean Intersection-over-Union across classes on NYUDv2 and SUN-RGBD test sets.NYUDv2 contains 1,449 RGBD images with 795 training and 654 testing images; SUN-RGBD V1 contains 10,335 images with 5,285 training and 5,050 testing images.
- Analysis of the ACM: ACM assigns higher weights to RGB features at lower levels when they contain more valid information, while depth receives higher weights where its features are more informative.At higher-level example locations, the branch with more informative feature maps receives the higher weight, and the weighted maps are added into the fusion branch.
- Analysis of the ACM: RGB weights exceed depth weights in Conv and Layer1, become nearly equal in Layer2 through Layer4, and are relatively low in Layer4.The reported weight statistics associate lower-level differences with RGB information and higher-level similarity with nearly equivalent RGB and depth information.
- Analysis of the ACM: The standard deviation of channel weights decreases from Conv to Layer3 but is high in Layer4, reflecting progressively homogenized distributions followed by stronger final feature selection.The paper describes Layer4 as selecting useful features and eliminating redundant ones before the encoder ends.
- Ablation Study: On NYUDv2, Model-1 and Model-2 obtain 44.3% and 46.8% mIoU, indicating accuracy boosts of 2.5% from the multi-branch architecture and 1.5% from ACM.Model-1 removes ACMs and post-Conv RGB and depth branches; Model-2 removes ACMs while retaining the multi-branch architecture.
- Comparison with state-of-the-art networks: ACNet reaches 48.3% mIoU on NYUDv2, outperforming other state-of-the-art models by 0.6%.On SUN-RGBD, ResNet-50 reaches the same mIoU as CFN with RefineNet-152 while using a more lightweight backbone.
4. CONCLUSIONS
ACNet is a multi-branch attention-based network for RGBD semantic segmentation that preserves separate RGB and depth processing while exploiting complementary features. Experiments on NYUDv2 and SUN-RGBD show that it outperforms state-of-the-art methods, while future work targets improved real-time performance.
- 4. CONCLUSIONS: ACNet uses multi-branch attention to selectively gather complementary RGB and depth features without destroying the original branches’ inference.The attention module weights features according to the information contained in each branch and supplements the fusion branch.
- 4. CONCLUSIONS: Experiments on NYUDv2 and SUN-RGBD show that ACNet outperforms state-of-the-art methods.The conclusion states this result across both evaluated datasets.
- 4. CONCLUSIONS: Future work will address real-time pixel-wise segmentation for RGBD semantic cognition and panoramic annular surrounding perception.The stated future direction focuses on improving real-time performance.