Source-linked AI summary
RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation
Jindong Jiang, Lunan Zheng, Fei Luo, Zhijun Zhang
TL;DR
RedNet addresses indoor RGB-D semantic segmentation with a residual encoder-decoder that fuses RGB and depth features and uses skip connections and pyramid supervision. On SUN RGB-D, the proposed architecture achieves state-of-the-art performance, including 47.8% accuracy with a ResNet-50 encoder.
Problem
Indoor semantic segmentation is a central computer-vision task for enabling visual scene understanding in indoor spaces.
Method
RedNet combines residual modules in the encoder and decoder with encoder-decoder skip connections, depth-feature fusion, and pyramid supervision across decoder layers.
Results
47.8% accuracy is achieved by RedNet with a ResNet-50 encoder on the SUN RGB-D testing set, outperforming most existing methods.
Takeaways & Limitations
RedNet with pyramid supervision achieves a state-of-the-art result on the SUN RGB-D dataset for indoor RGB-D semantic segmentation.
Abstract
from arXiv · showhide
Indoor semantic segmentation has always been a difficult task in computer vision. In this paper, we propose an RGB-D residual encoder-decoder architecture, named RedNet, for indoor RGB-D semantic segmentation. In RedNet, the residual module is applied to both the encoder and decoder as the basic building block, and the skip-connection is used to bypass the spatial feature between the encoder and decoder. In order to incorporate the depth information of the scene, a fusion structure is constructed, which makes inference on RGB image and depth image separately, and fuses their features over several layers. In order to efficiently optimize the network's parameters, we propose a `pyramid supervision' training scheme, which applies supervised learning over different layers in the decoder, to cope with the problem of gradients vanishing. Experiment results show that the proposed RedNet(ResNet-50) achieves a state-of-the-art mIoU accuracy of 47.8% on the SUN RGB-D benchmark dataset.
1 Introduction
Indoor semantic segmentation is important for visual scene understanding by service robots but remains difficult because indoor scenes contain challenging visual conditions. RedNet addresses this task with a residual RGB-D encoder-decoder, skip connections, depth fusion, and pyramid supervision.
- Motivation: Indoor semantic segmentation supports visual scene understanding needed by service robots operating in indoor spaces.
- Background: Encoder-decoder architectures recover spatial information through skip connections after spatial information is lost in the encoder.
- Motivation: High training memory consumption in dilated architectures limits their ability to use deeper structures.Dilated networks retain spatial resolution, requiring activation maps to remain stored for gradient computation.
- RedNet: RedNet uses residual modules in both encoder and decoder paths for indoor RGB-D semantic segmentation.The architecture also uses skip connections, depth fusion, and pyramid supervision for optimization.
- RedNet: Pyramid supervision applies supervised learning over different decoder layers to optimize the network.
2 Related Work
Prior work developed residual encoder-decoder architectures and depth-aware methods for semantic segmentation. RedNet combines residual processing on both downsample and upsample paths with depth fusion and skip connections for indoor RGB-D prediction.
- Residual Networks: Residual learning addresses model degradation as networks become deeper by fitting residual mappings and merging them with identity inputs through shortcut connections.Residual networks were also described as avoiding vanishing gradients through short paths between input and output.
- Residual Segmentation Networks: Earlier semantic-segmentation architectures placed residual modules on skip connections or throughout encoder and decoder paths.
- Encoder-Decoder Architectures: LinkNet used ResNet18 as an encoder and bottleneck units in the decoder for efficient feature upsampling.
- Indoor RGB-D Segmentation: Indoor semantic segmentation remains challenging because objects can have similar colors and structures under non-uniform illumination.
- Indoor RGB-D Segmentation: Depth information improves recognition of object classes with similar depth appearance and location in indoor scenes.
- RedNet: RedNet fuses depth features in the downsample path and uses skip connections to bypass fused information to the decoder for full-resolution prediction.
3 Approach
RedNet uses a residual encoder-decoder with separate RGB and depth branches, feature fusion, and skip connections to combine semantic and spatial information. Its decoder uses upsample residual units and pyramid supervision across multiple outputs.
- RedNet Architecture: The encoder has parallel RGB and depth branches with matching configurations, except the depth branch accepts a one-channel input.Depth features are fused into the RGB branch over five layers using element-wise summation.
- RedNet Architecture: RedNet uses residual layers throughout the encoder and decoder, with skip connections carrying spatial information between them.The architecture applies residual modules as its basic building block and bypasses spatial features from encoder to decoder.
- RedNet Architecture: The decoder uses four upsample residual layers followed by a final 2 × 2 transpose-convolution output layer.The proposed upsample residual unit increases feature-map resolution by a factor of 2, while the decoder uses standard residual blocks with two 3 × 3 convolutions.
- RedNet Architecture: Agent layers project ResNet-50 encoder features to fewer channels, reducing decoder memory consumption.These single 1 × 1 convolutions are used only with ResNet-50 because ResNet-34 lacks channel expansion.
- Pyramid Supervision: Pyramid supervision applies losses to four decoder side outputs and the final full-resolution output to address gradient vanishing.Each side output is produced by a 1 × 1 convolution, and the overall loss sums five cross-entropy losses with greater weight on downsampled outputs.
4 Experiment
RedNet is evaluated on SUN RGB-D using ResNet-34 and ResNet-50 encoders, standardized training, and three segmentation metrics. Comparisons show strong performance, while pyramid supervision improves all evaluated criteria.
- Experimental setup: RedNet is evaluated with ResNet-34 and ResNet-50 encoders on the SUN RGB-D indoor scene understanding benchmark.The dataset contains 10,335 densely annotated RGB-D images from 20 scenes.
- Experimental setup: The experiments use identical training strategies and hyperparameters for the two RedNet variants, with ImageNet-pretrained encoders and momentum SGD.Training also applies class reweighting, learning-rate decay, and weight decay.
- Evaluation: RedNet is evaluated on 5,050 SUN RGB-D test instances using pixel accuracy, mean accuracy, and intersection-over-union.These are the three reported segmentation criteria.
- Comparison results: 47.8% accuracy is achieved by RedNet(ResNet-50), despite competing residual architectures using ResNet-152 feature extractors.RedNet(ResNet-34) and RedNet(ResNet-50) outperform most existing methods, and the deeper encoder performs better with the same decoder.
- Ablation results: Pyramid supervision improves performance on all three criteria, with supervised RedNet(ResNet-34) outperforming unsupervised RedNet(ResNet-50).The comparison is reported as an ablation of pyramid supervision.
5 Conclusion
The conclusion presents RedNet as an RGB-D residual encoder-decoder for indoor semantic segmentation, combining residual and encoder-decoder skip connections with depth fusion and pyramid supervision. Comparative experiments report a state-of-the-art result on SUN RGB-D.
- Architecture: RedNet combines residual units, encoder-decoder skip connections, and encoder fusion of depth information for RGB-D semantic inference.The architecture uses short skip-connections within residual units and long skip-connections between encoder and decoder.
- Training: Pyramid supervision applies supervised learning across decoder layers to improve the encoder-decoder network's performance.The conclusion describes this as supervision over several decoder layers.
- Conclusion: Comparative experiments show that RedNet with pyramid supervision achieves a state-of-the-art result on the SUN RGB-D dataset.