Source-linked AI summary
Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Jinghua Wang, Zhenhua Wang, Dacheng Tao, Simon See, Gang Wang
TL;DR
RGB-D indoor semantic segmentation needs features that capture both shared structure and modality-specific cues without losing useful information during fusion. The paper introduces a multimodal deconvolutional architecture with a feature transformation network for common and specific features, achieving competitive results on NYU Depth V1 and V2.
Problem
Existing RGB-D fusion strategies can ignore modality correlations or lose RGB-specific texture features when modalities are concatenated.
Method
Separate convolutional and deconvolutional networks are bridged by a feature transformation network that learns common and modality-specific features, with MK-MMD measuring distribution similarity.
Results
The method achieves competitive performance on NYU Depth V1 and V2, outperforming prior state of the art by 6.3% on NYU V2 and all compared methods across 4-, 13-, and 40-class tasks.
Takeaways & Limitations
Common features let the two modalities borrow shared information from each other, while specific features preserve patterns visible in only one modality.
Takeaways & Limitations
The MK-MMD formulation assumes a Borel-measurable kernel with finite expected square-root diagonal value for the mean embedding to exist.
Abstract
from arXiv · showhide
In this paper, we tackle the problem of RGB-D semantic segmentation of indoor images. We take advantage of deconvolutional networks which can predict pixel-wise class labels, and develop a new structure for deconvolution of multiple modalities. We propose a novel feature transformation network to bridge the convolutional networks and deconvolutional networks. In the feature transformation network, we correlate the two modalities by discovering common features between them, as well as characterize each modality by discovering modality specific features. With the common features, we not only closely correlate the two modalities, but also allow them to borrow features from each other to enhance the representation of shared information. With specific features, we capture the visual patterns that are only visible in one modality. The proposed network achieves competitive segmentation accuracy on NYU depth dataset V1 and V2.
1 Introduction
Indoor RGB-D semantic segmentation is challenging because of factors such as poor illumination, occlusion, and random object distribution. The paper addresses limitations of multimodal feature learning by separating shared and modality-specific information within a deconvolutional architecture.
- Indoor semantic segmentation is difficult because object distributions are random and scenes suffer from poor illumination and occlusion.
- RGB-D cameras provide depth information independent of illumination, helping alleviate challenges in indoor semantic segmentation.
- Independent RGB and depth CNNs ignore modality correlations, while simple concatenation can lose RGB-specific texture features.
- The proposed architecture uses separate convolutional and deconvolutional networks for each modality, bridged by a feature transformation network.
- The feature transformation network disentangles common features from modality-specific features, allowing shared information and modality-only visual patterns to be represented.
- The paper introduces multimodal deconvolutional segmentation and lets modalities borrow common features to generate more robust deconvolutional features.
2 Related Work
Prior work spans multimodal feature learning, RGB-D indoor segmentation, and dense prediction architectures. Deconvolutional networks are presented as suitable for reconstructing dense, precise label maps, while earlier multimodal methods use alternative fusion and contextual modeling strategies.
- Multimodal learning research includes shared-feature autoencoders, domain-invariant representations, and methods that generate one modality from another.
- RGB-D indoor segmentation methods have used graphical models, kernel descriptors, superpixel Markov random fields, and convolutional networks.
- Fully convolutional networks produce input-sized label maps but can lose object detail and cannot handle varying semantic scales.
- Deconvolutional networks reconstruct label maps through deconvolutional and unpooling layers, making them suitable for dense and precise segmentation.
3 Approach
The approach uses paired RGB and depth convolutional–deconvolutional networks connected by a feature transformation network that separates common from modality-specific features. MK-MMD aligns common-feature distributions while distinguishing specific features, and cross-modal borrowing strengthens deconvolutional representations.
- RGB-D semantic segmentation takes an RGB image and corresponding depth map as input and predicts a pixel-wise semantic label map.
- Feature transformation: The feature transformation network decomposes each modality into common features and modality-specific features to represent shared and distinct visual information.Common features are encouraged to be similar across modalities, whereas specific features are encouraged to differ.
- Network structure: The network contains RGB and depth convolutional and deconvolutional networks plus a feature transformation network connecting convolutional to deconvolutional features.The convolutional networks use 14 convolutional layers, while the deconvolutional networks mirror their corresponding convolutional networks.
- Feature transformation: Each modality’s deconvolutional representation combines its specific features with both modalities’ common features, enabling cross-modal feature borrowing.The RGB and depth transformation layers respectively combine RGB-specific or depth-specific features with common features from both modalities.
- Deconvolution: Mirrored deconvolutional networks use pooling masks for unpooling and reverse convolution-like operations to reconstruct detailed label maps from sparse feature maps.Unpooling enlarges feature maps, while successive deconvolutional layers hierarchically reconstruct shape details.
- MK-MMD: MK-MMD compares common-feature distributions while encouraging modality-specific distributions to differ, avoiding reliance on identical individual common-feature vectors.The loss combines pixel-wise RGB and depth losses with common-feature similarity and specific-feature dissimilarity terms.
4 Training
Training proceeds in two stages, first learning single-object segmentation and then handling multiple neighboring objects from proposal-based patches. The implementation uses multiscale Gaussian-kernel MK-MMD and standard stochastic gradient descent with specified initialization and optimization settings.
- Two-stage training: Training first uses single-object patches to learn object-versus-surroundings segmentation, then proposal-based patches containing multiple objects to learn neighboring-object segmentation.
- MK-MMD configuration: MK-MMD uses 11 Gaussian kernels with bandwidths σ_u = 2^u−6 for u = 1,…,11, and its kernel weights are learned using the cited method.The four loss-balancing parameters are selected by cross-validation.
- Optimization: The convolutional networks are initialized from a VGG 16-layer model, while deconvolutional networks are initialized randomly and optimized with momentum SGD.The learning rate, weight decay, and momentum are 0.01, 0.0005, and 0.9, respectively.
- Optimization: The deconvolutional network is divided into five components trained sequentially using coarse outputs at each component.For example, the first component predicts a downsampled 30 by 40 label map.
5 Experiments
Experiments evaluate the proposed RGB-D deconvolutional network on NYU Depth V1 and V2 against independent, fused, concatenated-input, and distance-based baselines. The method produces strong segmentation results, with reported gains from modality correlation, common and specific feature learning, and MK-MMD-based feature comparison.
- Datasets and setup: Testing uses NYU Depth V1 and V2, including 13-class segmentation and V2 evaluations with 4, 13, and 40 classes.The datasets contain indoor RGB-D scenes grouped into 13 semantic categories for the main evaluations.
- Testing procedure: The testing pipeline segments 100 candidate patches or bounding boxes and combines their outputs by decision score fusion.Each patch is segmented independently before the final label map is produced.
- Qualitative results: The learned RGB-specific features emphasize texture-rich regions, while depth-specific features emphasize object edges and common features support the segmentation representation.Figure 3 compares these learned feature maps with FCN, C-DN, and the proposed method.
- Quantitative results: 6.3%: the proposed method outperforms prior state-of-the-art on NYU Depth V2 13-class segmentation.The method also outperforms the compared methods across the 4-class, 13-class, and 40-class V2 tasks.
- Ablation and feature comparison: 7.4% and 6.6%: the proposed method exceeds E-DN on NYU Depth V1 and V2, respectively, supporting MK-MMD over Euclidean distance for feature comparison.The paper attributes the difference partly to Euclidean distance being heavily affected by outliers.
- Ablation and fusion robustness: 2.3% and 5.1%: the proposed network exceeds B-DN on NYU Depth V1 and V2, while producing more similar modality-specific decision scores and greater fusion robustness.B-DN accuracy varies substantially with the linear fusion parameter, whereas the proposed network varies only slightly.
6 Conclusion
The paper proposes an RGB-D semantic segmentation network that transforms modality features into common and modality-specific representations, using cross-modal borrowing to improve deconvolutional features. It achieves competitive performance on NYU depth datasets V1 and V2.
- Network structure: The network uses convolutional and deconvolutional networks for each modality, connected through a feature transformation network.The transformation network converts convolutional features into common and modality-specific features.
- Feature transformation: MK-MMD measures similarity between feature distributions instead of comparing features one pair at a time.
- Feature transformation: Common features can be borrowed across modalities to learn robust deconvolutional features.This supports shared information between RGB and depth representations.
- Results: The method achieves competitive performance on NYU depth datasets V1 and V2.