Multimodal Deep Learning for Robust RGB-D Object Recognition

Andreas Eitel; Jost Tobias Springenberg; Luciano Spinello; Martin Riedmiller; Wolfram Burgard

Source-linked AI summary

Multimodal Deep Learning for Robust RGB-D Object Recognition

Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, Wolfram Burgard

arXiv:1507.06821v2Jul 24, 2015cs.CVcs.LGcs.NEcs.RO

TL;DR

RGB-D object recognition must remain reliable despite imperfect sensor data in robotics. The paper proposes a two-stream CNN with late fusion, depth encoding, staged training, and noise-aware augmentation, achieving state-of-the-art RGB-D Object results and improved recognition in noisy real-world settings.

Problem
RGB-D object recognition in robotics must handle imperfect sensor data, while large labeled depth datasets needed to train depth CNNs are unavailable.
Method
A two-stream CNN processes RGB and depth separately, combines them through late fusion, and uses depth colorization, stage-wise training, and realistic missing-data augmentation.
Results
The method achieves 91.3 ± 1.4% overall accuracy on the RGB-D Object dataset and improves recognition accuracy in challenging noisy RGB-D Scenes settings.
Takeaways & Limitations
The approach supports robust RGB-D object recognition by learning features from both modalities and training depth processing against realistic real-world noise.

Abstract

from arXiv · show

Robust object recognition is a crucial ingredient of many, if not all, real-world robotics applications. This paper leverages recent progress on Convolutional Neural Networks (CNNs) and proposes a novel RGB-D architecture for object recognition. Our architecture is composed of two separate CNN processing streams - one for each modality - which are consecutively combined with a late fusion network. We focus on learning with imperfect sensor data, a typical problem in real-world robotics tasks. For accurate learning, we introduce a multi-stage training methodology and two crucial ingredients for handling depth data with CNNs. The first, an effective encoding of depth information for CNNs that enables learning without the need for large depth datasets. The second, a data augmentation scheme for robust learning with depth images by corrupting them with realistic noise patterns. We present state-of-the-art results on the RGB-D object dataset and show recognition in challenging RGB-D real-world noisy settings.

I. INTRODUCTION

The paper proposes a two-stream CNN for RGB-D recognition that targets imperfect sensor data through depth encoding, realistic noise augmentation, and staged training. It reports state-of-the-art recognition on the RGB-D Object dataset and improved accuracy in noisy real-world settings.

RGB-D recognition combines RGB appearance and texture with depth-based shape information that is invariant to lighting or color variations.
The architecture uses separate CNN streams for color and depth, followed by joint fine-tuning with a fusion network before classification.
Depth images are encoded as rendered RGB images, enabling reuse of ImageNet-pretrained CNNs despite limited labeled depth data.
Realistic missing-data patterns augment depth training examples to improve robustness against occlusions and sensor noise.
The method outperforms the state of the art on the RGB-D Object dataset and improves recognition accuracy in noisy RGB-D Scenes settings.

II. RELATED WORK

The related work spans CNN-based recognition, hand-designed and learned RGB-D features, and depth-aware detection using pretrained RGB networks. The paper differs through its depth colorization and late-fusion architecture.

Earlier RGB-D recognition methods often rely on hand-designed SIFT and shape features, while newer approaches use unsupervised feature learning.
Prior deep-learning work primarily targeted 2D images, with later efforts incorporating depth for scene labeling and object detection.
HHA encoding represents height, horizontal disparity, and surface-normal angle in three channels for CNN processing of depth data.
The proposed method differs from related work in both depth-to-color encoding and the fusion approach used to combine RGB and depth information.
The architecture processes RGB and depth independently before combining the streams through late fusion.

A. Input preprocessing

Input preprocessing adapts RGB-D data to ImageNet-compatible CNN inputs while preserving object shape and making depth resemble three-channel image data. The proposed jet colorization is computationally inexpensive and outperforms HHA for recognition.

ImageNet-pretrained CaffeNet expects 227 × 227 RGB inputs, motivating preprocessing of both RGB and depth modalities.
The proposed resizing method scales the longest image side and tiles border pixels instead of warping the object to a fixed aspect ratio.
Jet colorization normalizes depth to 0–255 and maps near-to-far distances from red through green to blue across three channels.
Colorizing depth provides common structure between depth and RGB images, allowing CNNs designed for RGB inputs to learn suitable depth representations.
The computationally inexpensive depth-to-color encoding outperforms HHA for object recognition.

B. Network training

The multimodal CNN is trained in three stages using labeled RGB images, preprocessed depth images, and one-hot image labels. Individual streams are trained first, followed by joint fine-tuning.

The training dataset contains labeled RGB images, preprocessed depth images, and one-hot image labels with dimensionality equal to the number of classes.
Training first fits the RGB and depth stream networks individually, then jointly fine-tunes the combined architecture.

1) Training the stream networks:

The RGB and depth streams are initialized from ImageNet-pretrained CaffeNet parameters and trained separately with modality-specific softmax classifiers before independent classification.

Each stream extracts a modality-specific representation and initializes its parameters from an ImageNet-trained CaffeNet.
The depth stream adds a randomly initialized softmax layer on its extracted representation and minimizes the training data’s negative log likelihood.
The RGB stream is trained by applying the analogous softmax optimization procedure.
After training, the two networks can separately classify RGB and depth inputs.

2) Training the fusion network:

After separate stream training, the method concatenates the fine-tuned RGB and depth representations and trains an additional fusion network, with optional joint optimization of all parameters.

The fusion network concatenates the RGB and depth streams’ fine-tuned last-layer responses before classification.
The resulting multimodal representation is processed by an additional fusion stream ending in a softmax classification layer.
The training setup is illustrated with RGB-D scene examples containing noise, occlusions, and artificially induced missing-depth patterns.
Fusion training can jointly optimize the stream and fusion parameters, or keep the individually trained stream weights fixed.

C. Robust classification from depth images

The depth-robustness method models characteristic real-world missing-depth patterns and randomly applies them during training to improve recognition under noisy sensor conditions.

Depth sensors produce missing information near object boundaries and surfaces because material reflectivity and coating affect measurements.
The method samples representative binary noise masks from indoor Kinect recordings and combines patches from groups with different missing-reading counts.
During depth-network training, each depth sample is randomly replaced with a noised variant with probability 50%.
The augmentation objective uses a Hadamard product together with Bernoulli and discrete uniform sampling distributions.

IV. EXPERIMENTS

Experiments evaluate multimodal recognition on household-object and real-world scene datasets, using Caffe-based CNNs and standard image augmentation; the fusion model outperforms prior approaches on the RGB-D dataset.

The Washington RGB-D Object Dataset contains household objects from 51 classes, while RGB-D Scenes provides an additional noisy real-world robustness setting.
The experiments use CaffeNet-based CNN components with convolutional, pooling, fully connected, and softmax classification layers.
The multimodal CNN outperforms all previous approaches in recognition accuracy on the reported RGB-D dataset comparison.
The training procedure includes random 227 × 227 crops from 256 × 256 inputs and random horizontal flipping.

B. RGB-D Object dataset

On the RGB-D Object dataset, the multimodal CNN was evaluated across ten cross-validation splits and achieved 91.3 ± 1.4% overall accuracy using RGB and depth. Per-class recall was high for many classes, though mushrooms and peaches were the weakest.

The Washington RGB-D Object Dataset contains 41,877 RGB-D images across 51 household-object classes and 300 instances.Evaluation uses ten cross-validation splits, with roughly 35,000 training and 7,000 testing images per split.
91.3 ± 1.4% overall accuracy was achieved by the best multimodal CNN using RGB and depth.The RGB-only and depth-only accuracies were 84.1 ± 2.7% and 83.8 ± 2.7%, respectively.
The multimodal result was reported as the highest accuracy on this dataset at the time.The depth colorization fusion slightly outperformed the more computationally intensive HHA fusion, which did not improve performance.
Approximately half of the object classes achieved per-class recall of ≈99%.Mushrooms and peaches had the worst class recall.

C. Depth domain adaptation for RGB-D Scenes

The RGB-D Scenes experiments tested whether depth augmentation improves recognition under noisy real-world conditions. They compared a baseline depth-only network with a domain-adapted network on six object classes.

The RGB-D Scenes dataset contains six overlapping object classes and many noisy depth images from real-world scenes.
Two depth-only networks were trained on the Object dataset and tested on Scenes using groundtruth bounding boxes.The baseline followed the standard training procedure, while the second network used the proposed depth augmentation.
The domain-adapted depth network was compared with the baseline using six-class recognition results reported in percent.
Standard image warping performed poorly, supporting the authors’ view that shape information is lost during preprocessing.

V. CONCLUSION

The paper concludes that its two-stream multimodal CNN combines RGB and depth information for object recognition while adapting depth processing to limited labeled data. It reports state-of-the-art RGB-D Object performance and improved recognition in noisy real-world scenes.

The proposed architecture uses two convolutional streams that automatically fuse RGB and depth information before classification.
Depth-to-image encoding enables large ImageNet-pretrained CNNs to be leveraged for depth recognition.
The method achieves state-of-the-art performance on the RGB-D Object dataset.
Noise-aware depth augmentation improves recognition accuracy on the RGB-D Scenes dataset.The conclusion connects this improvement to noisy real-world robotics environments.