Source-linked AI summary
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Saurabh Gupta, Ross Girshick, Pablo Arbeláez, Jitendra Malik
TL;DR
The paper addresses how to learn effective representations for object detection and segmentation from RGB-D images. It introduces geocentric depth features and an integrated detection-segmentation system, reporting improved object detection, instance segmentation, and semantic segmentation performance.
Problem
RGB-D perception needs richer representations than semantic pixel labels or bounding boxes alone to support object-level scene understanding.
Method
The system uses HHA depth encoding, RGB-D contours and 2.5D proposals, a modified R-CNN detector, foreground labeling for instance masks, and detection features for semantic segmentation.
Results
56% relative improvement in RGB-D object detection is reported over existing methods, alongside improvements over baselines for instance segmentation and state-of-the-art semantic segmentation results.
Takeaways & Limitations
Geocentric depth features and integrated RGB-D processing improve object detection while extending the system to instance and semantic segmentation.
Takeaways & Limitations
Direct comparison with Wang et al. is unavailable because their method uses a non-standard train-
Abstract
from arXiv · showhide
In this paper we study the problem of object detection for RGB-D images using semantically rich image and depth features. We propose a new geocentric embedding for depth images that encodes height above ground and angle with gravity for each pixel in addition to the horizontal disparity. We demonstrate that this geocentric embedding works better than using raw depth images for learning feature representations with convolutional neural networks. Our final object detection system achieves an average precision of 37.3%, which is a 56% relative improvement over existing methods. We then focus on the task of instance segmentation where we label pixels belonging to object instances found by our detector. For this task, we propose a decision forest approach that classifies pixels in the detection window as foreground or background using a family of unary and binary tests that query shape and geocentric pose features. Finally, we use the output from our object detectors in an existing superpixel classification framework for semantic scene segmentation and achieve a 24% relative improvement over current state-of-the-art for the object categories that we study. We believe advances such as those represented in this paper will facilitate the use of perception in fields like robotics.
1 Introduction
The paper presents an integrated RGB-D scene-understanding system that combines object detection with instance and semantic segmentation. It exploits depth-derived contours, region proposals, and learned depth features to improve scene interpretation.
- Integrated RGB-D scene understanding: The integrated system processes RGB-D images for object detection, instance masks, and pixel-level scene-surface labels.It generalizes the R-CNN architecture to use pixel-wise depth alongside RGB information.
- RGB-D contours and proposals: RGB-D contours and 2.5D region proposals provide the basis for classifying candidate object regions.The proposal module combines depth and color features within the MCG framework.
- RGB-D object detection: A three-channel depth representation—horizontal disparity, height above ground, and angle with gravity—produces stronger CNN features than disparity or depth alone.These features are used in a modified R-CNN framework for RGB-D object detection.
- RGB-D object detection: 56% relative improvement in RGB-D object detection is achieved over existing methods using the proposed depth features and region candidates.The reported improvement is relative to existing RGB-D detection methods.
- Instance segmentation: Instance segmentation is formulated as foreground labeling for pixels within detected object regions.The system infers pixel-level masks for individual object instances rather than only bounding boxes.
- Semantic segmentation: Object detections also improve semantic segmentation by supplying additional superpixel features, yielding state-of-the-art results for the studied categories.Semantic segmentation labels pixels by category without distinguishing between instances.
2 2.5D Region Proposals
The paper extends contour detection and MCG to RGB-D images, using geometric depth cues for 2.5D region proposals. These proposals improve coverage over RGB-based and Lin et al.'s baselines while remaining general across category sets.
- 2.5D Region Proposals: Depth-based region proposals generalize MCG to RGB-D images using contour, color, shape, and proposal-level geometric features.The geometric features include statistics of disparity, height above ground, angle with gravity, and world coordinates.
- Contour Detection: RGB-D contour detection combines structured random forests with normal gradients and geocentric pose features.The added cues improve contour precision and raise Fmax by 1.2 percentage points for normal gradients and another 0.6 points for geocentric pose and appearance features.
- Contour Detection: 1.5 percentage points: the final contour detector exceeds current state-of-the-art methods in Fmax.The final system follows improvements from normal gradients, geocentric pose features, and appearance features.
- Candidate Ranking: Coverage improves substantially when RGB contours are replaced by RGB-D contours, with a smaller consistent gain from depth features during candidate re-ranking.Coverage is evaluated as a function of the number of region proposals using average intersection-over-union across classes and instances.
- Candidate Ranking: The RGB-D proposal method consistently outperforms Lin et al. even on their category subset, despite being trained for generic objects.The comparison highlights both effectiveness and generality, since generic proposals also cover categories outside the evaluated subset.
3 RGB-D Object Detectors
The section generalizes R-CNN to RGB-D detection by learning CNN features from geocentrically encoded depth images and evaluating depth, augmentation, and fusion choices. The final system combines RGB and HHA features and reports object-detection results alongside qualitative outputs.
- Encoding Depth Images for Feature Learning: HHA encodes depth as horizontal disparity, height above ground, and angle with gravity for CNN feature learning.The representation emphasizes complementary depth, surface-normal, and height discontinuities.
- Feature Learning: ImageNet initialization consistently outperforms random initialization when finetuning CNNs on depth images.The paper uses an RGB-trained CNN architecture and transfers its initialization to depth representation learning.
- Encoding Depth Images for Feature Learning: 25.2% mean AP b results from finetuning with HHA, a 25% relative improvement over disparity-based finetuning.Disparity-based finetuning reached 20.1% mean AP b.
- Data Augmentation: 26.1% mean AP b follows 2× synthetic-data augmentation, while 15× augmentation slightly reduces performance because of increased synthetic-data bias.The synthetic scenes replace non-furniture objects with cuboids, changing generated-image statistics.
- RGB-D Detection Results: 32.5% mean AP b is achieved by combining RGB and HHA features after 2× synthetic-data finetuning.Individual RGB and HHA systems achieved 19.7% and 26.1%, respectively; early fusion reached 21.2% and late fusion 31.9%.
- Results: The system produces object detections and qualitative true-positive and false-positive visualizations for bed, chair, lamp, sofa, and toilet categories.The visualizations also overlay inferred instance segmentations, which can fix some mis-localized false positives.
4 Instance Segmentation
The instance-segmentation module predicts foreground masks within detected object windows using random forests and superpixel smoothing. It is evaluated with region detection average precision against box-, region-, and empirical foreground-mask baselines.
- Method: The method formulates instance segmentation as foreground-versus-background labeling within each detection window.Random forests produce pixel predictions that are subsequently smoothed by averaging over superpixels.
- Learning Framework: Random forests classify warped 50×50 detection-window pixels using unary and binary feature tests.The approach uses one monolithic classifier to avoid the nonlinearity of a single classifier’s alternative and the data scarcity of location-specific classifiers.
- Evaluation: AP r at a 0.5 region-overlap threshold evaluates instance segmentation while respecting individual object instances.The metric replaces bounding-box overlap with region intersection over union.
- Results: 32.1% mean AP r exceeds the best baseline’s 28.1% mean AP r.The baselines use box masks, averaged proposal regions, or empirical foreground masks projected to superpixels.
- Results: For some categories, AP r exceeds AP b, indicating that the instance segmentor corrected some mis-localized detections.This result shows that mask prediction can improve the evaluated localization outcome for certain categories.
5 Semantic Segmentation
The semantic segmentation system augments superpixel features with object-detector information, improving performance over prior approaches and RGBD-DPM features, especially for detector-supported categories.
- Method: Object detections provide additional features for superpixels, based on overlapping detections across object categories.These features are added to the existing superpixel classification features.
- Evaluation: The reported metrics include frequency-weighted average Jaccard, average Jaccard, and category-restricted average Jaccard.Frequency-weighted average Jaccard weights category-wise pixel IoU by category pixel frequency.
- Results: 47% fwavacc is achieved on the 40-class semantic segmentation task, the best performance among the compared approaches.The evaluation reports pixel-wise Jaccard performance across 40 categories.
- Results: 35.1% average performance is obtained for categories with added object-detector features, up from 28.4%.This corresponds to a 24% relative improvement.
- Results: Adding RGBD-DPM detector features raises fwavacc from 45.2 to 45.6, while adding the paper’s detectors raises it further to 47.0.The comparison separates gains from generic detector features and the proposed detectors.