Source-linked AI summary
Indoor Semantic Segmentation using depth information
Camille Couprie, Clément Farabet, Laurent Najman, Yann LeCun
TL;DR
Indoor semantic labeling lacks feature-learning approaches that directly incorporate depth. This paper adapts a multiscale convolutional network to RGB-D scene labeling, achieving reported gains over prior work and illustrating video processing with temporal smoothing.
Problem
Indoor full-image semantic labeling requires methods that handle RGB-D scenes, while prior approaches commonly relied on hand-crafted features or did not exploit depth in feature learning.
Method
A multiscale convolutional network learns features directly from RGB and depth inputs, followed by classifier prediction aggregation over superpixels.
Results
The method improves classwise accuracy by 4% over previous works and pixelwise accuracy by almost 6% compared with Silberman et al.’s results.
Takeaways & Limitations
The approach supports indoor scene labeling without designing depth-specific hand-crafted features and can illustrate video labeling with near-real-time processing characteristics.
Takeaways & Limitations
Video performance cannot be quantitatively evaluated because the NYU v2 video sequences lack ground-truth labels.
Abstract
from arXiv · showhide
This work addresses multi-class segmentation of indoor scenes with RGB-D inputs. While this area of research has gained much attention recently, most works still rely on hand-crafted features. In contrast, we apply a multiscale convolutional network to learn features directly from the images and the depth information. We obtain state-of-the-art on the NYU-v2 depth dataset with an accuracy of 64.5%. We illustrate the labeling of indoor scenes in videos sequences that could be processed in real-time using appropriate hardware such as an FPGA.
1 Introduction
The paper targets full-image semantic labeling of challenging indoor environments using RGB-D data and adapts feature learning to exploit depth information.
- 1 Introduction: NYU depth v1 introduced full-image semantic labeling with depth information across twelve object categories.The dataset contains 2347 image, depth-map, and ground-truth triplets.
- 1 Introduction: Prior semantic-segmentation datasets commonly centered objects under favorable lighting, whereas NYU depth targets realistic indoor environments.
- 1 Introduction: NYU depth v2 expanded the task to 894 object classes and hundreds of video sequences totaling 407024 frames.
- 1 Introduction: The work adapts Farabet et al.’s multiscale network to learn features for indoor scene labeling with RGB and depth inputs.
- 1 Introduction: The authors describe this as the first use of depth information in a feature-learning approach for full scene labeling.
2 Full scene labeling
The labeling system learns multiscale RGB-D features with convolutional networks, then aggregates classifier predictions within superpixels; temporal superpixels extend it to video.
- 2.1 Multi-scale feature extraction: Each scale is processed by a three-stage convolutional network, and feature maps are concatenated after coarser maps are upsampled.
- 2.1 Multi-scale feature extraction: The RGB-D input is represented through multiple scales of a Laplacian pyramid, providing feature vectors with large contextual windows around pixels.
- 2.1 Multi-scale feature extraction: Depth is treated as an additional input channel alongside RGB, with local neighborhoods normalized to zero mean and unit standard deviation.
- 2.1 Multi-scale feature extraction: A two-layer classifier is trained with negative log-likelihood loss, and superpixels smooth predictions by aggregating classifier outputs within each region.
- 2. Full scene labeling: Temporal-consistent superpixels can smooth frame-by-frame video predictions, reducing object flicker in quasi-linear time.
3 Results
Experiments on NYU depth v2 evaluate RGB-only and RGB-D multiscale ConvNets across classwise and pixelwise segmentation accuracy, including image and video settings. Depth improves several depth-consistent classes and yields gains over prior work, while video results remain illustrative because no ground truth is available.
- 3.1 Validation on images: The NYU depth v2 experiments use 407024 RGB-depth pairs, 1449 labeled frames, 894 categories, and the provided split of 795 training and 654 test images.Training uses all 894 categories as output classes without changing class frequencies or adding data augmentation.
- 3.1 Validation on images: The study compares a multiscale ConvNet trained on RGB channels with another trained using additional depth information.The RGB and RGB-D networks were trained for 105 and 98 epochs, respectively, each in less than two days on a regular server.
- 3.1 Validation on images: Depth provides gains of 15% or more for the floor, ceiling, and furniture classes, whose depth appearances are comparatively consistent.Floors and ceilings tend to exhibit similarly oriented depth gradients across the dataset, whereas some objects may occur in foreground or background positions.
- 3.2 Comparison with Silberman et al.: Against Silberman et al., the multiscale ConvNet improves structure by 4% in pixelwise accuracy, while RGB-D results gain 4% classwise and almost 6% pixelwise accuracy.The four-class evaluation uses Ground, Furniture, Props, and Structure, and depth particularly improves ground prediction.
- 3.3 Test on videos: Video predictions are produced frame by frame and refined with temporally consistent superpixels, reducing flicker in quasi-linear time.The video examples cannot be quantitatively evaluated because no ground truth is available; some wall-in-foreground results could improve with more training examples.
4 Conclusion
The model supports flexible indoor-scene applications through depth-aware feature learning and alternative class clusterings, while several additional techniques could improve the present system.
- Depth information improves recognition of object classes with similar depth appearance and location.
- RGB-only inputs are better for object classes whose depth maps have high variability.
- Different object-class clusterings, including a 4-class grouping, support application-specific uses such as inferring support relations.
- Unsupervised feature learning, MRF smoothing, and training-set extension could improve the present system.