Source-linked AI summary
SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla
TL;DR
Indoor scene understanding needs large supervised datasets, but manually collecting and labelling them is costly and existing indoor depth datasets are limited. The paper builds SceneNet, an annotated synthetic 3D-scene library that generates noise-modelled depth training data, and finds that synthetic pretraining followed by real-data fine-tuning improves per-pixel segmentation while using depth alone. It also establishes depth-segmentation results on SUN RGB-D and identifies data volume as an important factor.
Problem
Deep-learning scene understanding requires large supervised datasets, while existing indoor datasets are limited and manual labelling is labour intensive and expensive.
Method
The paper builds SceneNet, an annotated synthetic indoor 3D-scene library, and generates noise-injected depth training data from randomized physically realistic scenes.
Results
5.4% class accuracy and 3.6% global accuracy gains result from SceneNet pretraining followed by NYUv2 fine-tuning over NYU-DHA.
Takeaways & Limitations
Rendered synthetic depth frames can achieve near-state-of-the-art indoor per-pixel labelling despite using only depth data, while providing virtually unlimited training data.
Takeaways & Limitations
The experiments quantify performance only on static images and remain limited by GPU speed and memory when rendering or training at very large data scales.
Abstract
from arXiv · showhide
Scene understanding is a prerequisite to many high level tasks for any automated intelligent machine operating in real world environments. Recent attempts with supervised learning have shown promise in this direction but also highlighted the need for enormous quantity of supervised data --- performance increases in proportion to the amount of data used. However, this quickly becomes prohibitive when considering the manual labour needed to collect such data. In this work, we focus our attention on depth based semantic per-pixel labelling as a scene understanding problem and show the potential of computer graphics to generate virtually unlimited labelled data from synthetic 3D scenes. By carefully synthesizing training data with appropriate noise models we show comparable performance to state-of-the-art RGBD systems on NYUv2 dataset despite using only depth data as input and set a benchmark on depth-based segmentation on SUN RGB-D dataset. Additionally, we offer a route to generating synthesized frame or video data, and understanding of different factors influencing performance gains.
1. Introduction
Indoor scene understanding supports high-level robotic and modeling tasks, but deep learning requires large supervised datasets that are costly to collect. The paper proposes synthetic annotated 3D scenes as a scalable source of training data for depth-based semantic labelling.
- 1. Introduction: Per-pixel semantic labelling is studied as a scene-understanding task supporting high-level activities such as navigation, object placement, and 3D modelling.Semantic understanding provides information about object identity, location, volume, and physical extent.
- 1. Introduction: Deep learning approaches require large quantities of supervised training data, making manual dataset collection labour intensive and potentially impractical.The paper focuses on the challenge of obtaining sufficient training data for scene understanding.
- 1. Introduction: NYUv2 contains 795 training images for 894 object classes, while SUN RGB-D contains 5,285 training images for 37 classes.Both datasets are described as limited relative to the data needed for strong performance on unseen data.
- 1. Introduction: Existing indoor datasets rely on human labelling, which is tedious and expensive, motivating synthetic data from computer-graphics repositories.The paper identifies synthetic data and gaming environments as potential sources of training data.
- 1. Introduction: The paper proposes an annotated 3D-scene dataset that can generate virtually unlimited ground-truth training data for real-world indoor per-pixel labelling.The stated contribution is to show the potential of this dataset for challenging real-world indoor datasets.
2. Related Work
Prior indoor per-pixel labelling datasets are small, noisy, incompletely annotated, and affected by human labelling errors. SceneNet addresses these limitations by generating annotated synthetic scenes and training data from computer-graphics assets.
- 2. Related Work: NYUv2 and SUN RGB-D provide limited per-pixel labels, low-quality raw depth maps, and RGB frames, while missing or incorrect human labels are common.Figure 1 illustrates missing labels and a frame assigning the same ground-truth label to a toilet and sink.
- 2. Related Work: Fully labelled 3D models for arbitrary viewpoints are absent from both datasets; SUN3D provides only eight annotated video sequences, and NYUv2 annotates one frame per video.NYUv2 videos were not captured with accurate 3D reconstruction or annotation generation in mind.
- 2. Related Work: SceneNet provides potentially unlimited labelled training data by building a library of synthetic indoor scenes with perfect ground truth.The system uses a new library of synthetic indoor scenes called SceneNet to train a deep learning algorithm for per-pixel labelling.
- 2. Related Work: Earlier synthetic scenes were small, about 4m×3m×3m, and contained only one or two instances of characteristic objects.SceneNet is positioned against this limited scale and object variety.
3. Synthesizing Training Data
SceneNet builds annotated synthetic indoor scenes and augments them with automatically generated, physically realistic arrangements. Its 3D annotations support arbitrary viewpoints, scene variation, and training-data generation for indoor scene understanding.
- SceneNet Basis Scenes: SceneNet Basis Scenes (SN-BS) is an open-source repository of annotated synthetic indoor scenes whose 3D labels support arbitrary viewpoints.The repository contains metrically accurate models and annotations that can be adapted to different granularities.
- SceneNet Basis Scenes: SN-BS contains five scene categories with at least 10 annotated scenes per category, each composed of 15–250 objects.Scene complexity can be controlled algorithmically, and the models are publicly hosted.
- Scene variation: 3D annotations enable scene variation by replacing objects, perturbing positions, and adding new objects without independently relabelling every generated image.These operations increase intra-class shape variation and produce new scenes from existing annotated content.
- Automatic furniture arrangement: New physically realistic scenes are generated automatically by sampling objects from online repositories and optimizing their arrangements with simulated annealing.The generated scenes inherit annotations because sampled objects already carry labels.
- Automatic furniture arrangement: The optimization combines feasibility, pairwise-distance, visibility, wall-distance, and angle constraints to arrange objects in meaningful configurations.Pairwise statistics capture contextual relationships such as beds with nightstands and chairs with tables; hierarchical optimization supports cluttered scenes.
- Automatic furniture arrangement: Adding pairwise constraints produces sensible relative furniture positions, while adding all constraints removes occlusions from the generated views.Without pairwise or visibility constraints, objects are scattered or occlude one another.
4. Results
Synthetic depth data improves indoor semantic segmentation when fine-tuned on real data, while dropout adds little after synthetic-data training. The depth-only system approaches state-of-the-art performance but remains limited on appearance-dependent classes.
- Data and experimental setup: 10,030 noisy depth images were rendered from random viewpoints with object-visibility constraints and depth perturbations resembling NYUv2.The experiments used DHA inputs and compared synthetic, fine-tuned, dropout, and real-data training variants on NYUv2 and SUN RGB-D.
- Comparisons with NYUv2 training: 5.4% class and 3.6% global accuracy gains followed fine-tuning SceneNet-DHA on NYUv2, compared with NYU-DHA.Convergence was also twice as fast for SceneNet-FT-NYU-DHA.
- Comparisons with Eigen and Fergus: SceneNet-FT-NYU-DHA achieved near state-of-the-art performance despite using depth alone, but remained below the RGBD-and-normals system overall.Its main compromises were tv and windows, while geometry remained a strong cue for functional object categories.
- Comparisons with dropout: 3.2% class and 1.9% global accuracy improvements came from dropout with NYU training, whereas synthetic-data variants showed only minor additional gains.The authors suggest that increased data acts as an implicit regulariser.
- Confidence in the final predictions: Test-time dropout gave similar performance gains without it, while uncertainty was mainly concentrated at object boundaries.The uncertainty measure was the ratio of the second-best to best label probabilities.
13 Class Semantic Segmentation: NYUv2
On the 13-class NYUv2 task, synthetic-data pretraining followed by fine-tuning consistently improves depth-based segmentation, while depth remains weaker for appearance-dependent categories.
- Results on 13 Class Semantic Segmentation: NYUv2: Books, painting, tv, and windows remain the main compromised classes because depth is limited for segmenting them.The authors attribute these categories’ difficulty to the expected role of RGB information.
- Confidence in the final predictions: The uncertainty visualization uses the ratio of second-best to best label probabilities, with brighter colors indicating higher uncertainty.The displayed predictions are those of SceneNet-FT-NYU-DHA on NYUv2 test images.
5. Conclusions
The paper presents synthetic rendered depth frames as a route to near-state-of-the-art indoor per-pixel labelling using depth alone. SceneNet also provides annotated 3D scenes for generating virtually unlimited training data and future sequential-data experiments.
- 5. Conclusions: Large numbers of rendered synthetic depth frames enable near-state-of-the-art per-pixel labelling despite using only depth data.The conclusion frames synthetic data as a promising route for improving indoor scene segmentation.
- 5. Conclusions: SceneNet’s annotated 3D scenes can generate virtually unlimited training data and potentially support annotated videos for sequential learning.Proposed future applications include RNNs, reinforcement learning, and physical scene understanding.
13 Class Semantic Segmentation: SUNRGBD
On SUN RGB-D, synthetic-data pretraining followed by real-data fine-tuning improves 13-class depth segmentation, while additional dropout contributes little and depth remains limited for several classes.
- Results on 13 Class Semantic Segmentation: SUN RGBD: SceneNet-FT-SUNRGBD-DHA and SceneNet-FT-SUNRGBD-DO-DHA perform nearly the same, showing diminishing returns from dropout.The table reports similar gains across the 13 semantic classes.
- Results on 13 Class Semantic Segmentation: SUN RGBD: Books, painting, tv, and windows remain weaker categories, highlighting depth’s limited role for segmenting these objects.The limitation is reported for the 13-class SUN RGB-D evaluation.