Source-linked AI summary
SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
John McCormac, Ankur Handa, Stefan Leutenegger, Andrew J. Davison
TL;DR
Dense labels for many scene-understanding and geometric tasks are costly or difficult to obtain, motivating large-scale synthetic RGB-D data. SceneNet RGB-D generates photorealistic indoor trajectories with varied scenes and ground truth; its scope is bounded by static scenes and missing physical attributes.
Problem
Dense annotations for semantic and instance segmentation are expensive, while fine-grained optical flow and metrically accurate 3D object poses can be almost impossible to obtain by hand.
Source: §1 · p. 1Method
Results
The paper presents a large-scale dataset of photorealistic synthetic indoor trajectories with ground truth for scene-understanding and geometric computer-vision tasks.
Source: §1 · p. 2Takeaways & Limitations
Randomized scenes, lighting, textures, and camera trajectories provide varied synthetic training examples and sequential views for temporal and 3D scene-understanding research.
Source: §1 · p. 2, §2 · p. 3Scenes are static and lack physical attributes such as mass and friction, limiting immediate use in interactive, physically realistic dynamic scenes.
Source: §8 · p. 10
Abstract
from arXiv · showhide
We introduce SceneNet RGB-D, expanding the previous work of SceneNet to enable large scale photorealistic rendering of indoor scene trajectories. It provides pixel-perfect ground truth for scene understanding problems such as semantic segmentation, instance segmentation, and object detection, and also for geometric computer vision problems such as optical flow, depth estimation, camera pose estimation, and 3D reconstruction. Random sampling permits virtually unlimited scene configurations, and here we provide a set of 5M rendered RGB-D images from over 15K trajectories in synthetic layouts with random but physically simulated object poses. Each layout also has random lighting, camera trajectories, and textures. The scale of this dataset is well suited for pre-training data-driven computer vision techniques from scratch with RGB-D inputs, which previously has been limited by relatively small labelled datasets in NYUv2 and SUN RGB-D. It also provides a basis for investigating 3D scene labelling tasks by providing perfect camera poses and depth data as proxy for a SLAM system. We host the dataset at http://robotvault.bitbucket.io/scenenet-rgbd.html
1. Introduction
The paper targets the scarcity of large labelled datasets for computer vision, especially where tasks require dense pixel-level or geometric ground truth. It introduces photorealistic synthetic RGB-D data generated automatically from varied, physically feasible indoor scenes.
1. Introduction
- Per-pixel annotations for semantic labelling and instance segmentation are prohibitively expensive, while fine-grained optical flow and metrically accurate 3D object poses can be almost impossible to obtain by hand.Source: §1 · p. 1
- The dataset aims to provide large-scale photorealistic RGB-D videos with complete ground truth for a wide range of computer-vision problems.Source: §1 · p. 2
- Ray-traced images include lighting effects and motion blur, while smooth camera trajectories create sequential clips for research on temporal fusion.Scene contents are sampled from public scene repositories and placed through physics simulation, producing feasible configurations without manual scene design or annotation.Source: §1 · p. 2
2. Background
The background situates SceneNet RGB-D among synthetic-data approaches and contrasts its random, photorealistic indoor trajectories with existing datasets. Its design emphasizes varied configurations and sequential views, while the value of randomness for learning remains an open question.
2. Background
- Synthetic imagery has supported 2D–3D alignment, object detection, optical flow, and human-action recognition.Source: §2 · p. 2
- SceneNet [9] supplied labelled synthetic scenes and depth-image semantic ground truth; training on 10K synthetic images followed by fine-tuning improved segmentation performance on NYUv2 and SUN RGB-D.Source: §2 · p. 2
- SceneNet RGB-D builds on open-source scene layouts [9], textured objects, and ray tracing, while gaming-engine data can lack open-source flexibility because of proprietary issues.Source: §2 · p. 2
- Unlike SUN-CG [24], SceneNet RGB-D targets photorealistic sequential trajectories with camera poses for 3D correspondences, while random textures, lights, and trajectories vary appearance across geometrically identical scenes.Source: §2 · p. 3
- Random scene generation may reduce predictability and support on-the-fly examples, but whether randomness is preferable to designed scenes remains open.Source: §2 · p. 3
3. Dataset Overview
The dataset pairs rendered trajectories with dense labels and geometric ground truth. Its construction balances rendering cost against sequence variation and supplies distinct train, validation, and test data.
3. Dataset Overview
- Each trajectory lasts five minutes at 320×240 resolution, with one frame per second; each render takes 2–3 seconds on an NVIDIA GTX 1080 GPU.The chosen sampling balances video frame rate against rendering cost and redundant images.Source: §3 · p. 3
- The rendering schedule trades frame rate against the computational cost of producing many similar images, and render time also trades off against image quality.Source: §3 · p. 3
- Camera poses are calculated at 25Hz, but only every 25th pose is rendered; depth is the ray length from the camera origin to the first surface hit.Motion blur is simulated by sampling between shutter-open and shutter-close poses; instance labels use a separate rendering pass.Source: §3 · p. 3
- Instance labels map to 255 WordNet semantic categories, and optical flow can be calculated from camera pose and depth under the static-scene assumption.Source: §3 · pp. 3–4
- Reprojected surfaces yield 3D positions and voxel correspondence indices across a trajectory, approximating correspondences from a perfect SLAM system.Source: §3 · p. 4
- The dataset has 5M training images, and its validation and test sets have 300K images from 1K different room layouts; layouts, objects, and trajectories are unique across the three splits.ShapeNet objects are split 80-10-10% for train, validation, and test within each WordNet class.Source: §3 · p. 4
4. Obtaining metric scales of CAD models
The method scales CAD objects to metric dimensions by sampling category-specific heights measured in SUN RGB-D, while coarse categories can yield multimodal distributions and implausible object dimensions.
4. Obtaining metric scales of CAD models
- Object heights are sampled from SUN RGB-D category distributions to scale CAD models while preserving their aspect ratios.SUN RGB-D provides metric 3D bounding boxes, which the authors use to estimate category height distributions.Source: §4 · p. 5
- Coarse categories can mix objects with very different heights, so applying a floor-lamp height to a bedside lamp can make it refrigerator-like in scale.The authors identify limited class granularity as a drawback and point to future work on addressing it.Source: §4 · p. 5
5. Generating random scenes with physics
Scenes are populated with context-sensitive object samples and settled through physics simulation, yielding physically plausible but not necessarily human-organized configurations.
5. Generating random scenes with physics
- The authors choose an off-the-shelf physics engine because rendering is the computational bottleneck, leaving the CPU available for simulation while the GPU renders.They also cite ease of use, reasonable-looking layouts, and potential future dynamic scenes as reasons for the choice.Source: §5 · p. 5
- Object density is sampled by size: 0.1–0.5 large objects/m2 and 0.5–3.0 small objects/m2, with small objects defined as under 0.4 m.The number of objects follows from the scene floor area.Source: §5 · p. 5
- Object categories are sampled according to scene-type frequencies in SUN RGB-D, then a model instance is chosen uniformly within the selected class.This is intended to place context-relevant objects in scenes, such as sinks or toilets in bathrooms.Source: §5 · p. 5
- Objects receive a constant 10 kg mass and convex collision hulls; lowering their centers of gravity biases them toward upright poses, though hulls can prop objects up unnaturally.A table, for example, can be supported by a small object beneath its center.Source: §5 · p. 5
- Project Chrono simulates object movement and interactions for 60 seconds so objects settle into physically plausible configurations without unrealistic intersections.The layouts themselves are static nonconvex collision objects; the resulting scenes need not be organized in a human manner.Source: §5 · p. 5
6. Generating random trajectories
The paper automates camera trajectory generation with random motion designed to resemble human viewing, then simulates two bodies to produce smooth paths and records scene and trajectory data in a structured file.
6. Generating random trajectories
- Automated random camera trajectories address the scaling limits of collecting human-operated SLAM paths, with motion biased toward central areas and a mix of fast and slow rotations.The design emphasizes yaw and pitch over roll, which the authors describe as less prominent in human trajectories.Source: §6 · pp. 5–6
6.1. Two body camera trajectories
- Two simulated bodies represent the camera and its look-at point; fixing the up vector along positive y locks roll and defines the camera coordinate system.Random forces and drag drive the bodies, and the physical approach produces smooth trajectories and can provide IMU-style accelerometer measurements.Source: §6.1 · p. 6
- Initialization checks restart trajectories when either body leaves the bounding volume or fewer than 10 object instances are visible within the first 500 poses.The checks help prevent paths that remain outside the scene and show only its outer wall.Source: §6.1 · p. 6
6.2. Scene and Trajectory Description
- The trajectory file records timestamps, camera and look-at positions in world coordinates, frame rate, and shutter duration.Object height in meters and each object's full 3×4 transformation are also saved.Source: §6.2 · p. 8
7. Rendering photorealistic RGB frames
The rendering pipeline combines GPU ray tracing with photon mapping, randomized scene appearance, a camera model, and exposure-sampled motion blur. These choices balance photorealistic effects, ground-truth generation, and rendering cost.
7. Rendering photorealistic RGB frames
- The open-source OptiX ray tracer supports GPU rendering and flexible ground-truth outputs, including depth and object-instance numbers.The renderer adds features such as specular materials, ground-truth materials, and multiple photon maps.Source: §7 · p. 8
7.1. Photon Mapping
- Photon mapping approximates the rendering equation in two passes, adding global illumination, diffuse color bleeding, and caustics to ray-traced direct illumination.Photon maps are maintained throughout each static-scene trajectory, and Figure 7 compares direct and indirect photon mappings.Source: §7.1 · p. 8, Figure 7 · p. 8
7.2. Rendering Quality
- The final renders use 16 samples per pixel and 4 photon maps, at approximately 3 seconds per image on one GPU.More than 8 photon maps exceed 32GB of memory; above that threshold, maps must be stored to disk or recomputed for each trajectory frame.Source: §7.2 · pp. 8–9, Figure 9 · p. 9
7.3. Random Layout Textures and Lighting
- Random textures and lighting vary a layout’s visual characteristics, but this work renders only one version of each layout.The pipeline selects 1–5 lights, with randomized hue, power, and location, and randomly assigns textures to layout components.Source: §7.3 · p. 9, Figure 10 · p. 9
7.4. Camera Model and CRF
- The camera uses a global-shutter pinhole model with a 20 cm focal length and 60° horizontal and 40° vertical fields of view, plus a hard-coded response curve.The nonlinear camera response function maps irradiance to quantized brightness.Source: §7.4 · p. 9, Figure 11 · p. 10
7.5. Motion Blur
- Motion blur is rendered by sampling interpolated camera poses across the shutter exposure and integrating irradiance; depth and instance-label ground truth use the exposure midpoint.The approach changes the sampled poses for each pixel rather than averaging RGB values after rendering.Source: §7.5 · p. 10, Figure 12 · p. 10
8. Conclusion
The conclusion presents SceneNet RGB-D as a large-scale source of realistic synthetic per-pixel labels, while noting limits in object data and dynamic-scene coverage.
8. Conclusion
- The authors report progress on physically realistic layouts, camera trajectories, and photorealistic rendering, while noting that object-scale and label consistency still need work.Their automated search for the category “box” included ShapeNet mailboxes, requiring manual intervention to prevent such mistakes.Source: §8 · p. 10
- Static scenes and missing object attributes such as mass and friction limit immediate use in interactive, physically realistic dynamic scenes.The authors note that on-the-fly rendering for such systems could be possible with enough compute power.Source: §8 · p. 10
- The pipeline’s randomness can generate a continuous stream of unseen training examples designed to target current model limitations.The authors anticipate tighter interleaving of training-data generation and model training.Source: §8 · p. 10