Source-linked AI summary
InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset
Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, Stefan Leutenegger
TL;DR
Existing indoor datasets support SLAM and semantic understanding but face scalability, annotation, or realism limitations. InteriorNet builds a large synthetic RGB-D-inertial dataset from professional assets and layouts, with realistic scene variation, rendering, and trajectories, and demonstrates its use through SLAM evaluation.
Problem
Existing datasets support SLAM and semantic understanding but real-world ground-truth preparation is costly and synthetic datasets need greater scale, realism, and variability.
Method
InteriorNet constructs a synthetic indoor dataset from millions of detailed furniture models and professional layouts, rendering varied RGB-D-inertial sequences with realistic lighting, scene changes, and trajectories.
Results
The authors demonstrate the dataset's usefulness by evaluating SLAM algorithms on its rendered sequences.
Takeaways & Limitations
InteriorNet provides photo-realistic indoor footage and diverse trajectory types for benchmarking SLAM and training or evaluating semantic understanding.
Takeaways & Limitations
Lighting and scene changes are configured randomly because real-world ground truth for them is unavailable, while data-driven rearrangement is left for future work.
Abstract
from arXiv · showhide
Datasets have gained an enormous amount of popularity in the computer vision community, from training and evaluation of Deep Learning-based methods to benchmarking Simultaneous Localization and Mapping (SLAM). Without a doubt, synthetic imagery bears a vast potential due to scalability in terms of amounts of data obtainable without tedious manual ground truth annotations or measurements. Here, we present a dataset with the aim of providing a higher degree of photo-realism, larger scale, more variability as well as serving a wider range of purposes compared to existing datasets. Our dataset leverages the availability of millions of professional interior designs and millions of production-level furniture and object assets -- all coming with fine geometric details and high-resolution texture. We render high-resolution and high frame-rate video sequences following realistic trajectories while supporting various camera types as well as providing inertial measurements. Together with the release of the dataset, we will make executable program of our interactive simulator software as well as our renderer available at https://interiornetdataset.github.io. To showcase the usability and uniqueness of our dataset, we show benchmarking results of both sparse and dense SLAM algorithms.
1 Introduction
InteriorNet addresses the scalability and realism limits of datasets used for SLAM and semantic understanding through a large synthetic indoor-scene pipeline. It combines professionally designed assets and layouts with varied configurations, trajectories, rendering, and simulator support.
- Motivation: Synthetic datasets avoid the costly, time-consuming ground-truth preparation required by real-world scanning while supporting SLAM benchmarking and semantic learning.The paper positions synthetic rendering as a scalable complement to real recorded datasets.
- Contributions: The scene database contains around 1M furniture CAD models and 22M interior layouts created for real-world production and decoration.These assets provide the foundation for large-scale indoor-scene generation.
- Contributions: The dataset includes 15k long sequences and 5M images from randomly selected layouts, with varied lighting, rearrangement, and ground-truth modalities.The pipeline supports RGB-D, semantic, trajectory, inertial, and event-camera data.
- Contributions: InteriorNet releases rendered data, ExaRenderer, ViSim, and a subset of the 3D models and layouts used for evaluation.The tools support dataset creation and camera-trajectory ground-truth synthesis.
2 Background
Prior datasets support semantic labeling and visual SLAM but remain limited by scale, model diversity, or manual rendering and annotation requirements. InteriorNet is presented against both synthetic and real-world alternatives.
- Synthetic semantic datasets: SUN-CG provides 45K layouts and 2.6K unique object meshes, while InteriorNet uses professionally designed assets and layouts at much larger scale.The comparison concerns layout and object-model scale as well as source design quality.
- Real-world datasets: ScanNet contains 2.5M frames from 1.5K indoor scenes with manual annotations created using an Amazon Mechanical Turk tool.It is identified as the closest related real-world dataset.
- SLAM datasets: Real-world SLAM datasets obtain trajectory ground truth from GPS or motion capture and map ground truth from laser scanners in some datasets.These datasets include TUM RGB-D, EuRoC, KITTI, New College, and Active Vision examples.
- Synthetic SLAM datasets: UnrealCV and ICL-NUIM provide synthetic visual SLAM data, but require manual rendering setup or contain limited numbers of models and sequences.The paper contrasts these constraints with its larger rendering environment.
3 Dataset Overview
The dataset overview combines a million-scale object database, tens of millions of professional layouts, configurable scene variants, and an end-to-end renderer producing diverse images and ground truth.
- Object and layout database: The database contains 1,042,632 furniture CAD models from 42 manufacturers, categorized into 158 classes and mapped to NYU40 categories.The models use real-world dimensions and high-resolution geometry with associated textures.
- Object and layout database: Professional designers created 22,652,123 layouts spanning 16 room types, from studios to 42-room apartments.Most layouts are used in real-world decoration.
- Configurations: Each layout receives configurations that rearrange furniture or randomize lighting to simulate daily-life variation.Furniture changes use physics-based automation, while lighting changes can be manual or automatic.
- Rendering and ground truth: The pipeline renders 640×480 images at 25 Hz, producing 1,000 images per trajectory with motion-blurred views from shutter-open and shutter-close poses.Trajectories vary linear velocity, angular velocity, and trajectory type.
- Rendering and ground truth: Per-frame outputs include NYU40 semantic labels, 3D object boxes, Euclidean depth, noisy Kinect-like depth, instance segmentation, and optical flow.These annotations are generated from model settings and rendering passes or simulated sensors.
- Dataset structure: The dataset contains 20M photo-realistic images across around 1.7M layouts and a sparse 5M-image subset with higher diversity.The data are organized into 20 subsets by number of rooms.
4 Simulating Realistic Scenes
InteriorNet introduces scene changes and lighting variation to make synthetic indoor data more closely reflect daily-life conditions. These changes are controlled through physics-based object manipulation and configurable lighting.
- Scene variation: The dataset simulates daily-life challenges by adding moving objects and variations in lighting.These features extend beyond static synthetic scenes.
- Scene variation: The pipeline generates configurations by rearranging furniture or randomizing lighting to diversify layouts and simulate daily life.Physics-based rearrangement and manual or automatic lighting changes are both supported.
- Object rearrangement: Furniture rearrangement uses convex collision hulls, object mass, and friction coefficients supplied or assigned for physics-based simulation.Masses range from 0.05 to 43.3 kg, while friction coefficients range from 0.08 to 0.27.
- Lighting variation: Lighting setups encode type, energy, distance, position, direction, and brightness, with controls for RGB value, temperature, brightness, and on/off state.The simulator also automatically generates lighting combinations.
5 Trajectory Generation
InteriorNet generates varied camera trajectories and augments them with a learned style model to better capture realistic motion rather than simple smooth paths.
- Trajectory types: InteriorNet generates three trajectory types: two-body random, hand-held, and look-forward.Each type exposes speed and angular-velocity parameters.
- Two-body random trajectories: Two-body random trajectories simulate colliding physical bodies for camera position and look-at point, with camera heights constrained between 1 m and 2 m.The up direction is randomized by up to 5° from gravity.
- Specialized motion types: Hand-held trajectories model downward viewing bias, while look-forward trajectories add translational motion along the viewing direction.The look-forward motion uses a velocity-directed target and proportional force for smoothing.
- Motivation: Existing automated trajectories often use simple smooth motion that misses camera-motion nuances affecting visual data, including walking-induced acceleration impulses and motion blur.The effect is especially relevant in low-light indoor conditions.
- Learned trajectory synthesis: A data-driven method automatically synthesizes millions of realistic camera trajectories while satisfying scene constraints.The approach is motivated as an alternative to manually captured motion-capture trajectories, which are limited to small datasets.
6 ExaRenderer: Photo-realistic RGB Renderer
ExaRenderer combines CPU/GPU path tracing with configurable camera and sensor simulation to produce photo-realistic RGB imagery and associated inertial or event data.
- Renderer architecture: ExaRenderer is a fast Embree-based renderer with CPU and GPU variants, flexible APIs, and dynamic distribution across computing resources.It supports a common subset of path-tracing operations from leading commercial renderers.
- Path tracing: Path tracing approximates realistic global illumination and simulates soft shadows, depth of field, motion blur, indirect lighting, color bleeding, and caustics.The paper emphasizes realistic caustics, lower memory needs for large-scale rendering, and support for dynamic scenes compared with photon mapping.
- Camera simulation: The renderer supports pinhole, perspective, depth-of-field, fisheye, and panorama cameras, with fixed RGB resolution of 640×480 pixels.Panorama and fisheye outputs use 5,000×2,500 and 600×600 resolutions, respectively.
- Inertial simulation: IMU readings are derived from a cubic B-spline representation of camera position and orientation by differentiating position twice and incorporating gravity.The continuous-time representation supports sampling at arbitrary rates.
- Event simulation: Event-camera outputs are generated by rendering at high frame rate and emitting timestamped events when per-pixel brightness changes cross a user-set intensity threshold.The renderer can produce novel camera designs such as event cameras.
- Large-scale rendering: The system rendered about 20M images using 1,300 Nvidia GTX TitanX GPUs and 2,000 Intel Xeon Phi CPUs over approximately 4 days.Rendering used 256 samples per pixel.
7 Evaluation on Simultaneous Localization and Mapping
The dataset is evaluated with ORBSLAM2.0 and ElasticFusion across varied scene conditions and trajectory difficulties, showing accurate estimates in sample cases while exposing challenging motion regimes.
- Evaluation setup: The verification evaluates ORBSLAM2.0 in RGB-D mode and ElasticFusion using absolute trajectory error on selected sequences.ORBSLAM2.0 uses default configuration parameters.
- Scene and trajectory conditions: Sample type-1 trajectories are tested in regular scenes, altered lighting, and object-displaced scenes, with estimated trajectories compared against ground truth over dense reconstructions.Additional tests vary trajectory type and difficulty.
- Challenging motion: ORBSLAM2.0 cannot track all frames for type-1 trajectories with very high position and angular velocities, indicating that the synthesized motions create difficult tracking cases.The evaluation includes multiple trajectory types and difficulty levels.
- Quantitative result: 0.0345 m average ATE, with 0.02 m standard deviation across scenes, was obtained for another 24.91 m random trajectory evaluated with several random models.The reported variation indicates that model choice affects results.
- Tracking robustness: As trajectory difficulty increases, the percentage of frames tracked by ORBSLAM2.0 decreases across 50 trajectories.Difficulty is based on maximum position and angular velocities.
8 Conclusions
InteriorNet presents a large, photo-realistic indoor video dataset built from professionally designed assets and varied through lighting, object rearrangement, and diverse camera trajectories.
- Dataset scale: The dataset accesses millions of professionally designed interior layouts, furniture models, and object models to create large-scale indoor video sequences.The assets were designed to high specification.
- Scene variability: Lighting and object rearrangement introduce scene variability intended to simulate changes in daily life and natural lighting.The dataset produces photo-realistic footage at high frame rate.
- Temporal variation: InteriorNet synthesizes diverse trajectory types because the authors argue that the temporal aspect of scene understanding deserves closer attention.The dataset’s usefulness is demonstrated by evaluating SLAM algorithms.
- Scope boundary: Lighting and scene changes are configured randomly because real-world ground truth for those changes is unavailable.A data-driven approach to scene rearrangement is left for future work.