Source-linked AI summary
The Replica Dataset: A Digital Replica of Indoor Spaces
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, Richard Newcombe
TL;DR
Machine-learning research needs realistic, controllable digital environments for perception and embodied agents. Replica addresses this need with 18 photo-realistic indoor reconstructions containing dense geometry, HDR textures, semantic annotations, and reflectors. The dataset supports realistic rendering and simulation-based research across embodied navigation, segmentation, geometry inference, and related tasks.
Problem
Machine-learning research lacks realistic digital environments that support active perception, embodied-agent training, and visually, geometrically, and semantically realistic modeling.
Method
Replica constructs 18 indoor scene reconstructions with dense meshes, HDR textures, per-primitive semantic class and instance information, planar reflectors, and mesh fixes for holes and reflective surfaces.
Results
Replica provides highly realistic renderings and high color, geometry, and semantic resolution for simulation and 3D perception research.
Takeaways & Limitations
Replica supports active-perception and embodied-agent research in AI Habitat while extending beyond static image datasets to realistic 3D environments.
Abstract
from arXiv · showhide
We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic class and instance information, and planar mirror and glass reflectors. The goal of Replica is to enable machine learning (ML) research that relies on visually, geometrically, and semantically realistic generative models of the world - for instance, egocentric computer vision, semantic segmentation in 2D and 3D, geometric inference, and the development of embodied agents (virtual robots) performing navigation, instruction following, and question answering. Due to the high level of realism of the renderings from Replica, there is hope that ML systems trained on Replica may transfer directly to real world image and video data. Together with the data, we are releasing a minimal C++ SDK as a starting point for working with the Replica dataset. In addition, Replica is `Habitat-compatible', i.e. can be natively used with AI Habitat for training and testing embodied agents.
I. INTRODUCTION
Replica targets highly realistic digital replicas of real spaces to support scalable machine-learning research and embodied-agent development. Its scenes enable active perception and realistic rendering beyond static image datasets.
- Replica combines real-environment replicas with simulators such as AI Habitat for scalable training of embodied agents.The supported tasks include navigation, instruction following, and question answering.
- Unlike static image datasets, Replica supports active perception by generating views from anywhere inside each 3D scene.
- Replica contains 18 scenes and provides dense geometry, high-resolution HDR textures, semantic annotations, and renderable mirror and glass reflectors.
- Replica renderings are difficult to distinguish from raw captures because both depict the same spaces from the same camera poses.Small artifacts and the absence of motion blur reveal the rendered images in the comparison.
- Compared with Matterport 3D and ScanNet, Replica is presented as offering cleaner geometry and texturing for semantic rendering.The comparison highlights geometry and texturing artifacts, missing surfaces, and incomplete rooms in those datasets.
II. RELATED WORK
Existing 3D datasets broadly comprise human-generated synthetic scenes or reconstructions of real environments, with varying semantic and visual realism.
- Existing 3D datasets fall broadly into synthetic CAD-based scenes and reconstructions of real environments.These categories differ in semantic and visual realism.
A. Synthetic Scenes
Synthetic indoor datasets provide scale and controllable variation, but their appearance and semantics may remain less realistic than real spaces.
- SUNCG provides a large collection of synthetic indoor environments, but its scenes often lack realistic appearance and semantic simplicity.
- SceneNet offers 57 scenes and 3,699 object instances with automatic replacement of similarly sized objects within classes.
- Stanford Scenes contains 130 scenes and 1,723 object instances, while RobotriX offers 16 scenes with more realistic appearance.
- InteriorNet contains 1M CAD assets, 22M interior environments, and 20M rendered images for SLAM benchmarking and machine learning.
- Synthetic datasets still do not capture real-space imperfections, clutter, and semantic variety despite improving realism.
B. Real Scenes
Real-environment reconstruction datasets provide semantically realistic scenes at substantial scale, but reconstruction artifacts and annotation constraints remain. Replica trades smaller scale for higher color, geometry, and semantic resolution plus HDR textures and reflectors.
- Matterport3D, Gibson, and Stanford 2D-3D-S provide semantically realistic reconstructions, with some containing hundreds of scenes.
- Geometry artifacts and lighting problems occur throughout Matterport-scanner-based reconstruction datasets.
- Matterport3D contains 90 houses, 2,056 rooms, 50,811 object instances, and 40 semantic classes.
- Matterport3D and ScanNet annotations use Felsenszwalb segmentation, whose extracted boundaries can be inaccurate between objects.
- Replica is smaller but provides high color, geometry, and semantic resolution together with HDR textures and renderable reflectors.
III. DATASET CREATION
Replica is built from synchronized RGB-D capture and reconstruction pipelines that produce posed, textured 3D geometry with high-dynamic-range appearance.
- Capture: A custom RGB-D rig captures time-aligned IMU, RGB, infrared, and wide-angle greyscale sensor data.The wide-angle greyscale stream and IMU support pose estimation for reconstruction.
- Reconstruction: Depth and pose measurements are fused into a TSDF, then converted into meshes, simplified, and textured.Meshes are extracted with Marching Cubes, simplified with Instant Meshes, and textured using a PTex-like system.
- Appearance: HDR textures combine exposures into 16-bit floating-point RGB texels with an overall dynamic range of about 85,000:1.This corresponds to more than 16 f-stops, compared with 8-bit RGB encoding in other datasets.
A. Mesh and Reflector Fixing
Replica improves mesh completeness and realism by manually correcting reflective planar surfaces and filling scanning holes through an assisted repair workflow.
- Reflector and hole fixing: Planar reflective surfaces and small holes missed during scanning are manually corrected to improve mesh quality.Reflectors are represented as planar polygons whose boundaries are specified in custom software.
- Reflector and hole fixing: Closed boundary-edge cycles identify holes automatically before annotators select holes for algorithmic filling.CGAL triangulates each boundary into an initial patch that is then refined and smoothed.
- Reflector and hole fixing: The hole-filling workflow uses Liepa’s approach to generate, refine, and smooth patches.The resulting filled regions are illustrated in Fig. 5.
B. Semantic Annotation
Replica combines 2D instance annotation with mesh-level refinement and represents the result as a hierarchical segmentation forest supporting multiple segmentation views.
- Annotation workflow: Rendered mesh images are annotated with 2D instance masks, then fused back onto the mesh using voting.The rendering set ensures every mesh primitive is observed at least once.
- Annotation workflow: The fused 3D annotations are refined with neighborhood-based segmentation and corrected down to individual mesh primitives.A 3D painting tool supports review, refinement, and correction after fusion.
- Privacy: Replica also marks areas requiring anonymization through blurring or pixelation to protect privacy.This annotation is performed as part of the semantic-labeling process.
- Segmentation forest: The segmentation forest organizes primitives into segments and semantic object entities, with each tree representing one semantic instance.Rendering different forest levels yields different segmentations, while same-class instances can be rendered as a class segmentation.
IV. DATASET DESCRIPTION
Replica provides 18 varied indoor reconstructions with dense, richly textured geometry, reflectors, semantic labels, and tooling for realistic rendering and embodied-AI research.
- Scene collection: Replica contains 18 scenes spanning apartments, offices, a house, multi-room spaces, a hotel room, and apartment rooms.Six FRL apartment setups represent the same space at different points in time.
- Scene contents: Each scene includes dense geometry, high-resolution HDR textures, reflectors, and semantic class and instance annotations.The annotations are designed for high fidelity and accurate object boundaries.
- Scene contents: Glass and mirror information can be rendered to add realism and photometric accuracy.Reflector data is part of the released scene representation.
- Rendering realism: Renderings from the same pose can closely resemble raw RGB captures, with small artifacts and absent motion blur distinguishing them.The comparison qualitatively underscores the realism of the reconstructions.
- Semantic variety: The dataset includes structural elements, common objects, and small entities such as wall plugs, cups, and coasters.Its semantic classes exceed those of several common datasets, with straightforward mappings to other class lists.
- Tooling: The release includes a minimal C++ SDK, while AI Habitat can render RGB, depth, semantic-instance, and semantic-class images directly into PyTorch tensors.The Habitat integration supports deep-learning applications and rendering at up to 10k frames per second.
A. Data Organization
Each Replica scene is organized as a dense geometric reconstruction with HDR textures, reflector descriptions, semantic and planar segmentation, and AI Habitat exports.
- mesh.ply encodes the scene’s dense quad surface and includes per-vertex colors for low-resolution, non-HDR rendering.
- HDR appearance is stored in high dynamic range PTex files under textures/*.
- glass.sur represents scene reflectors using transforms, polygons, surface normals, and reflectance values distinguishing mirrors from partially transparent glass.
- semantic.json and semantic.bin provide semantic segmentation, while preseg.json and preseg.bin encode planar/non-planar segmentation.
- Habitat exports include meshes with semantic instance ids, class mappings, and navmesh occupancy information for agent simulation.
- The semantic and preseg files use segmentation forests whose nodes have class names, parent-child links, unique ids, and associated primitive ids.
V. CONCLUSION
Replica provides high-resolution, realistic 3D reconstructions with HDR textures, reflectors, and semantic annotations, supporting machine learning beyond static image collections.
- Replica combines high texture, geometry, and semantic resolution with HDR textures and renderable reflector information.
- Its realistic scenes support AI-agent and machine-learning research requiring data beyond static image datasets.
- Replica can serve as a generative model for benchmarking 3D perception systems, including SLAM and dense reconstruction.
- The dataset also facilitates research into augmented- and virtual-reality telepresence.