Source-linked AI summary
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, Joshua M. Susskind
TL;DR
Real images often lack per-pixel ground truth for fundamental scene understanding tasks. Hypersim addresses this gap with a photorealistic synthetic dataset and a pipeline for producing richly annotated indoor scenes, and pre-training on it significantly improves transfer to real-world semantic segmentation and 3D shape prediction.
Problem
Per-pixel ground truth labels are difficult or impossible to obtain from real images, while existing synthetic datasets commonly lack public 3D assets, semantic instance labels, or disentangled lighting components.
Method
Hypersim uses professional-art-created indoor scenes, a view-sampling and cloud-rendering pipeline, and interactive mesh annotation to produce images with geometry, dense labels, and disentangled image representations.
Results
Pre-training on Hypersim significantly improves semantic segmentation on NYUv2 and 3D shape prediction on Pix3D, achieving state-of-the-art performance on the most challenging Pix3D test set.
Takeaways & Limitations
The dataset combines publicly available 3D assets, semantic instance segmentations, complete scene information, and disentangled image factors for geometric, multi-task, and inverse-rendering research.
Takeaways & Limitations
Existing synthetic datasets and simulators do not collectively provide all of the public assets, semantic labels, geometry, and disentangled representations targeted by Hypersim.
Abstract
from arXiv · showhide
For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations and complete camera information for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, computation time, and annotation effort. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a popular open-source natural language processing model. We also evaluate sim-to-real transfer performance on two real-world scene understanding tasks - semantic segmentation and 3D shape prediction - where we find that pre-training on our dataset significantly improves performance on both tasks, and achieves state-of-the-art performance on the most challenging Pix3D test set. All of our rendered image data, as well as all the code we used to generate our dataset and perform our experiments, is available online.
1. Introduction
Hypersim addresses missing and fragmented supervision in photorealistic indoor scene datasets by combining public 3D assets, dense labels, complete scene information, and disentangled image factors. Its pipeline generates annotated views efficiently, and pre-training on the resulting dataset improves transfer to real-world scene understanding tasks.
- Motivation: Existing synthetic datasets often lack publicly available 3D assets, semantic instance segmentations, or disentangled lighting and shading components.These gaps limit direct 3D supervision, semantically meaningful segmentation, and inverse-rendering applications.
- Contribution: Hypersim contains 77,400 images from 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry.The dataset uses publicly available 3D assets and includes complete geometry, materials, lighting, camera information, and semantic instance segmentations.
- Contribution: Each image is factored into diffuse reflectance, diffuse illumination, and a non-diffuse residual capturing view-dependent lighting effects.The representation supports glossy and specular effects alongside the diffuse components.
- Pipeline: The computational pipeline samples camera views, renders images in the cloud, and propagates interactive mesh annotations to rendered images.Rendering can proceed before mesh annotation is complete, and scenes can be re-annotated without re-rendering.
- Results: Pre-training on Hypersim significantly improves semantic segmentation and 3D shape prediction transfer, including state-of-the-art performance on the most challenging Pix3D test set.The paper also analyzes dataset properties and generation costs, with data and code released online.
2. Related Work
Prior work provides synthetic imagery, object-aligned geometry, intrinsic decompositions, view sampling, and annotation tools, but Hypersim targets their combination for complete indoor scenes. Its scene-level assets and labels support supervision unavailable from many existing alternatives.
- Synthetic data: Synthetic data supports computer vision, while nonphotorealistic environments trade rendering efficiency for a larger real-to-synthetic image domain gap.The related work distinguishes photorealistic indoor datasets from faster nonphotorealistic environments.
- 3D supervision: Existing real-image datasets may align CAD models imperfectly and leave visible objects without corresponding models, whereas Hypersim aligns segmented 3D models to every pixel.This provides exact image-to-3D correspondence at scene scale.
- Intrinsic representations: Prior intrinsic factorization work renders individual ShapeNet objects, while Hypersim applies the same diffuse, illumination, and residual decomposition to entire scenes.The distinction is object-level versus scene-level rendering.
- View sampling: Uniform random view sampling produces uninformative views in staged scenes, and existing data-driven methods do not directly fit Hypersim’s unlabeled parallel annotation setting.Hypersim therefore requires a view-sampling strategy suited to cluttered and partially empty scenes.
- Annotation: Interactive 3D annotation tools commonly use pre-segmentation strategies to reduce manual annotation effort.Hypersim’s annotation tool belongs to this broader family of mesh-labeling systems.
3. Data Acquisition Methodology
Hypersim’s scene collection prioritizes scale, photorealism, visual diversity, and consistent representations, then filters marketplace assets for complete, usable indoor environments. The resulting collection contains 461 scenes after quality and pipeline-compatibility checks.
- Scene selection: The scene search prioritized many assets, strong photorealism and visual diversity, and consistent file formats and internal representations.Because the goal was a static dataset, the authors accepted slower rendering in exchange for scale and photorealism.
- Source collection: The Evermotion Archinteriors Collection provided over 500 photorealistic indoor scenes in standalone V-Ray-compatible asset files.Its representation supported programmatic scene manipulation through V-Ray’s Python API.
- Filtering: Scenes depicting isolated objects, lacking royalty-free distribution, or failing rendering, mesh-export, or view-generation checks were excluded.The filtering process tested both asset usability and the ability to generate valid camera trajectories.
- Filtering: 461 scenes remained after excluding 107 of the 568 scenes in the Archinteriors collection.Twenty-one exclusions resulted from failures of the view sampling heuristic.
4. Computational Pipeline
Hypersim’s computational pipeline samples camera trajectories, renders scene views, and produces semantic annotations and geometric outputs. It combines triangle-density-aware view sampling, interactive mesh annotation, and post-processing across 461 scenes to generate 77,400 views.
- Pipeline overview: The pipeline processes each scene through camera-view generation, cloud rendering, and semantic annotation to produce images with ground-truth labels and geometry.Manual filtering decisions are recorded so the dataset can be reproduced exactly.
- Camera trajectory generation: Triangle-density-aware view saliency favors views observing finely tessellated salient objects while penalizing empty pixels.The model uses the number of unique observed triangles and the fraction of non-empty pixels.
- Camera trajectory generation: 774 camera trajectories, each containing 100 camera poses, produced 77,400 distinct views for rendering.Ten initially generated trajectories were manually removed to maintain valid viewing regions.
- Interactive mesh annotation: Interactive mesh annotation uses semantic-instance and semantic-label views with selection filters that support accurate labeling through rough painting gestures.The tool can group object parts into instances and apply labels without disturbing other scene regions.
- Annotation and post-processing: 461 scenes were annotated with instance-level NYU40 labels, then mesh annotations were propagated to images and tight 9-DOF instance boxes were generated.These boxes support direct application to 3D object detection problems.
5. Analysis
Hypersim contains diverse, cluttered indoor scenes with dense semantic annotations and geometric information, while its rendering and annotation costs are quantified. The dataset also supports sim-to-real evaluation on NYUv2 and Pix3D.
- Scenes: 127.3 objects per scene on average, exceeding ScanNet’s 14 and Replica’s 84 objects per scene.Scenes range from fewer than 0.1 to 11 million triangles.
- Segmentation Images: 88.3% of pixels have semantic class labels and 52.0% have semantic instance IDs.These rates are lower than GTA5’s 98.3% class-label coverage but higher than ScanNet’s 76%.
- Segmentation Images: 8.9 classes and 49.9 objects occur per image on average, while 51.5% of images contain at least 21 objects.The reported clutter supports the view-sampling heuristic’s ability to generate informative views.
- Sim-to-real Evaluation: Pre-training on Hypersim and fine-tuning on NYUv2 outperformed full-data training for both 13-class and 40-class segmentation.The comparison uses 25% of NYUv2 training data for 13 classes and 50% for 40 classes.
- Depth Images: 5.4 meters is the average depth, with depth values following a log-normal distribution.The surface-normal distribution is biased toward planar surfaces and disagrees with the surface-isotropy prior used by SIRFS for scene-centric decomposition.
- Rendering Costs: $57K total cost generated the dataset, requiring 231 vCPU years and 2.4 years of wall-clock time.This was 0.56× the reported $103K cost of training Megatron-LM from scratch.
- Sim-to-real Evaluation: Pre-training on Hypersim achieved state-of-the-art Pix3D performance on the S2 test set using APmesh, APmask, and APbox.The comparison is against Mesh R-CNN trained directly on Pix3D.
- Rendering Costs: $0.67 and 26 vCPU hours were the average per-image rendering cost and compute time at 1024×768 resolution.Rendering costs in dollars are linearly related to compute times, and both are log-normally distributed.
6. Experiments
The experiments test whether Hypersim pre-training transfers to real-world semantic segmentation and 3D shape prediction. The protocol pre-trains on Hypersim, fine-tunes on the corresponding real dataset, and evaluates on that dataset.
- Experimental Protocol: Pre-training on Hypersim, fine-tuning on a real dataset, and evaluation on that dataset define the sim-to-real protocol.The protocol is applied to semantic segmentation and 3D shape prediction.
- Semantic Segmentation: NYUv2 evaluation uses 795 training images, 654 test images, 13-class and 40-class tasks, and mIoU as the metric.RGB images are used for training and testing.
- Semantic Segmentation: +6.2 mIoU for 13-class and +5.0 mIoU for 40-class segmentation were obtained after Hypersim pre-training on NYUv2.These gains exceeded PBRS’s reported +1.6 mIoU for 40-class segmentation but were below SceneNet-RGBD’s reported +8.1 mIoU for 13-class segmentation.
- Semantic Segmentation: Hypersim’s 77K images and 0.5K scenes are substantially smaller than PBRS’s 568K images and 45K scenes and SceneNet-RGBD’s 5,000K images and 16K scenes.The authors attribute the competitive performance to increased photorealism.
- 3D Shape Prediction: Pix3D evaluation uses 10,069 images, 395 unique triangle meshes, the S2 test split, and APmesh, APmask, and APbox.S2 is identified as the most challenging Pix3D test set.
7. Conclusions
Hypersim combines photorealistic images, 3D assets, semantic instance segmentations, and disentangled image representations through a computational pipeline and annotation tool. Its measured costs and sim-to-real results support broader use of photorealistic synthetic data for scene understanding.
- Contributions: Hypersim combines images, 3D assets, semantic instance segmentations, and a disentangled image representation in one dataset.The dataset was created from synthetic scenes using a computational pipeline and interactive annotation tool.
- Cost Analysis: Generating the full dataset costs roughly half as much as training a popular open-source natural language processing model.The paper analyzes money, computation time, and annotation effort.
- Sim-to-real Transfer: Pre-training on Hypersim improves two real-world scene understanding tasks and achieves state-of-the-art performance on the most challenging Pix3D test set.The two tasks are semantic segmentation and 3D shape prediction.
- Implications: The authors identify geometric learning, multi-task learning, inverse rendering, automatic mesh segmentation, and generative modeling as supported application areas.They also propose co-designing rendering and learning algorithms to amortize rendering costs more effectively.