PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets

Zhengqin Li, Ting-Wei Yu, Shen Sang, Sarah Wang, Meng Song, Yuhan Liu, Yu-Ying Yeh, Rui Zhu, Nitesh Gundavarapu, Jia Shi, Sai Bi, Zexiang Xu, Hong-Xing Yu, Kalyan Sunkavalli, Miloš Hašan, Ravi Ramamoorthi, Manmohan Chandraker

arXiv:2007.12868v3cs.CV

TL;DR

Indoor scene appearance entangles geometry, materials, and lighting, yet high-quality ground truth—especially for spatially varying materials and lighting—is difficult to acquire. OpenRooms creates photorealistic scenes from real scans with extensive geometry, material, lighting, and semantic labels. Networks trained on the dataset achieve accurate inverse-rendering results on real images, and the framework also supports segmentation, augmented-reality demonstrations, and physics-engine integration.

  • Problem

    High-quality ground truth for indoor scenes is difficult to acquire, particularly for complex spatially varying materials and lighting.

    Source: §1 · p. 1

  • Method

    OpenRooms assigns geometry, spatially varying materials, lighting, and semantic labels to real indoor scans and renders them as a photorealistic dataset.

    Source: §1 · p. 1

  • Results

    Networks trained on OpenRooms achieve accurate inverse-rendering results on real images, and the dataset also supports semantic segmentation and joint shape, material, and semantic estimation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OpenRooms assets can be integrated with PyBullet, including mapping SVBRDFs to friction coefficients for navigation and rearrangement studies.

    Source: §1 · p. 2

    Depth and normal estimation on NYU performs slightly worse than Li et al.’s dataset, possibly because its SUNCG-based data has more diverse and complex geometry than OpenRooms’ ShapeNet-based furniture.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

We propose a novel framework for creating large-scale photorealistic datasets of indoor scenes, with ground truth geometry, material, lighting and semantics. Our goal is to make the dataset creation process widely accessible, transforming scans into photorealistic datasets with high-quality ground truth for appearance, layout, semantic labels, high quality spatially-varying BRDF and complex lighting, including direct, indirect and visibility components. This enables important applications in inverse rendering, scene understanding and robotics. We show that deep networks trained on the proposed dataset achieve competitive performance for shape, material and lighting estimation on real images, enabling photorealistic augmented reality applications, such as object insertion and material editing. We also show our semantic labels may be used for segmentation and multi-task learning. Finally, we demonstrate that our framework may also be integrated with physics engines, to create virtual robotics environments with unique ground truth such as friction coefficients and correspondence to real scenes. The dataset and all the tools to create such datasets will be made publicly available.

1. Introduction

High-quality ground truth for indoor scenes is difficult to acquire, especially for spatially varying materials and lighting, while plausible synthetic datasets are also non-trivial to create. OpenRooms builds photorealistic, scan-based scenes with extensive ground truth and tools intended to support research across vision, graphics, and robotics.

1. Introduction

  • Accurate ground truth for indoor geometry, materials, and lighting is hard to acquire, and plausible synthetic indoor datasets are also difficult to create.Geometry can be scanned, but complex spatially varying materials and lighting are particularly challenging to capture.Source: §1 · p. 1
  • OpenRooms assigns physically based spatially varying materials and lighting to real indoor scans, and provides tools for creating photorealistic datasets.Its lighting ground truth includes windows, environment maps, area lights, visibility, shadows, and inter-reflections.Source: §1 · p. 1

2. Related Work

  • OpenRooms targets gaps in prior work by building from scans rather than artist-created assets and providing extensive material and lighting ground truth with publicly available tools.Prior work aligned CAD models to scanned point clouds, while another dataset with complex material and spatially varying lighting annotations used artist-created assets.Source: §1 · p. 1, §2 · pp. 2–3, Table 1 · p. 3

1. Introduction

  • The dataset contains over 100K HDR images with geometry, SVBRDF, per-pixel lighting and visibility, and semantic labels, and can be extended using the released tools.The framework builds on ScanNet scans, CAD alignment, reflectance, and illumination repositories.Source: §1 · p. 1
  • Networks trained on OpenRooms achieve accurate inverse-rendering results on real images and support semantic segmentation, multi-task learning, and photorealistic AR examples.Demonstrated AR applications include object insertion and material editing.Source: §1 · p. 2
  • The assets can be integrated with PyBullet, where SVBRDFs are mapped to friction coefficients for studies of navigation and rearrangement.The authors also note a one-to-one correspondence between real videos and simulations.Source: §1 · p. 2, §2 · p. 2

3. Building a Photorealistic Indoor Dataset

OpenRooms turns indoor scans into photorealistic synthetic scenes by creating consistent geometry, assigning spatially varying materials and lighting, and rendering varied views. Its dataset includes extensive semantic and lighting supervision across more than 118,000 images.

3.1. Creating CAD Models from 3D Scans

  • Room layouts are annotated from fused scan point clouds, while aligned CAD furniture is adjusted to avoid floating objects and intersections.Room height is set to 3 meters because ScanNet views generally do not cover ceilings; initial furniture poses come from Scan2CAD.Source: §3.1 · p. 3

3.2. Assigning Complex Materials to Indoor Scenes

  • Furniture parts receive semantically informed materials using automatic UV mapping and PartNet segmentation, with annotators assigning categories to each part.The material tool groups SVBRDFs into appearance-based categories and allows over-segmented parts to be merged.Source: §3.2 · pp. 3–4

3.3. Ground Truth Lighting for Indoor Scenes

  • Lighting supervision covers visible and invisible light sources, per-pixel direct and indirect illumination, and each source’s direct shading and visibility.Windows use rectangular planes with environment maps; lamps use 3D bounding boxes, and visibility is computed from direct shading with and without occlusion.Source: §3.3 · pp. 4–5

3.4. Rendering with a Physically-based Renderer

  • A GPU-accelerated physically based renderer models up to seven inter-reflection bounces, while view selection favors broad room coverage over scan-biased viewpoints.The assets can also be used in other rendering environments, including real-time engines that do not require fully accurate global illumination.Source: §3.4 · p. 5

3.5. OpenRooms Dataset Statistics

  • 1,287 ScanNet scenes yield 118,233 HDR images at 480 × 640 resolution, with labels for 44 CAD-model classes.The split includes 108,159 training images and 10,074 validation images; class distribution follows real ScanNet indoor scenes.Source: §3.5 · p. 5
  • The dataset uses 1,075 SVBRDFs across nine categories and provides spatially varying environment maps and shading with direct-only and direct-plus-indirect illumination.Every rendered image has at least one light source on; the dataset also includes parameterized visible and invisible sources and their direct contributions and visibility maps.Source: §3.5 · p. 6

4. Applications

OpenRooms supports inverse-rendering and semantic tasks with ground truth that transfers to real-image applications. Its scene assets also support physics-based robotics environments with material and lighting variation.

4.1. Inverse Rendering

  • Intrinsic-decomposition results are comparable to prior state of the art, while NYU depth and normal estimation performs slightly worse than Li et al.’s dataset.The authors suggest the latter difference may reflect more diverse and complex geometry in SUNCG than in their ShapeNet-based furniture.Source: §4.1 · p. 7, Table 2 · p. 7, Table 3 · p. 7
  • Training on OpenRooms boosts NYUv2 light-source detection accuracy by around 5% for both bounding-box regression and segmentation.The task detects windows and lamps.Source: §4.1 · p. 7, Table 4 · p. 7, Figure 10 · p. 8
  • The inverse-rendering network predicts per-pixel spatially varying lighting, and material editing handles spatially varying lighting effects and specularity well.OpenRooms also provides ground truth for direct lighting, shading, and visibility, which the network does not predict.Source: §4.1 · p. 7, §4.2 · pp. 7–8, Figure 14 · p. 8
  • OpenRooms pretraining yields semantic-segmentation results comparable to InteriorNet, with improvements as training-image counts increase for both datasets.Models are fine-tuned and evaluated on NYUv2 with 40 labels.Source: §4.1 · p. 7, Table 5 · p. 7, Figure 11 · p. 8
  • OpenRooms’ inverse-rendering and semantic ground truth supports multi-task learning; the paper illustrates this by adding a segmentation head to its inverse-rendering network.Quantitative results are reported in the supplementary material.Source: §4.1 · p. 7, Figure 12 · p. 8
  • On object insertion, the OpenRooms-trained network outperforms methods that cannot handle spatially varying or high-frequency lighting and performs better in a quantitative user study.It also generates more consistent lighting than, possibly because OpenRooms has more diverse outdoor lighting and physically meaningful indoor lighting.Source: §4.2 · p. 7, Table 6 · p. 7, Figure 13 · p. 8

4.3. Applications to Robotics and Embodied Vision

  • OpenRooms scenes can become interactive PyBullet environments with changeable furniture, lighting, and materials, supporting navigation and rearrangement studies with correspondence to real scenes.The paper inserts an iGibson robot into a classroom scenario and notes potential use for sim-to-real transfer.Source: §4.3 · p. 8, Figure 15 · p. 9, Figure 17 · p. 9
  • Friction coefficients are derived from albedo and roughness ground truth; specular materials have lower coefficients, and different friction values lead to different pushing outcomes.Figure 16 illustrates per-pixel friction coefficients for the same scene with different materials.Source: §4.3 · pp. 8–9, Figure 16 · p. 9

5. Conclusion and Future Work

The conclusion presents OpenRooms as a user-generated framework for photorealistic indoor datasets, demonstrated on more than 1,000 ScanNet scenes. It combines detailed ground truth with applications in inverse rendering, segmentation, augmented reality, and physics-engine integration.

5. Conclusion and Future Work

  • The framework is demonstrated on over 1,000 indoor scenes from ScanNet and starts from public repositories of scans, shapes, and materials.The dataset and creation tools are planned for public release.Source: §5 · p. 9
  • OpenRooms provides ground truth for complex materials and spatially varying lighting, including direct and indirect illumination, light sources, per-pixel environment maps, and visibility.Source: §5 · p. 9
  • The paper demonstrates inverse-rendering and segmentation training, augmented-reality applications, and physics-engine integration with friction coefficients for future robotics studies.The conclusion identifies navigation, rearrangement, and sim-to-real transfer as potential future studies.Source: §5 · p. 9

A. Summary of the Supplementary Material

The supplementary material collects further dataset and tool details, results, comparisons, demonstrations, and technical descriptions.

A. Summary of the Supplementary Material

B. Video

The accompanying video illustrates OpenRooms’ dataset-creation process and applications, while supplementary user studies report comparisons for object insertion.

B. Video

  • The video illustrates synthetic versions of acquired scans, ground truth for materials, lighting, semantics, instances, and friction, plus inverse-rendering, editing, and robotics applications.Source: B. Video · p. 9
  • In the supplementary user studies, the trained network outperforms previous state-of-the-art methods in comparisons against ground-truth lighting and in pairwise object-insertion comparisons.The lighting study’s ideal performance is 50%.Source: Table 7 · p. 10, Table 8 · p. 10
  • The video also demonstrates dataset-creation steps and notes that the tools are publicly released as part of the framework.Source: B. Video · p. 10

C. Applications: Photorealistic Image Editing

OpenRooms-trained inverse-rendering networks support photorealistic editing on real images, with the object-insertion user study outperforming prior methods. Further evaluations report benefits from cascade refinement and show applications and limitations across lighting, normals, and layout prediction.

C. Applications: Photorealistic Image Editing

  • 38.89% for OpenRooms-trained object insertion outperforms (34.84%) and (29.15%) on measured real-image lighting.The study uses 20 images and reports 50% as ideal indistinguishability; the authors conclude object insertion performs better than methods built on previous datasets.Source: C. Applications: Photorealistic Image… · p. 10
  • Per-pixel environment maps recover spatially varying highlights when replacing a real image’s material with a glossy material.The per-pixel lighting representation captures complex spatial variation in illumination.Source: C. Applications: Photorealistic Image… · p. 10

D. Inverse Rendering Trained on OpenRooms

  • Cascade refinement and the bilateral solver improve most intrinsic-component predictions on the synthetic test set, with refinement removing noise and recovering high-frequency lighting and normal signals.The bilateral solver further reduces noise by enhancing the smoothness prior.Source: D. Inverse Rendering Trained on… · p. 10, Table 9 · p. 10
  • On real images, the synthetic-trained network generalizes well, and cascade refinement and bilateral solving are more apparent, likely because initial predictions are noisier.Source: D. Inverse Rendering Trained on… · p. 10
  • OpenRooms pre-training benefits light-source detection for both windows and lamps, while cascade structure makes per-pixel lighting predictions sharper and less noisy.The lighting predictions are reported as consistent with spatially varying intensity and ground-truth light-source position.Source: D. Inverse Rendering Trained on… · pp. 10–11
  • On real examples, both and the OpenRooms network produce less noisy normals than, but the OpenRooms network can oversmooth them.The authors link possible oversmoothing to Scan2CAD-derived scenes with few large furniture items and suggest adding small objects to increase scene complexity.Source: D. Inverse Rendering Trained on… · p. 11
  • Floor-SP trained on 1,069 newly annotated ScanNet scenes performs better on noisy scans, but incomplete-room annotations can depress corner and edge evaluation while leaving room-segmentation IoU unaffected.False corners and edges may be needed to close layouts when scans do not cover the full room.Source: Table 10 · p. 11, D. Inverse Rendering Trained on… · p. 12

E. Ground Truth for Friction Coefficients

OpenRooms maps its microfacet BRDF materials to friction coefficients by rendering reflectance disks and transferring labels through deep reflectance codes. The resulting examples associate specular materials with smaller friction coefficients.

Reflectance disks

  • Rendered reflectance disks match captured disks in the distribution of specular highlights and intensities.The virtual capture mimics the reflectance-disk setup of Zhang et al..Source: E. Ground Truth for Friction… · pp. 12–13

Deep reflectance codes

  • A pretrained network maps reflectance disks to low-dimensional deep reflectance codes, which are mapped to friction coefficients using nearest-neighbor search.The paper builds a lookup table from sampled microfacet BRDF parameters to friction coefficients, with bilinear interpolation or nearest-neighbor search.Source: E. Ground Truth for Friction… · p. 13
  • Specular materials are more likely to have small predicted friction coefficients.Source: E. Ground Truth for Friction… · p. 13

F. Applications: Robotics, Embodied Vision

OpenRooms scenes can be converted into interactive PyBullet environments, with material-derived friction and correspondence to real scenes. Demonstrations cover navigation and pushing tasks under varied scene properties.

Integrating OpenRooms with PyBullet

  • Each scene object becomes a URDF link with separate rendering and collision meshes, and albedo and roughness are used to estimate friction for interactive physics simulation.Mass, center of mass, and inertial matrix can also be specified in the URDF or physics engine.Source: F. Applications: Robotics, Embodied… · p. 14

Qualitative examples

Navigation

  • The navigation example provides position-based states and forward/backward and turning actions, plus RGB, normals, depth, semantic, and joint-state observations.The setup supports navigation studies under different material properties and lighting conditions.Source: F. Applications: Robotics, Embodied… · p. 15

Pushing with different frictions

  • The pushing experiment compares carpet and wax floors while keeping other conditions the same, with the cabinet’s post-push position offset used for comparison.The floor friction map is derived from albedo and roughness and averaged for the physics engine; the dataset’s spatially varying friction ground truth is not yet supported by popular simulators such as PyBullet.Source: F. Applications: Robotics, Embodied… · p. 15

G. Segmentation, Multi-Tasking, Adaptation

The paper evaluates OpenRooms labels and models for semantic and instance segmentation, joint shape-material-semantic estimation, and depth domain adaptation. It also demonstrates scene reconstruction from SUNRGBD scans.

Semantic segmentation

Instance segmentation

Multi-task learning

  • The multi-task model jointly estimates albedo, normals, depth, roughness, and semantics, with competitive intrinsic estimates relative to the model without a segmentation branch.Table 15 also compares a segmentation-only model and reports multiple losses and segmentation metrics.Source: G. Segmentation, Multi-Tasking,… · p. 17, Table 15 · p. 19

Domain Adaptation for depth estimation

  • Unsupervised depth adaptation uses 100,000 OpenRooms images as source and 15,000 unlabeled NYU frames as target; the authors report that adaptation largely addresses the domain gap.The target-supervised comparison uses 1,440 NYUv2 training frames.Source: G. Segmentation, Multi-Tasking,… · p. 18

SUNRGBD scene reconstruction

  • SUNRGBD reconstruction handles partial, sparse scans by grid-searching CAD-model poses against point clouds, and produces plausible material and lighting appearances across viewpoints.The authors report that scan-quality differences require dataset-specific geometry handling, while material and lighting mapping transfers across datasets with minimal effort.Source: H. Dataset Creation using SUNRGBD Data · p. 18

I. Microfacet BRDF Model

The section defines a simplified microfacet BRDF using diffuse albedo, surface normal, and roughness. It also specifies view- and light-direction terms and the distribution, Fresnel, and geometry components.

I. Microfacet BRDF Model

  • The simplified microfacet BRDF uses diffuse albedo A, normal N, and roughness R as its material parameters.The model is attributed to.Source: §I · p. 19
  • The Fresnel term is given as F(v, h), and the geometry term as G(l, v, N, R) = G1(v, N)G1(l, N).Source: §I · p. 19
  • The geometry component defines G1(v, N) using N · v and k, with k set from roughness R.Source: §I · p. 19
  • The view direction v and light direction l define the half angle vector h used by the BRDF terms.Source: §I · p. 19
  • The model sets F0 = 0.05, following; the cited passage does not specify assignments for D(h,R) or F(v,h), so those mappings are omitted.Source: §I · p. 19

J. Ground Truth for Lighting

OpenRooms provides detailed lighting supervision, including per-light contributions with direct and indirect illumination and visibility separated. Its renderer supports this supervision and uses combined sampling to capture radiance from small light sources efficiently.

J. Ground Truth for Lighting

  • OpenRooms provides per-light ground truth for source positions, colors, and intensities, plus pixel contributions with direct and indirect illumination and visibility handled separately.These supervisions support modeling scene appearance with individual light sources turned on or off.Source: J. Ground Truth for Lighting · p. 19, Figure 37 · p. 21
  • For per-light shading from windows, the renderer uses each window’s plane parameters and counts environment-map rays that pass through its plane approximation.This permits isolating a window’s contribution when multiple windows are present.Source: J. Ground Truth for Lighting · p. 20
  • Rendering without the visibility term is implemented by omitting OptiX’s rtTrace ray-occlusion test.Source: J. Ground Truth for Lighting · p. 20

K. Physically-Based GPU Renderer

  • The renderer combines uniform hemisphere sampling with light-source sampling using multiple importance sampling, capturing small-source radiance with fewer samples.Source: K. Physically-Based GPU Renderer · p. 20
  • Figure 36 shows scanned SUNRGBD scenes reconstructed from different views with different material assignments.Source: Figure 36 · p. 21

Related papers

Loading 2007.12868v3…