Source-linked AI summary
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon-Young Lee, Hailin Jin, Thomas Funkhouser
TL;DR
Indoor scene understanding is constrained by scarce and noisy per-pixel ground truth, while prior synthetic-data studies did not systematically evaluate context and rendering. The paper constructs a large synthetic dataset from realistic indoor scenes, studies rendering and lighting across three tasks, and finds that physically based pretraining improves state-of-the-art performance.
Problem
Real indoor datasets are scarce and noisy, and prior synthetic-data work lacked systematic evidence about indoor context and rendering choices.
Method
The paper creates a 500K-image dataset from 45K human-designed 3D houses and trains convolutional networks with synthetic pretraining followed by real-data finetuning.
Results
Pretraining improves state-of-the-art performance on normal prediction, semantic segmentation, and object boundary detection, with physically based realistic lighting and soft shadows superior to other rendering methods.
Takeaways & Limitations
Realistic scene context, lighting, and physically based rendering provide useful synthetic training data for indoor scene understanding.
Abstract
from arXiv · showhide
Indoor scene understanding is central to applications such as robot navigation and human companion assistance. Over the last years, data-driven deep neural networks have outperformed many traditional approaches thanks to their representation learning capabilities. One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection. To address this problem, a number of works proposed using synthetic data. However, a systematic study of how such synthetic data is generated is missing. In this work, we introduce a large-scale synthetic dataset with 400K physically-based rendered images from 45K realistic 3D indoor scenes. We study the effects of rendering methods and scene lighting on training for three computer vision tasks: surface normal prediction, semantic segmentation, and object boundary detection. This study provides insights into the best practices for training with synthetic data (more realistic rendering is worth it) and shows that pretraining with our new synthetic dataset can improve results beyond the current state of the art on all three tasks.
1. Introduction
Indoor scene understanding needs abundant, accurate per-pixel data, but real datasets are scarce and noisy. The paper introduces a large synthetic dataset and studies how context, lighting, and rendering affect three prediction tasks.
- Motivation: Real indoor datasets are limited by collection requirements, human annotation effort, sensor noise, and labeling errors.The NYUv2 normal-prediction dataset cited here contains 1449 images.
- Motivation: Synthetic 2D render pairs address data scarcity, but prior work lacked systematic studies of indoor scene context and rendering choices.Earlier studies often used independent 3D-object repositories and unrealistic rendering for efficiency.
- Contribution: The paper introduces a 500K-image dataset generated from 45K human-designed 3D houses with realistic contextual support structures, objects, and lights.Access to source models enables dense per-pixel training data for multiple tasks.
- Study design: The study examines lighting, rendering methods, and object context for surface normal prediction, semantic segmentation, and object edge detection.The dataset supports controlled manipulation of viewpoints, context, lighting, and shading.
- Results: Pretraining with the dataset followed by real-data finetuning improves state-of-the-art performance across all three tasks, while physically based rendering with realistic lighting and soft shadows outperforms alternatives.The paper attributes the rendering advantage to context-dependent effects such as realistic illumination and soft shadows.
2. Background
Prior synthetic-data work expanded training for several vision tasks but often used limited context, unrealistic rendering, or incomplete scene supervision. This paper differentiates its approach through large, human-designed indoor scenes and dense full-scene labels.
- Synthetic data: Synthetic data has been used for object pose, optical flow, semantic segmentation, and object-feature learning.These applications generate training examples from rendered or modified 3D content.
- Prior rendering: Earlier object-rendering approaches used arbitrary backgrounds or motion and often omitted physically based effects such as shadows and reflections.Environment-map lighting was reported to benefit pose estimation, but the cited approaches did not provide the same full indoor-scene setting.
- Prior datasets: Handa et al.’s scene dataset contained rooms on the order of tens, limiting contextual variation relative to this paper’s 45K realistic house layouts.Its scene descriptions also lacked colors and surface materials, restricting outputs mainly to depth channels.
- Prior datasets: Depth-image object replacement methods cannot guarantee orientation or stylistic consistency with surrounding objects, unlike human-created indoor scenes designed for context relevance.The cited contrast concerns both scene diversity and contextual alignment.
- Prior datasets: The 90K-image object-2D database lacks room structures and cannot provide full-scene per-pixel ground truth, whereas this work can generate arbitrarily many paired renders and labels.The limitation follows from its 3D data containing only isolated objects.
- Prior datasets: Realistic game-engine data preserves contextual scenes for semantic segmentation, but its per-pixel label collection process is not completely automated.The cited game-engine setting is also limited to outdoor context.
3. Data
The dataset is built from corrected, human-designed indoor scenes and rendered from diverse camera viewpoints under controlled lighting and rendering configurations. The pipeline also filters training images and produces dense labels for three tasks.
- 3. Data: SUNCG provides 45,622 scenes, over 5M object instances, 2,644 unique objects, and 84 categories with human-designed layouts, alignments, and materials.The authors correct geometry and material deficiencies, including missing solid walls and emissive lighting surfaces, for physically based rendering.
- 3.1. Camera Sampling: Camera selection seeks diverse contextual views by sampling viewpoints, directions, heights, and tilts while excluding positions within 10cm of obstacles.The process first selects views across six horizontal direction sectors in each room.
- 3.2. Image Rendering: The authors render selected cameras with four algorithm-and-lighting combinations spanning fast OpenGL directional lighting and physically based Mitsuba rendering with local lights.This design enables controlled comparison of rendering realism and illumination.
- 3.2. Image Rendering: OPENGL-DL uses a camera-aligned headlight and two diagonal directional lights without local illumination, shadows, or indirect illumination.It represents the fast, less physically realistic end of the rendering range.
- 3.2. Image Rendering: OPENGL-IL adds point and spot lights approximating indoor appliances but still excludes shadows and indirect illumination.The method selects the best eight light sources per object by illumination intensity.
- 3.2. Image Rendering: MLT-OL uses high-definition spherical sky panoramas cast through transparent windows to model outdoor illumination naturally.The physically based setup is designed to generate photorealistic rendering.
- 3.3. Image Selection and Labels: The pipeline filters images for similarity to NYUv2 in color and depth distributions, then generates per-pixel normals, semantic labels, and object boundaries from the full 3D models.The selection removes examples with insufficient lighting or atypical depth distributions.
4. Indoor Scene Understanding Tasks
The paper evaluates synthetic-data pretraining for surface normal estimation, semantic segmentation, and object boundary detection, comparing rendering methods, context, lighting, and fine-tuning on NYUv2. Across these tasks, physically based rendering and realistic lighting generally provide stronger results, while synthetic pretraining improves performance and can sharpen predictions.
- Experimental scope: The study compares synthetic-data training protocols across surface normal estimation, semantic segmentation, and object boundary detection.Experiments include different rendering conditions, object context, lighting, and fine-tuning on NYUv2.
- 4.1. Normal Estimation: MLT pretraining followed by NYUv2 fine-tuning achieves the best normal-estimation performance and outperforms the state of the art.MLT pretraining also significantly outperforms OpenGL pretraining without fine-tuning.
- 4.1. Normal Estimation: Image selection improves normal-estimation performance over using all rendered images, while combined indoor and outdoor lighting outperforms outdoor lighting alone.These comparisons indicate that training-image quality and indoor illumination matter for pretraining.
- 4.1. Normal Estimation: Normal predictions from MLT-pretrained models show sharper edges and more local details, while substantial angular error concentrates on walls with noisy ground truth.Synthetic training can outperform or correct NYUv2 ground truth in large flat regions.
- 4.2. Semantic Segmentation: Synthetic pretraining improves semantic segmentation over direct NYUv2 training, and physically based rendering outperforms OpenGL rendering for color-based models.Color information is especially important for the evaluated 40-class segmentation task.
- 4.3. Object Boundary Detection: Synthetic-data pretraining consistently improves all object-boundary evaluation metrics, with MLT rendering achieving the best performance.Pretraining sharpens true object boundaries while suppressing edges within objects and in the background, though it is more sensitive to noise before fine-tuning.
5. Conclusion
The paper introduces a large-scale synthetic dataset of contextually meaningful indoor scenes with varied lighting and rendering settings, showing that physically based pretraining improves indoor scene understanding.
- 5. Conclusion: 500K rendered images come from contextually meaningful 3D indoor scenes with different lighting and rendering settings.The dataset also includes the indoor scene models used for rendering.
- 5. Conclusion: Pretraining with physically based rendering and realistic lighting boosts performance on indoor scene understanding tasks.The reported improvement is relative to state-of-the-art methods.