Source-linked AI summary
Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, Zihan Zhou
TL;DR
Structured 3D modeling needs large, accurate annotations for global structures, but manual labeling is costly and limited. Structured3D automatically derives unified structure annotations and photo-realistic renderings from professional interior designs, then uses them with real images for room layout estimation. The resulting training improves performance on public benchmarks.
Problem
Manual annotation of many structure instances is challenging and inefficient, and existing datasets have limited size, diversity, and structure-type coverage.
Method
Structured3D automatically extracts diverse 3D structures and their relationships from professional interior designs and generates photo-realistic renderings with industry-leading rendering engines.
Results
Using Structured3D with real images to train deep networks for room layout estimation improves performance on public benchmarks.
Takeaways & Limitations
The dataset supports structured 3D modeling research by providing unified annotations spanning multiple semi-global and global structures.
Takeaways & Limitations
Real room-layout benchmarks can contain labeling errors, such as non-cuboid rooms labeled as cuboid or unlabeled walls.
Abstract
from arXiv · showhide
Recently, there has been growing interest in developing learning-based methods to detect and utilize salient semi-global or global structures, such as junctions, lines, planes, cuboids, smooth surfaces, and all types of symmetries, for 3D scene modeling and understanding. However, the ground truth annotations are often obtained via human labor, which is particularly challenging and inefficient for such tasks due to the large number of 3D structure instances (e.g., line segments) and other factors such as viewpoints and occlusions. In this paper, we present a new synthetic dataset, Structured3D, with the aim of providing large-scale photo-realistic images with rich 3D structure annotations for a wide spectrum of structured 3D modeling tasks. We take advantage of the availability of professional interior designs and automatically extract 3D structures from them. We generate high-quality images with an industry-leading rendering engine. We use our synthetic dataset in combination with real images to train deep networks for room layout estimation and demonstrate improved performance on benchmark datasets.
1 Introduction
Structured3D addresses limitations of manually annotated structure datasets by automatically generating rich 3D annotations and photo-realistic images from professional interior designs. Its unified representation supports multiple related structures, and the dataset improves room layout estimation on public benchmarks.
- Local-feature-based reconstruction is sensitive to texture, repetition, lighting, and occlusion, while point-cloud models provide limited high-level 3D understanding.
- Manual structure annotation is laborious and error-prone, producing datasets with limited size, scene diversity, and coverage of multiple structure types.
- Its unified primitive-plus-relationship representation captures diverse structures and preserves relationships among them, including lines, wireframes, planes, floorplans, and room layouts.
- Structured3D automatically extracts rich 3D structure annotations from professional house designs and generates photo-realistic indoor renderings.
- The dataset contains annotations for 21,835 rooms in 3,500 scenes and more than 196k photo-realistic 2D renderings.
- Training deep networks with Structured3D improves room layout estimation performance on public benchmarks.
2 Related Work
Existing structured 3D datasets are constrained by manual annotation, limited scale, missing structure labels, or insufficient realism. Structured3D addresses these gaps with professionally designed scenes and high-quality rendering.
- Manual ground-truth annotation is a major reason existing structured 3D datasets contain no more than a few thousand images.
- The comparison framework covers object detection, scene understanding, image synthesis, and structured 3D modeling applications.
- Structured 3D scene modeling remains difficult because reliable detection of structures such as wireframes and room layouts from raw sensor data is still an active research problem.
- Synthetic indoor datasets such as SUNCG, SceneNet RGB-D, and InteriorNet provide scene diversity and many frames but lack ground-truth structure annotations.
- Some synthetic datasets also suffer from limited realism in their 3D models or 2D renderings.
3 A Unified Representation of 3D Structure
The paper represents structured 3D scenes as geometric primitives connected by relationships, reducing redundancy while preserving interactions among structures. This unified representation derives planes, wireframes, cuboids, Manhattan layouts, floorplans, semantic objects, and object abstractions.
- Unified representation: The unified representation models structure as relationships among primitives, preserving mutual relationships while reducing redundancy.Separate plane and wireframe representations can duplicate boundaries and lose their relationships.
- Primitives: Primitives include planes, lines formed by plane intersections, and junction points formed where lines meet.
- Relationships: Relationship matrices encode plane-line incidence and intersection, line-point incidence and intersection, and other structural organization.
- Specialized relationships: Cuboids and Manhattan worlds are represented through structured arrangements or groupings of plane primitives, while semantic objects group primitives hierarchically.A plane can participate in multiple relationship instances, including cuboids and Manhattan-world classes.
- Discussion: The listed primitives and relationships are not exhaustive and can be extended to parametric surfaces and other symmetry groups.
- Relation to existing models: The framework is geometry-focused and fine-grained, unlike semantic scene graphs that use object poses and coarse spatial relations.
- Derived models: The representation derives planes, wireframes, cuboids, Manhattan layouts, floorplans, and abstracted object shapes from the same primitives and relationships.Floorplans use floor-associated lines and junctions plus projected semantic labels for rooms, doors, and windows.
4 The Structured3D Dataset
Structured3D is a large-scale indoor-scene dataset combining ground-truth 3D room structures with realistic 2D renderings. Its unified primitive-and-relationship representation supports multiple structured modeling tasks and future extensions.
- 4 The Structured3D Dataset: The dataset represents structured scenes through geometric primitives and their relationships, with the goal of supporting holistic 3D scene understanding.The representation is designed to encode multiple structure types rather than a single task-specific annotation.
- 4 The Structured3D Dataset: Professional house designs are processed automatically to extract room primitives, including planes, lines, junctions, and their relationships.The source designs provide precise geometry and object information for ceilings, floors, walls, and openings.
- 4 The Structured3D Dataset: The dataset produces photo-realistic renderings using ray tracing and Monte Carlo global-illumination approximation, with panoramic and perspective camera settings.Rooms are rendered with varied furniture configurations and lighting, alongside depth maps and semantic masks.
- 4 The Structured3D Dataset: Structured3D supports methodology and application research through multiple structure and modality annotations, including non-cuboid room layouts.The paper highlights uses in multi-task learning, domain adaptation, floorplan reconstruction, and image synthesis.
- 4 The Structured3D Dataset: The dataset can be extended with additional structures, rendering effects, and videos for applications such as visual SLAM.Possible rendering extensions include lighting changes, fisheye cameras, motion blur, and imaging noise.
5 Experiments
The experiments evaluate Structured3D for room layout estimation and study synthetic-data augmentation, training schemes, and domain adaptation. They also illustrate why real benchmark annotations can be unreliable for non-cuboid rooms.
- 5 Experiments: The room layout experiment trains deep neural networks using Structured3D, while comparison datasets contain only cuboid layout annotations.Structured3D contributes 12k cuboid-layout panoramas for a controlled experiment, split into 10k training, 1k validation, and 1k testing images.
- 5.1 Experiment Setup: The evaluation uses 3D IoU, Corner Error, and Pixel Error to measure layout overlap, corner distance, and plane-class prediction error.These are the three standard metrics adopted from prior room-layout evaluations.
- 5.1 Experiment Setup: LayoutNet predicts corner and boundary maps before layout optimization, whereas HorizonNet directly predicts three one-dimensional layout vectors.The study follows the default training settings of both methods.
- 5.2 Experiment Results: Training schemes compare synthetic-only, real-only, combined synthetic-plus-real, and synthetic-pretraining followed by real-data fine-tuning.The combined setting uses Balanced Gradient Contribution, while the pretraining setting transfers from synthetic to real data.
- 5.2 Experiment Results: Adding synthetic data boosts the performance of both LayoutNet and HorizonNet, and using more synthetic pretraining data is studied as a factor affecting real-dataset accuracy.The paper evaluates 1k, 5k, and 10k synthetic images for pretraining before real-data fine-tuning.
- 5.2 Experiment Results: Real benchmark examples include non-cuboid rooms labeled as cuboid and omitted front-wall annotations, exposing limitations in real-dataset ground truth.Structured3D avoids these cited errors by generating ground truth automatically from original design files.
6 Conclusion
The paper concludes that Structured3D provides extensive structured 3D annotations and photo-realistic renderings, and demonstrates benefits for real-data augmentation and room-layout domain adaptation. It presents the dataset as a step toward holistic 3D scene understanding.
- 6 Conclusion: Structured3D contains ground-truth 3D structure annotations for 21,835 rooms and more than 196k photo-realistic 2D renderings.The dataset is presented as a large synthetic resource for structured 3D modeling research.
- 6 Conclusion: The paper demonstrates the dataset’s benefit for augmenting real data and facilitating domain adaptation in room layout estimation.These demonstrations are identified as use cases among the dataset’s potential applications.
- 6 Conclusion: The authors position Structured3D as an important step toward machines capable of human-level holistic 3D scene understanding.They also plan to add more 3D structure annotations and explore additional ways to use the dataset.