Source-linked AI summary
Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image
Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, Jian Jun Zhang
TL;DR
Indoor semantic reconstruction must recover both scene understanding and object geometry from a single image, but prior methods address only parts of this problem or treat objects independently. The paper proposes end-to-end joint learning of room layout, camera pose, object boxes, and meshes using holistic scene context, and reports consistent improvements across layout estimation, 3D detection, and mesh reconstruction. The method is limited by its need for dense point clouds to supervise object-mesh learning.
Problem
Recovering complete indoor scene semantics and geometry from a single image is difficult, while existing methods typically address only one component or reconstruct objects independently.
Method
The method jointly learns room layout, camera pose, 3D object bounding boxes, and meshes, using contextual relations, density-aware topology modification, and joint losses.
Results
The approach consistently improves indoor layout estimation, 3D object detection, and mesh reconstruction, reaching state-of-the-art performance on SUN RGB-D and Pix3D.
Takeaways & Limitations
The results support integrative training because individual scene-parsing processes have implicit impacts on one another in total 3D reconstruction.
Takeaways & Limitations
Object-mesh learning requires dense point clouds, which are labor-consuming to obtain in real scenes.
Abstract
from arXiv · showhide
Semantic reconstruction of indoor scenes refers to both scene understanding and object reconstruction. Existing works either address one part of this problem or focus on independent objects. In this paper, we bridge the gap between understanding and reconstruction, and propose an end-to-end solution to jointly reconstruct room layout, object bounding boxes and meshes from a single image. Instead of separately resolving scene understanding and object reconstruction, our method builds upon a holistic scene context and proposes a coarse-to-fine hierarchy with three components: 1. room layout with camera pose; 2. 3D object bounding boxes; 3. object meshes. We argue that understanding the context of each component can assist the task of parsing the others, which enables joint understanding and reconstruction. The experiments on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently outperforms existing methods in indoor layout estimation, 3D object detection and mesh reconstruction.
1. Introduction
The paper addresses the difficulty of recovering complete indoor scene context from one image by jointly predicting layout, camera pose, object boxes, and meshes. Its integrative design combines scene context, density-aware mesh generation, and multilateral object relations to improve reconstruction and understanding.
- Single-image indoor reconstruction remains challenging because depth ambiguity, clutter, and environmental complexity hinder recovery of semantic and geometric scene context.
- The method jointly predicts room layout, camera pose, 3D object bounding boxes, and object meshes from a single image.The framework uses contextual knowledge to connect scene understanding with instance-level reconstruction.
- Joint learning uses complementary information: mesh spatial occupancy assists 3D detection, while detection provides object alignment for reconstruction.
- Experiments report state-of-the-art performance across layout estimation, 3D detection, and mesh reconstruction on SUN RGB-D and Pix3D.
- Density-aware topology modification prunes mesh edges using local density, improving robustness across diverse indoor shapes and complex backgrounds.This avoids relying on a strict distance threshold to remove detached faces.
- Multilateral relations and attention model each object's surroundings to improve decisions about indoor object locations and poses.The approach considers relations among all in-room objects rather than only isolated or pair-wise interactions.
2. Related Work
Prior indoor reconstruction methods separately estimate layout, boxes, or shapes, while scene-level approaches face limitations in representation resolution, retrieval dependence, or scene context. Instance-level reconstruction is especially difficult because indoor scenes contain many diverse objects with varied geometry and topology.
- Early methods represented rooms with bounding boxes, and later approaches predicted furniture poses but remained limited to 3D object bounding boxes.
- Shape-retrieval methods search datasets for appearance-similar models, making accuracy and time efficiency dependent on dataset size and diversity.
- Single-object reconstruction methods use point clouds, patches, or primitives, but require postprocessing to obtain meshes.
- Instance-level scene reconstruction remains difficult because many indoor objects create a high-dimensional space of shapes with diverse geometry and topology.
- Scene-level methods reconstruct multiple object shapes with voxels, whose resolution is limited, while Mesh R-CNN treats objects as isolated geometries without scene context.
3. Method
The method jointly reconstructs indoor room layout, object bounding boxes, and meshes through a coarse-to-fine hierarchy of interconnected networks. Layout and camera estimation guide object detection, while object alignment and scene-level constraints support mesh generation and joint inference.
- Architecture: Three modules form a ‘box-in-the-box’ hierarchy: LEN estimates layout, ODN detects 3D object boxes, and MGN generates meshes.The full-scene mesh embeds their outputs through joint inference.
- Layout and detection: LEN predicts camera pose and the layout bounding box, while ODN estimates object box properties from 2D detections.The object properties include projection offset, distance, spatial size, and orientation.
- Layout and detection: ODN uses attention-weighted multilateral relations among in-room objects to regress each target object’s 3D bounding box.Appearance and geometry features determine relational contributions from neighboring or similar objects.
- Mesh generation: MGN generates object-centric meshes and adaptively modifies topology using local ground-truth density rather than a fixed distance threshold.The method cuts mesh edges according to a classifier, improving robustness across mesh scales and diverse shapes.
- Joint learning: Joint training combines individual LEN, ODN, and MGN losses with cooperative and global terms linking camera pose, object detection, and scene meshes.The global term aligns reconstructed meshes with ground-truth scene surfaces, while the loss weights balance the components.
4. Results and Evaluation
The evaluation spans SUN RGB-D and Pix3D, testing layout, camera pose, 3D detection, object pose, and mesh reconstruction. Results and ablations show benefits from joint training, relational features, and density-aware topology modification.
- Experimental Setup: Experiments use SUN RGB-D for layout, camera pose, and 3D detection, and Pix3D for object mesh reconstruction.SUN RGB-D contains labeled layouts, object boxes, and coarse point clouds; Pix3D provides aligned furniture images and models.
- Object Reconstruction: Qualitative comparisons on Pix3D evaluate Mesh R-CNN, AtlasNet-Sphere, TMN at two thresholds, and the proposed method.The comparison uses object images without masks, with category encoding for AtlasNet and TMN to support a fair comparison.
- Quantitative Analysis: Joint training improves layout estimation and camera pose estimation, while layout gains 2 points over state-of-the-art methods.Layout is evaluated by average IoU, and camera pose by mean absolute pitch and roll errors.
- Quantitative Analysis: Object-pose evaluation measures translation, rotation, and scale errors on NYU v2, with results indicating faithful object poses alongside reasonable spatial occupancy.The evaluation follows six object categories and ground-truth 2D boxes from the comparison method.
- Mesh Reconstruction: The density-aware topology modifier improves mesh reconstruction by retaining small-scale topology and cutting edges rather than removing faces.Local density is more robust across mesh scales than a fixed distance threshold, while edge cutting reduces incorrect connections and creates compact boundaries.
- Ablation Analysis: Ablations show that joint training consistently improves layout, detection, and scene mesh reconstruction, while relational features specifically improve 3D detection.Both cooperative loss Lco and global loss Lg benefit final outputs, and combining them further advances accuracy.
5. Conclusion
The method jointly reconstructs indoor room layout, camera pose, object bounding boxes, and meshes from a single image. Joint training improves the subtasks, while dense point clouds remain a practical limitation for mesh learning.
- The approach end-to-end reconstructs room layout, camera pose, object bounding boxes, and meshes from a single image.
- Joint learning significantly improves performance across the individual reconstruction subtasks and advances the state of the art.
- The authors conclude that each scene-parsing process has an implicit impact on the others, supporting integrative training for total 3D reconstruction.
- Dense point clouds are required to learn object meshes, but obtaining them in real scenes is labor-consuming.The authors suggest self- or weakly supervised scene reconstruction as future work.
Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711, 2019. The supplementary material contains:
The supplementary material covers camera/world coordinate conventions and additional experimental and qualitative results. It also lists implementation and dataset-related supplementary topics.
- The supplementary material includes network architecture, parameter settings, training strategies, and additional SUN RGB-D and Pix3D results.
- A. Camera and World System Setting: The camera and world systems share a center, with the world y-axis perpendicular to the floor.
- A. Camera and World System Setting: Rotating the world system around its y-axis aligns its x-axis with the camera's forward direction and removes the camera yaw angle.
- A. Camera and World System Setting: After yaw removal, camera pose relative to the world system is expressed using pitch β and roll γ.
B. Network Architecture
The method uses separate object detection, layout estimation, and mesh generation networks, followed by joint fine-tuning with mixed SUN RGB-D and Pix3D supervision.
- Architecture: The architecture comprises an Object Detection Network, Layout Estimation Network, and Mesh Generation Network.
- Training strategy: The three networks are initially trained individually using SUN RGB-D for detection and layout, and Pix3D for mesh generation.
- Training strategy: Joint fine-tuning optimizes the networks together with joint losses on SUN RGB-D while blending Pix3D samples into batches.
- Training strategy: Pix3D samples regularize mesh generation because SUN RGB-D provides only partial object point-cloud scans.
- Training strategy: Hierarchical batches feed scene images to layout estimation and object images to detection and mesh prediction.
C. 3D Detection on SUN RGB-D
The paper reports full 3D object detection results on SUN RGB-D in Table 10.
- Table 10 contains the full 3D object detection results on SUN RGB-D.
- The reported evaluation concerns 3D object detection rather than layout estimation or mesh reconstruction.
- SUN RGB-D is the dataset used for the results reported in this section.
D. Object Class Mapping
The paper maps NYU-37 object labels to nine Pix3D categories for scene reconstruction, using topology similarity because mesh deformation requires Pix3D class codes.
- Pix3D defines nine mesh-reconstruction categories, including bed, bookcase, chair, desk, sofa, table, tool, wardrobe, and miscellaneous.
- The mapping is needed because the mesh-generation network is pretrained on Pix3D and mesh deformation requires an object class code as input.
- The object-detection network outputs each detected object's projection offset, distance, orientation, and size.
- The layout-estimation network predicts camera pitch and roll together with the layout center, size, and orientation in world coordinates.
- NYU-37 labels are manually mapped to Pix3D labels based on topology similarity for scene reconstruction.
E. More Comparisons of Object Mesh Reconstruction on Pix3D
The paper provides additional qualitative comparisons of object mesh reconstruction against TMN, using the same threshold setting as the TMN paper.
- Figure 8 compares the proposed method with Topology Modification Network on object mesh reconstruction.
- TMN's threshold τ is set to 0.1 to match the setting reported in its paper.
F. More Samples of Scene Reconstruction on SUN RGB-D
The paper adds qualitative scene-reconstruction samples from SUN RGB-D and reports the evaluation setup for object detection and mesh-generation architecture.
- Figure 9 shows reconstruction results for test samples from SUN RGB-D.
- The mesh-generation architecture uses dc for the number of object categories and Ne for the number of points sampled on edges.
- The edge classifier follows the AtlasNet decoder architecture except that its final layer is replaced by a fully connected classification layer.
- Table 10 compares detected-object average precision on SUN RGB-D, where higher AP is better.
- CooP is evaluated using a model trained on NYU-37 object labels for a fair comparison.
- Additional qualitative mesh-reconstruction comparisons with TMN are presented alongside input images, TMN outputs, and the proposed outputs.