Source-linked AI summary
Neural 3D Scene Reconstruction with the Manhattan-world Assumption
Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, Xiaowei Zhou
TL;DR
Low-textured indoor planes remain difficult to reconstruct because photometric or stereo constraints admit unreliable solutions and per-view optimization lacks consistency. The paper integrates Manhattan-world planar constraints into implicit neural representations, jointly optimizing 3D geometry and semantics. On ScanNet and 7-Scenes, it reports large reconstruction-quality gains over prior methods, while remaining limited to Manhattan-world scenes.
Problem
Low-textured indoor planes remain difficult for multi-view reconstruction because photometric constraints permit many plausible solutions and per-view planar optimization can be inconsistent across views.
Method
ManhattanSDF represents scene geometry, appearance, and semantics with implicit neural fields, applying planar constraints and jointly optimizing 3D semantics with geometry.
Results
The method significantly outperforms state-of-the-art methods in reconstruction quality on ScanNet and 7-Scenes, especially in planar regions.
Takeaways & Limitations
Joint 3D optimization produces accurate and complete planes while maintaining details in non-planar regions and improves robustness to inaccurate 2D segmentation.
Takeaways & Limitations
The method only considers the Manhattan-world assumption, although some scenarios require a more general assumption such as the Atlanta-world assumption.
Abstract
from arXiv · showhide
This paper addresses the challenge of reconstructing 3D indoor scenes from multi-view images. Many previous works have shown impressive reconstruction results on textured objects, but they still have difficulty in handling low-textured planar regions, which are common in indoor scenes. An approach to solving this issue is to incorporate planer constraints into the depth map estimation in multi-view stereo-based methods, but the per-view plane estimation and depth optimization lack both efficiency and multi-view consistency. In this work, we show that the planar constraints can be conveniently integrated into the recent implicit neural representation-based reconstruction methods. Specifically, we use an MLP network to represent the signed distance function as the scene geometry. Based on the Manhattan-world assumption, planar constraints are employed to regularize the geometry in floor and wall regions predicted by a 2D semantic segmentation network. To resolve the inaccurate segmentation, we encode the semantics of 3D points with another MLP and design a novel loss that jointly optimizes the scene geometry and semantics in 3D space. Experiments on ScanNet and 7-Scenes datasets show that the proposed method outperforms previous methods by a large margin on 3D reconstruction quality. The code is available at https://zju3dv.github.io/manhattan_sdf.
1. Introduction
The paper targets unreliable reconstruction of low-textured indoor planes and introduces a Manhattan-world implicit representation that jointly optimizes geometry and semantics in 3D space.
- Traditional multi-view stereo methods struggle with low-textured floors and walls because stereo matching is unreliable in these regions.
- Planar-constraint methods typically optimize per-view depth maps, which can leave depth estimation and plane segmentation inconsistent across views.
- Implicit neural representations encode scene geometry and semantics in 3D space, enabling jointly optimized and globally consistent reconstruction and segmentation.The method predicts signed distance, color, and semantic logits for arbitrary 3D points.
- Experiments on ScanNet and 7-Scenes report significant reconstruction-quality gains over state-of-the-art methods, especially for planar regions.
- The method integrates Manhattan-world constraints into implicit neural representation optimization and uses a loss that jointly optimizes semantic labels with scene geometry.
2. Related work
Related work spans multi-view stereo, neural scene reconstruction, and semantic segmentation, with recent methods increasingly representing scenes and semantics using neural functions.
- MVS: Multi-view stereo commonly estimates per-image depth and fuses the depth maps into final 3D reconstructions.
- MVS: Traditional and learned multi-view stereo methods often perform poorly on texture-less regions because dense feature matching is difficult there.
- Neural scene reconstruction: Neural scene reconstruction methods predict 3D point properties with networks, while recent approaches use implicit neural functions instead of discrete voxels.
- Semantic segmentation: Semantic segmentation research includes both image-based methods and 3D methods operating on point clouds, voxels, or neural radiance fields.
- ManhattanSDF: The proposed overview combines implicit scene representations, differentiable volume rendering, 2D supervision, and Manhattan-world planar constraints.
3. Method
ManhattanSDF represents indoor scene geometry and appearance with implicit fields learned from images, then adds Manhattan-world planar constraints and jointly optimized 3D semantics to improve low-textured reconstruction.
- Learning scene representations from images: ManhattanSDF models scene geometry and appearance with signed-distance and color fields learned from images using volume rendering.A 3D point is mapped to signed distance, while appearance prediction uses position, view direction, surface normal, and geometry features.
- Learning scene representations from images: Image-only training struggles with reasonable geometry, while depth supervision improves reconstruction but remains limited where input depth maps are incomplete.The depth loss compares rendered and input depths for rays with estimated depth values.
- Scene reconstruction with planar constraints: Planar constraints use 2D floor and wall segmentation to enforce shared normal directions under the Manhattan-world assumption.Floor normals are constrained to the vertical z-axis, while wall normals are related to a jointly optimized learnable normal constrained to be vertical to the floor normal.
- Joint optimization of semantics and geometry: Incorrect 2D segmentation can produce inaccurate geometry, motivating joint optimization of 3D semantic labels with scene geometry and appearance.The method adds an MLP that predicts semantic logits for 3D points and renders them into image space.
- Joint optimization of semantics and geometry: The joint loss uses rendered semantic probabilities in geometric losses while cross-entropy supervision prevents both floor and wall probabilities from vanishing.When an input label is wrong, reducing its weighted geometric loss can lower the corresponding rendered probability and correct the semantic label; multiview consensus improves segmentation accuracy.
4. Implementation details
The implementation uses PyTorch, DeepLabV3+ from Detectron2, one NVIDIA TITAN Xp GPU, unit-sphere camera normalization, and 640 × 480 image resizing.
- Implementation details: The method is implemented in PyTorch with DeepLabV3+ from Detectron2 for 2D semantic segmentation and trained on one NVIDIA TITAN Xp GPU.Cameras are normalized inside a unit sphere and network parameters are initialized so the SDF approximates a unit sphere with inward-facing normals.
- Implementation details: Images are resized to 640 × 480 for both 2D semantic segmentation and reconstruction processing.
5. Experiments
Experiments on ScanNet and 7-Scenes evaluate reconstruction, semantic segmentation, and novel-view synthesis against MVS, volume-rendering, and ablation baselines. The proposed method improves reconstruction quality through depth supervision, planar regularization, 3D semantics, and joint optimization.
- Datasets and metrics: The experiments use ScanNet and 7-Scenes, with RGB-D fusion ground truth and accuracy, completeness, precision, recall, and F-score for reconstruction.Semantic segmentation is evaluated using floor and wall IoU.
- Baselines: COLMAP, plane-enhanced MVS methods, ACMP, NeRF, UNISURF, NeuS, and VolSDF serve as reconstruction baselines.Meshes from volume-rendering methods are extracted with Marching Cubes and re-fused using TSDF fusion.
- Ablation studies: 0.095 precision and 0.061 recall improvements result from adding estimated sparse-depth supervision to VolSDF, although reconstruction remains noisy and incomplete.The depth loss improves convergence, but incomplete input depth maps limit performance in texture-less regions.
- Ablation studies: 0.047 precision and 0.032 recall improvements show that learning semantics in 3D space assists reconstruction.The comparison is between VolSDF-D and VolSDF-D-S.
- Ablation studies: 0.174 precision and 0.151 recall improvements result from replacing Lgeo with Ljoint, while preserving planar quality and improving non-planar reconstruction.The joint optimization configuration achieves the most coherent ablation reconstruction results.
- State-of-the-art comparisons: The method significantly outperforms COLMAP and volume-rendering methods, while producing more coherent planar regions than methods applying planar priors to MVS.Across ScanNet and 7-Scenes, it ranks second in accuracy to COLMAP, matches planar-prior MVS completeness, and achieves the best combined performance.
- Semantic segmentation: Semantic segmentation improves over DeepLabV3+ in floor and wall regions, whose predictions show noisy and view-inconsistent boundaries.The learned 3D semantics are rendered back into image space for evaluation.
6. Conclusion
The method integrates Manhattan-world constraints into indoor scene reconstruction, using semantic information to improve planar geometry while preserving non-planar details. Its limitation is reliance on the Manhattan-world assumption.
- ManhattanSDF uses semantic information in planar regions to guide indoor scene geometry reconstruction.
- Joint optimization of 3D semantics and geometry improves robustness to inaccurate 2D segmentation.
- The method reconstructs accurate, complete planes while maintaining details in non-planar regions.
- The approach is limited to Manhattan-world scenes, although its constraints could be modified for broader assumptions such as the Atlanta-world assumption.
A. Network architecture
The network architecture maps spatial points and view directions to signed distance, color, and semantic outputs.
- The network takes a spatial point x and view direction v as inputs.
- It outputs signed distance d, color c, and semantic logits s for the queried point and view direction.
B. Evaluation metrics
The evaluation section defines reconstruction metrics, reports per-scene experiments, and examines optimization of the wall normal parameter.
- Reconstruction metric definitions use point clouds sampled from predicted and ground-truth meshes.
- Experiments use eight randomly selected scenes from ScanNet and 7-Scenes and compare against state-of-the-art MVS and implicit-neural-representation methods.
- Quantitative reconstruction results are reported for each individual scene in Table 5.
- The floor normal nf is fixed as ⟨0, 0, 1⟩, while the wall normal nw is optimized with network parameters because wall alignment is difficult to ensure.
- The convergence of nw is evaluated using a cost function based on four clustered wall-normal directions per scene.
E. Novel View synthesis results
Novel-view synthesis is evaluated qualitatively on ScanNet using camera poses shifted along the z-axis. The method also addresses non-planar detail preservation and segmentation errors, but is unsuitable for non-Manhattan scenes.
- Novel camera poses are created by adding 0.2 along the z-axis to training camera poses for ScanNet comparisons.
- The method is not suitable for non-Manhattan scenes containing sloped ground or walls that are not mutually vertical.
- Zoom-in visualizations examine geometric details in non-planar regions.
- Ground-truth meshes are rendered in novel views as references because ground-truth novel-view images are unavailable.
- Joint optimization can correct reconstruction errors caused when non-planar regions are incorrectly classified as planar.
- Because the scene uses one SDF network with Eikonal regularization, planar-region convergence can influence reconstruction quality in non-planar regions.
J. Comparison with supervised method.
The paper reports DeepVideoMVS results in Table 6, which compares methods on ScanNet.
- Table 6 reports results for DeepVideoMVS.
- DeepVideoMVS is evaluated as a supervised method comparison.
- The comparison is conducted on the ScanNet dataset.