Source-linked AI summary
NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, Wenping Wang
TL;DR
NeuRIS addresses poor indoor 3D reconstruction in large texture-less areas by adaptively integrating estimated normal priors into neural volume rendering. It evaluates prior faithfulness through multi-view consistency, retaining appearance information for unreliable regions, and significantly outperforms state-of-the-art methods on indoor-scene reconstruction.
Problem
Large texture-less indoor areas challenge conventional matching-based reconstruction, while existing depth- and TSDF-based methods struggle with coherent, detailed, and memory-efficient reconstruction.
Method
NeuRIS adaptively integrates learned normal priors into neural volume rendering, evaluating their faithfulness through multi-view consistency and removing the normal constraint when it is unreliable.
Results
NeuRIS significantly outperforms state-of-the-art methods in indoor-scene reconstruction quality, with adaptive priors supporting both large smooth shapes and detailed irregular regions.
Takeaways & Limitations
The method combines globally consistent geometric guidance for texture-less regions with appearance-based reconstruction for small objects, sharp features, and thin structures.
Takeaways & Limitations
Depth-based alternatives can suffer from frame-to-frame incoherence, scale ambiguities, noisy surfaces, and floating outliers, while TSDF-based alternatives incur high memory consumption at high resolution.
Abstract
from arXiv · showhide
Reconstructing 3D indoor scenes from 2D images is an important task in many computer vision and graphics applications. A main challenge in this task is that large texture-less areas in typical indoor scenes make existing methods struggle to produce satisfactory reconstruction results. We propose a new method, named NeuRIS, for high quality reconstruction of indoor scenes. The key idea of NeuRIS is to integrate estimated normal of indoor scenes as a prior in a neural rendering framework for reconstructing large texture-less shapes and, importantly, to do this in an adaptive manner to also enable the reconstruction of irregular shapes with fine details. Specifically, we evaluate the faithfulness of the normal priors on-the-fly by checking the multi-view consistency of reconstruction during the optimization process. Only the normal priors accepted as faithful will be utilized for 3D reconstruction, which typically happens in the regions of smooth shapes possibly with weak texture. However, for those regions with small objects or thin structures, for which the normal priors are usually unreliable, we will only rely on visual features of the input images, since such regions typically contain relatively rich visual features (e.g., shade changes and boundary contours). Extensive experiments show that NeuRIS significantly outperforms the state-of-the-art methods in terms of reconstruction quality.
1 Introduction
NeuRIS addresses indoor reconstruction challenges caused by texture-less regions by adaptively integrating normal priors into neural rendering. It preserves detail in irregular structures and outperforms state-of-the-art methods in reconstruction quality.
- Motivation: Indoor scenes contain large texture-less areas and repetitive patterns that challenge correspondence-based dense reconstruction methods.Examples include white walls, floors, and reflecting surfaces.
- Approach: NeuRIS uses learned normal priors as globally consistent geometric constraints to mitigate geometry ambiguity in texture-less regions.The priors are integrated alongside appearance supervision from the input images.
- Adaptive prior use: NeuRIS evaluates normal-prior faithfulness on-the-fly using multi-view consistency and removes the normal constraint when consistency is not satisfied.Those regions instead rely on appearance information, which is often sufficient for sharp or irregular shapes with richer visual features.
- Benefits: Normal priors improve reconstruction of texture-less regions containing large smooth objects, while adaptive use preserves reconstruction of complex shapes with geometric details.The method is designed to use priors where they are reliable and appearance information where they are not.
- Results: NeuRIS significantly outperforms state-of-the-art methods on ScanNet in indoor-scene reconstruction quality.The paper reports extensive validations and comparisons.
2 Related works
Prior indoor reconstruction methods include traditional and learning-based MVS, while neural representations offer compact scene encoding. NeuRIS builds on normal priors because they are consistent in planar regions and across views.
- Multi-view stereo: Traditional multi-view stereo reconstructs textured surfaces plausibly but struggles with texture-less indoor regions.Learning-based MVS methods address this setting through depth-based and TSDF-based approaches.
- Depth-based methods: Depth-based methods often suffer from incompleteness, noisy surfaces, and scale ambiguities because depth maps are estimated individually.These inconsistencies limit coherent reconstruction across views.
- TSDF-based methods: TSDF-based methods regress volumetric scene representations but are constrained by computational resources and may lack reconstruction details.Atlas, for example, processes only a limited number of images and produces results lacking details.
- Neural representations: Coordinate-based neural representations encode scenes compactly and flexibly by mapping 3D coordinates to field values with MLPs.They have been applied to images, shapes, and 3D scenes.
- Normal priors: Estimated normal priors show high consistency in planar regions and across input views, motivating their integration into neural volume rendering.They also provide clues about underlying geometry while avoiding problems associated with depth priors.
3 Method
NeuRIS reconstructs indoor scenes with a global neural surface representation, using normal priors to constrain texture-less regions while adaptively removing unreliable supervision around irregular geometry.
- Two-phase optimization: The first phase jointly fits multi-view colors and estimated normal maps, producing coarse geometry with good large flat shapes but limited local detail.Normal priors help texture-less planar regions, but unreliable priors can produce inaccurate gross shapes for thin structures and small irregular objects.
- Adaptive prior checking: The second phase evaluates normal-prior faithfulness using multi-view photometric consistency induced by the current estimated normals and depths.A local plane associated with each reference pixel is projected to neighboring views through a homography, and patch similarity is measured with NCC.
- Scene representation: NeuRIS represents scene geometry with an SDF-based neural surface and uses a color network for view-dependent appearance.The surface is defined as the SDF’s zero level-set, while volume rendering provides differentiable supervision from images.
- Adaptive prior checking: Only priors whose corresponding geometry passes the consistency test supervise later optimization; unreliable priors are removed and color information is retained.The adaptive weight is one when the summed NCC meets threshold ϵ and zero otherwise.
- Training: Training samples pixels and ray points, then combines color, prior, and Eikonal losses to optimize the rendered scene representation.The color loss compares rendered colors with reference pixels, while the normal-prior loss is weighted by the adaptive consistency decision.
4 Experiments
Experiments show that NeuRIS improves indoor reconstruction, normal estimation, and novel-view synthesis, while its adaptive geometry check preserves difficult details such as thin structures and chair legs.
- 3D reconstruction: NeuRIS produces more complete and smooth geometry with fine details and fills holes caused by occlusions and incomplete scans.Qualitative comparisons show improved accuracy and completeness relative to other methods.
- Normal predictions: NeuRIS improves normal estimation by using multi-view information to correct inaccurate monocular predictions caused by local observations, occlusions, or ambiguities.The rendered normals achieve better cosine similarity to ground-truth normals than the predictions of method over 8 scenes and 493 images.
- Novel view synthesis: NeuRIS achieves an average PSNR of 24.4, compared with 23.3 for NeRF and 22.7 for NeuS, on 500 novel views across 8 scenes.The improved rendering quality is attributed to the method’s high-quality geometry.
- Ablation studies: Ablations show that normal priors improve planar reconstruction, while geometry checking removes unreliable normals and enables successful reconstruction of chair legs.The adaptive check addresses the failure of naively applying all normal priors to small or thin structures.
- Ablation studies: NeuRIS reconstructs both background desk surfaces and foreground thin structures without requiring foreground extraction as preprocessing.This allows hybrid scenes containing thin structures and general objects to be handled together.
5 Conclusion and future work
NeuRIS adaptively integrates normal priors into neural volume rendering to improve reconstruction in texture-less areas while preserving fine-detail reconstruction. The method has practical indoor-geometry applications, but its several-hour per-scene optimization limits very large-scale reconstruction.
- NeuRIS adaptively integrates normal priors into neural volume rendering with geometric constraints.This lets the network use prior knowledge in texture-less areas while retaining fine-detail reconstruction capacity for small, relatively textured objects.
- The framework supports reconstruction of texture-less areas and fine details of small objects with relatively more texture.
- NeuRIS has practical uses in VR/AR and applications requiring precise indoor geometry.
- Several hours of per-scene optimization hinder reconstruction at very large scale.Future work targets faster training with hybrid neural representations such as multi-resolution hash encoding.
Acknowlegements
The acknowledgements recognize experimental assistance, funding support, and computational resources.
- The authors thank Yuan Liu and Nenglun Chen for help with experiments.
- Christian Theobalt and Lingjie Liu are acknowledged for funding support through ERC and Lise Meitner fellowships.
- Computational resources were mainly provided by the HKU GPU Farm.