Source-linked AI summary
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, Hujun Bao
TL;DR
Real-time monocular 3D reconstruction must overcome the quality and computation limits of separately estimated depth maps. NeuralRecon jointly predicts local sparse TSDF surfaces and fuses fragments recurrently, achieving accurate, coherent reconstruction at real-time speed. Experiments on ScanNet and 7-Scenes show advantages over prior methods in reconstruction quality and running speed.
Problem
Real-time image-based dense reconstruction requires accurate, coherent geometry but remains limited by low quality and high computation demands.
Method
NeuralRecon incrementally predicts and fuses local sparse TSDF volumes from monocular key-frame fragments using sparse 3D convolutions and GRU-based fusion.
Results
33 key frames per second and approximately 10× faster than Atlas, while outperforming state-of-the-art methods in reconstruction quality and speed.
Takeaways & Limitations
NeuralRecon provides dense, coherent 3D scene geometry in real time and can support downstream tasks including 3D detection, semantic segmentation, and neural rendering.
Abstract
from arXiv · showhide
We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time.
1. Introduction
NeuralRecon addresses the quality, coherence, and computational challenges of real-time monocular 3D reconstruction by jointly predicting and fusing local TSDF geometry. Unlike depth-first pipelines, it reconstructs surfaces directly from key-frame windows, achieving coherent real-time results.
- Real-time dense reconstruction remains difficult because image-based methods combine low reconstruction quality with high computational demands.
- Depth-based pipelines estimate each key-frame depth independently, fuse depths into TSDF volumes, and repeatedly reconstruct overlapping geometry.This creates redundant computation and prevents each estimate from being conditioned on prior reconstruction.
- NeuralRecon jointly reconstructs and fuses local 3D geometry directly as sparse TSDF volumes from monocular images and SLAM-estimated camera poses.It unprojects image features into 3D, processes them with sparse convolutions, and refines predictions coarse-to-fine.
- Direct surface prediction enables locally coherent geometry while learning local smoothness and global shape priors.
- 33 key frames per second makes NeuralRecon real-time and approximately 10× faster than Atlas.The system was evaluated on ScanNet and 7-Scenes and reportedly outperformed multiple state-of-the-art methods and Atlas by a large margin.
2. Related Work
Related work largely uses single-view depth maps as intermediate representations, while volumetric methods predict scene geometry directly. NeuralRecon differs through sparse 3D convolution, coarse-to-fine TSDF prediction, and recurrent fusion for sequential reconstruction.
- Multi-view Depth Estimation: Real-time multi-view depth methods commonly process 2D depth cost volumes with CNNs, while some impose planar or surface-normal structure.
- Multi-view Depth Estimation: Offline Multiple View Stereo methods target high-resolution accuracy but are constrained by GPU memory when using 3D cost volumes and 3D CNNs.
- Volumetric Reconstruction: Most prior approaches use single-view depth maps, whereas Atlas predicts global TSDF and semantic labels once after aggregating the entire sequence.
- NeuralRecon Architecture: Its architecture uses three-level coarse-to-fine prediction, increasing sparse-voxel density while maintaining global hidden feature volumes for recurrent fusion.
- Volumetric Reconstruction: RoutedFusion is related through its replacement of simple linear addition with data-dependent fusion, whereas NeuralRecon uses learning-based sequential TSDF fusion.
- Neural Implicit Representations: NeuralRecon predicts a discrete sparse TSDF volume with sparse 3D convolutions rather than querying an MLP at image-feature and 3D-coordinate pairs.
3. Methods
NeuralRecon incrementally reconstructs local sparse TSDF geometry from monocular fragments, fusing features across fragments with a 3D GRU and integrating results into a global volume.
- Key Frame Selection: The system takes monocular images and SLAM camera poses, selecting sequential local fragments for incremental reconstruction.Each fragment contains N key frames selected using translation and rotation thresholds, with reconstruction restricted to its fragment bounding volume.
- Integration to the Global TSDF Volume: The fused local TSDF is transformed into global coordinates and used to replace corresponding voxels in the global TSDF volume.Marching Cubes is applied to the global volume at each time step to reconstruct the mesh.
- Image Feature Volume Construction: Image features are back-projected into a 3D feature volume and averaged across views using voxel visibility weights.The resulting multi-level feature volume is processed for volumetric reconstruction.
- Coarse-to-fine TSDF Reconstruction: Sparse convolutions and coarse-to-fine processing predict sparse TSDF volumes containing occupancy scores and signed-distance values.Voxels below the occupancy threshold are treated as void space and sparsified.
- GRU Fusion: A 3D GRU conditions current-fragment reconstruction on previous hidden states to selectively fuse historical and current geometric features.The update and reset gates control information flow, replacing conventional linear running-average TSDF fusion with a learned mechanism.
- Supervision and Implementation: Training uses occupancy binary cross-entropy and log-transformed SDF L1 losses at all coarse-to-fine levels.The network is trained end-to-end, with an ImageNet-pretrained image backbone and sparse 3D convolution implementation.
4. Experiments
NeuralRecon is evaluated on ScanNet and 7-Scenes using 3D geometry, 2D depth, efficiency, and ablation experiments. It generally achieves strong accuracy and real-time speed, while component studies examine feature fusion, GRU fusion, fragment bounding-volume fusion, and fragment length.
- Evaluation setup: Experiments use ScanNet and 7-Scenes, with 3D geometry and 2D depth metrics evaluated under protocols matched to baseline methods.F-score is emphasized for 3D quality because it captures both accuracy and completeness.
- Evaluation results: On ScanNet, NeuralRecon outperforms recent learning-based methods and slightly surpasses COLMAP on 3D geometry metrics.It also outperforms Atlas on accuracy, precision, and F-score, while having inferior completeness and recall compared with depth-based methods and Atlas.
- Evaluation results: On 7-Scenes, NeuralRecon achieves performance comparable to CNMNet and outperforms the other evaluated methods.The model is trained only on ScanNet, and the authors report that these results demonstrate generalization beyond the training domain.
- Efficiency: 33 key frames per second and 30 ms per key frame establish real-time performance, with NeuralRecon running approximately 10× faster than Atlas.The authors attribute the speed to incremental local-fragment reconstruction, sparse convolution, and avoiding redundant depth-map computation.
- Ablation study: Feature fusion with averaging improves precision by nearly 5% over conventional linear TSDF fusion and reconstructs smoother geometry.The comparison uses the same average operation for feature fusion and TSDF fusion.
- Ablation study: Replacing averaging with GRU improves recall by 4%, while fragment-bounding-volume fusion produces more complete surfaces with fewer ground artifacts.Using more views also improves accuracy; nine-view fragments yield over 2% higher F-score.
5. Conclusion
NeuralRecon incrementally reconstructs and fuses sparse TSDF volumes from monocular video using 3D sparse convolutions and GRU, producing accurate and coherent real-time geometry.
- NeuralRecon jointly reconstructs and fuses sparse TSDF volumes for each video fragment incrementally.The system combines 3D sparse convolutions with a gated recurrent unit.
- This design enables accurate and coherent 3D reconstruction in real time.
- Experiments show that NeuralRecon outperforms state-of-the-art methods in reconstruction quality and running speed.
- The reconstructed sparse TSDF volume can support downstream tasks including 3D object detection, semantic segmentation, and neural rendering.