Source-linked AI summary
Neural 3D Reconstruction in the Wild
Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, Noah Snavely
TL;DR
Internet photo collections pose challenges for reconstruction because they can be large, geometrically complex, and captured under varying illumination. The paper combines appearance modeling with hybrid voxel- and surface-guided sampling and introduces a benchmark; it reports improved efficiency and accuracy over classical and neural methods.
Problem
Internet photo collections can contain hundreds or thousands of images and varying illumination, while many reconstruction methods assume controlled capture and constant lighting.
Source: §1 · p. 2Method
The method reconstructs meshes with appearance embeddings and hybrid voxel- and surface-guided sampling, and introduces the Heritage-Recon benchmark and evaluation protocol.
Source: §3.1 · pp. 4–5, §4 · p. 5Results
The approach surpasses classical and neural reconstruction methods in efficiency and accuracy, and achieves the best or second-best quantitative performance in nearly all scenes and thresholds.
Source: §1 · p. 2, §5.3 · p. 7, §6 · p. 7Takeaways & Limitations
The benchmark and protocol provide an evaluation setting for Internet-photo reconstruction with incomplete ground-truth geometry.
Source: §1 · p. 2, §4 · p. 5Without domain-specific priors, the model can fail to produce accurate geometry in regions unseen in the image observations.
Source: §6 · p. 7
Abstract
from arXiv · showhide
We are witnessing an explosion of neural implicit representations in computer vision and graphics. Their applicability has recently expanded beyond tasks such as shape generation and image-based rendering to the fundamental problem of image-based 3D reconstruction. However, existing methods typically assume constrained 3D environments with constant illumination captured by a small set of roughly uniformly distributed cameras. We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections in the presence of varying illumination. To achieve this, we propose a hybrid voxel- and surface-guided sampling technique that allows for more efficient ray sampling around surfaces and leads to significant improvements in reconstruction quality. Further, we present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes. We perform extensive experiments, demonstrating that our approach surpasses both classical and neural reconstruction methods on a wide variety of metrics.
1 INTRODUCTION
Internet photo collections challenge reconstruction methods with large image sets, uneven viewpoints, complex geometry, and varying appearance. The paper combines appearance modeling and hybrid sampling with a new benchmark, reporting efficient and accurate reconstruction on in-the-wild scenes.
1 INTRODUCTION
- Internet collections may contain hundreds or thousands of images, while complex geometry and varying illumination challenge methods built for controlled capture.The introduction notes that standard datasets typically contain tens of images and that existing methods often assume constant illumination.Source: §1 · p. 2
- The approach models appearance variation with embeddings and reconstructs meshes through neural surface and volumetric rendering.Meshes directly represent scene geometry and can be imported into standard graphics pipelines.Source: §1 · p. 2
- Hybrid voxel- and surface-guided sampling uses SfM point clouds and surface estimates to focus samples near surfaces and improve geometric fitting.The paper identifies a straightforward integration of appearance modeling and NeuS-style rendering as computationally intractable for large collections on limited high-end GPUs.Source: §1 · p. 2
- The method surpasses classical and neural reconstruction methods in efficiency and accuracy on the Heritage-Recon benchmark.Heritage-Recon pairs Internet-derived images and SfM models with LiDAR geometry and an evaluation protocol for incomplete ground truth.Source: §1 · p. 2
2 RELATED WORK
Prior work spans classical multi-view pipelines, neural rendering, and implicit surface methods, but these approaches differ in their outputs, supervision needs, and suitability for unstructured Internet photos. This work focuses on direct geometry reconstruction from those photos.
2 RELATED WORK
- Classical multi-view stereo often estimates per-image depth maps and fuses them into point clouds before mesh extraction, making the stages sensitive to inconsistent depth estimates [Furukawa and Hernández 2015; Furukawa and Ponce 2009; Goesele et al. 2006; Hedman et al. 2017; Schönberger et al. 2016].The passage notes that these pipelines can yield noisy or incomplete reconstructions.Source: §2 · p. 2
- The sampling comparison narrows the sampled region from the whole unit sphere to sparse voxels around SfM-estimated surfaces, then to a smaller range around estimated surface positions.The caption says sphere-based sampling leaves most samples in empty regions, while the surface-guided strategy increases sample density near surfaces.Source: Figure 2 · p. 3
- NeRF-W models Internet scenes for novel-view synthesis, whereas this approach uses a surface-based representation to produce smooth, accurate meshes [Martin-Brualla et al. 2021].The passage states that NeRF-W cannot produce high-quality surface reconstructions.Source: §2 · p. 3
- Surface rendering can model geometry accurately but often needs extra constraints, while volume rendering handles complex scenes but makes accurate surface extraction difficult [Niemeyer et al. 2020; Yariv et al. 2020; Kondo et al. 2021; Martin-Brualla et al. 2021; Mildenhall et al. 2020].Methods unifying surface and volumetric representations enable accurate surfaces without masks; this work extends NeuS for unconstrained Internet photos [Oechsle et al. 2021; Wang et al. 2021; Yariv et al. 2021].Source: §2 · p. 3
- Unlike landmark- or city-scale neural rendering focused mainly on novel-view synthesis, this work models geometry from unstructured Internet photos.The paper demonstrates efficient, accurate, and highly detailed landmark surface reconstructions.Source: §2 · p. 3
3 METHOD
The method represents scene geometry and image-specific appearance with neural implicit functions, optimizing them through volume rendering. It combines voxel- and surface-guided sampling for efficient training, with additional measures for dynamic objects and textureless sky.
3 METHOD
- The scene uses an SDF function for geometry and image-specific appearance functions and embeddings, optimized for color consistency through volume rendering.The SDF approximates signed distance to the surface, represented as its zero level set.Source: §3 · p. 4
3.1 Efficient Sampling during Training
- For in-the-wild scenes, NeuS-style sampling can be inefficient: 1,024 coarse and 1,024 fine samples per ray could require an estimated 10 days of training on 32 GPUs.The estimate is for using the same sample counts as NeRF-W to train NeuS; the authors motivate a hybrid voxel- and surface-guided strategy instead.Source: §3.1 · p. 4
- Voxel-guided sampling narrows each ray’s search to a dilated sparse volume initialized from SfM points, and excluding non-intersecting rays can reduce required training rays by over 30%.The volume supplies a rough surface envelope and foreground-background separation.Source: §3.1 · p. 4
- After voxel-guided bootstrapping, surface-guided sampling uses cached SDF predictions to place samples in a narrower range around estimated surface positions.Figure 3 compares 1,024 NeuS fine-level samples with 24 surface-guided samples; the latter are denser and closer to the surface at the final iteration. Ablations report that omitting surface-guided sampling prevents convergence to the same accuracy even with sufficient time.Source: Figure 3 · p. 4, §3.1 · p. 5
- The hybrid retains sparse voxel-guided samples alongside dense surface-guided samples because surface-only sampling can create voxel-border artifacts from insufficient empty-space supervision.An additional importance-sampling iteration follows surface-guided sampling, for a total of n_v + 2 × n_s samples per ray.Source: §3.1 · p. 5
3.2 Additional Details
- Segmentation masks remove dynamic-object rays during training; imperfect semantic background masks receive a small-weight free-space penalty that removes sky while preserving foreground structures, as Figure 4 illustrates LiDAR alignment.Source: §3.2 · p. 5, Figure 4 · p. 5
4 THE HERITAGE-RECON BENCHMARK
Heritage-Recon pairs Internet-derived image collections with LiDAR ground-truth geometry to support evaluation of in-the-wild reconstruction. Its construction aligns scan and SfM coordinates and restricts evaluation to scan regions visible in the images.
4 THE HERITAGE-RECON BENCHMARK
- Heritage-Recon addresses the lack of a dataset pairing Internet photo collections with ground-truth 3D geometry by deriving a benchmark from Open Heritage 3D LiDAR scans.The benchmark combines scans with Internet-derived images and SfM models; the section introduces its data processing, metrics, and evaluation protocol.Source: §4 · p. 5
- The benchmark pairs four cultural-heritage landmarks’ LiDAR scans with Internet images and SfM camera poses and sparse point clouds.Images for Brandenburg Gate, Pantheon Exterior, and Lincoln Memorial come from MegaDepth; Palacio de Bellas Artes images were collected from Flickr.Source: §4 · p. 5
- After filtering SfM points and aligning them to LiDAR with ICP, the authors report less than one pixel of reprojection error across all scenes.Alignment quality is also visually inspected by overlaying depth maps rendered from aligned LiDAR scans with images; agreement of image and depth edges is one indicator.Source: §4 · p. 6
- Evaluation uses only LiDAR regions visible in the input images, estimated from SfM point clouds and expanded through LoFTR-generated semi-dense points.LiDAR points are filtered using voxels around the SfM points with a relatively large voxel size.Source: §4 · p. 6
5 EXPERIMENTS
Experiments compare reconstruction quality and efficiency against classical and learning-based baselines on Heritage-Recon. The method performs competitively across scenes and thresholds, while hybrid sampling improves training speed and reconstruction quality over the tested sampling alternatives.
5.1 Implementation Details
- Training is bootstrapped with voxel-guided sampling for 5000 iterations before surface-guided sampling is added.Experiments use eight NVIDIA A100 GPUs, and the final mesh is extracted only within the SfM-initialized volume.Source: §5.1 · p. 6
5.2 Baselines
- The baselines include COLMAP dense reconstruction, Vis-MVSNet, and NeRF-W, with method-specific mesh reconstruction pipelines for the learning-based approaches.COLMAP uses patch-match stereo and Poisson reconstruction; Vis-MVSNet depth maps are fused with COLMAP point fusion before Poisson reconstruction, and NeRF-W renderings are fused with KinectFusion.Source: §5.2 · p. 6
5.3 Evaluation
- Our method achieves the best or second-best quantitative performance in nearly all cases and has higher recall overall, while producing better visual quality than baselines.The evaluation compares converged and early-stopped models across scenes and thresholds.Source: Table 1 · p. 6, §5.3 · p. 7
- Evaluation measures accuracy and completeness with precision, recall, and F1 at Low, Medium, and High thresholds, plus an AUC integrating F1 across thresholds.The thresholds are selected per scene, with the maximum threshold set where F1 first reaches 80.Source: §5.3 · p. 6
- For larger scenes such as PBA, baselines can be up to 14× slower than the speed-optimized model, while the best model has up to a 3× slowdown.The reconstruction-time comparison also finds the proposed approach significantly faster in most cases, including after full convergence.Source: Figure 5 · p. 7, §5.3 · p. 7
- Hybrid sampling consistently achieves the best F1 AUC across almost all training steps and remains better than sphere-based and voxel-guided sampling after substantial training.On Lincoln Memorial, hybrid sampling also has the best training speed and visual quality.Source: §5.3 · p. 7, Figure 6 · p. 7
- The reported sampling approach requires 2∼3× longer training than hybrid sampling, while sphere-based sampling is incapable of achieving this accuracy in a reasonable time.Additional ablation studies are presented in the supplemental material.Source: §5.3 · p. 7
6 LIMITATIONS AND CONCLUSION
The method inherits NeRF-like limitations: inaccurate camera registration can affect reconstruction, and geometry in unseen regions may be inaccurate without domain-specific priors. The conclusion highlights hybrid sampling as a way to improve training time and points to inverse rendering and scene dynamics as future directions.
6 LIMITATIONS AND CONCLUSION
- Without domain-specific priors, learning surface locations only from known image observations can fail to produce accurate geometry in unseen regions.Inaccurate camera registration can also affect final reconstruction quality.Source: §6 · p. 7
- The paper concludes that hybrid voxel-surface-guided sampling significantly improves training time over baseline methods for learning accurate surface locations in complex scenes.The method targets high-quality surface reconstruction from Internet photo collections.Source: §6 · p. 7
- Future work envisions full inverse rendering and modeling scene dynamics across different time scales.Source: §6 · p. 7
- Heritage-Recon qualitative results show full meshes alongside orange-highlighted zoomed regions, where the models yield more complete and higher-quality meshes.Source: Figure 7 · p. 8