Source-linked AI summary
SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, Wenping Wang
TL;DR
Sparse-view surface reconstruction is difficult because conventional and neural methods struggle with limited input views, while per-scene optimization prevents generalization to new scenes. SparseNeuS learns generalizable geometry priors and uses hierarchical sparse-view strategies to reconstruct high-quality surfaces from as few as 2 or 3 images, with reported advantages over prior methods.
Problem
Sparse-view reconstruction is challenging because noise, weak textures, and reflections hinder dense matching, while neural methods typically require dense views and time-consuming per-scene optimization.
Method
SparseNeuS uses neural rendering with geometry encoding volumes and hierarchical sparse-view strategies, including multi-level geometry reasoning and multi-scale color blending.
Results
SparseNeuS generalizes to new scenes and produces high-quality reconstructions from sparse images, including as few as 2 or 3 images.
Takeaways & Limitations
The method supports high-quality multi-view surface reconstruction in sparse-input and previously unseen scenes.
Takeaways & Limitations
Comparison results for IDR involve extra object masks during per-scene optimization, unlike methods that do not require them.
Abstract
from arXiv · showhide
We introduce SparseNeuS, a novel neural rendering based method for the task of surface reconstruction from multi-view images. This task becomes more difficult when only sparse images are provided as input, a scenario where existing neural reconstruction approaches usually produce incomplete or distorted results. Moreover, their inability of generalizing to unseen new scenes impedes their application in practice. Contrarily, SparseNeuS can generalize to new scenes and work well with sparse images (as few as 2 or 3). SparseNeuS adopts signed distance function (SDF) as the surface representation, and learns generalizable priors from image features by introducing geometry encoding volumes for generic surface prediction. Moreover, several strategies are introduced to effectively leverage sparse views for high-quality reconstruction, including 1) a multi-level geometry reasoning framework to recover the surfaces in a coarse-to-fine manner; 2) a multi-scale color blending scheme for more reliable color prediction; 3) a consistency-aware fine-tuning scheme to control the inconsistent regions caused by occlusion and noise. Extensive experiments demonstrate that our approach not only outperforms the state-of-the-art methods, but also exhibits good efficiency, generalizability, and flexibility.
1 Introduction
SparseNeuS targets surface reconstruction when sparse views and scene generalization challenge existing methods. It combines generalizable geometry priors with coarse-to-fine reasoning, multi-scale color blending, and consistency-aware fine-tuning.
- Sparse-view reconstruction is difficult because noise, weak textures, and reflections hinder complete dense matches.
- Existing neural implicit methods typically depend on dense input views and time-consuming per-scene optimization, limiting generalization to new scenes.
- SparseNeuS learns generalizable priors from image features through geometry encoding volumes and predicts SDF-based surfaces from coordinates with corresponding volume features.
- The method generalizes across scenes, reconstructs from as few as 2 or 3 images, and outperforms state-of-the-art methods in reconstruction quality and computational efficiency.
- Multi-level geometry reasoning progressively reconstructs surfaces from coarse to fine using cascaded volumes and conditioned per-scene fine-tuning.
- Multi-scale color blending combines pixel- and patch-based radiance consistency to improve color prediction from sparse inputs.
- Consistency-aware fine-tuning detects regions with inconsistent projections from occlusion or noise and excludes them during optimization.
2 Related Work
Related work spans classical multi-view stereo, neural implicit geometry, and generalizable neural rendering. These approaches differ in representation and generalization, while sparse-view neural rendering can still yield noisy or incomplete geometry.
- Classical multi-view stereo reconstructs geometry by matching corresponding features across input images using voxels, point clouds, or depth maps.
- Neural implicit reconstruction represents 3D geometry with networks that output occupancy fields or signed distance functions.
- Generalizable neural rendering methods use sparse views and image radiance to synthesize novel views for unseen scenes, but extracted geometry often remains noisy, incomplete, or distorted.
3 Method
SparseNeuS hierarchically reconstructs surfaces from sparse views using geometry encoding volumes, multi-scale color blending, and consistency-aware fine-tuning.
- Geometry reasoning: SparseNeuS hierarchically recovers surfaces from cascaded geometry encoding volumes in a coarse-to-fine manner.The coarse volume captures local geometry information at low resolution, while the fine level is guided by the coarse geometry.
- Geometry reasoning: A geometry encoding volume aggregates projected multi-view image features through a variance-based cost volume and sparse 3D CNN.The resulting volume provides features that an MLP combines with positional-encoded coordinates to predict the SDF surface representation.
- Appearance prediction: SparseNeuS predicts novel-view colors by blending projected input-image colors with learned weights derived from image features, viewing direction, and volume features.Pixel-based blending uses weighted projected colors, while SDF-based volume rendering aggregates sampled colors and SDF values along each ray.
- Appearance prediction: Multi-scale color blending combines pixel-based and patch-based information to enforce local and contextual radiance consistency under sparse inputs.Patch colors are obtained efficiently using a local-plane assumption and homography-based projections.
- Per-scene fine-tuning: Consistency-aware fine-tuning detects regions with inconsistent cross-view projections and excludes them from optimization.This targets distortions caused by occlusion and image noise while refining geometry inferred by the generic model.
4 Datasets and Implementation
The evaluation uses DTU for training and testing, with three-view testing inputs and cropped images for training; implementation uses a two-stage training schedule.
- Datasets: DTU provides 75 non-overlapping training scenes and 15 testing scenes, with evaluation using three views per scene.Testing uses 600 × 800 images and foreground masks; each scene contains two sets of three images.
- Implementation: Training uses center-cropped 512 × 640 images for memory efficiency.
- Implementation: The generic model is trained in two stages, with 150k iterations for coarse networks followed by 150k iterations for fine networks while coarse networks remain fixed.Training uses two RTX 2080Ti GPUs with a batch size of 512 rays.
5 Experiments
Experiments compare SparseNeuS with generic neural rendering, per-scene neural surface reconstruction, and COLMAP on DTU and BlendedMVS. The results show strong quantitative and qualitative performance, while ablations support cascaded volumes, multi-scale blending, and consistency-aware fine-tuning.
- Comparisons: SparseNeuS is compared with generic neural rendering methods, per-scene optimization methods, and COLMAP.The compared methods include PixelNerf, IBRNet, MVSNerf, IDR, NeuS, VolSDF, UniSurf, and COLMAP.
- Quantitative comparisons: On DTU, SparseNeuS outperforms state-of-the-art methods by a large margin in Chamfer Distance under both generic and per-scene optimization settings.Per-scene fine-tuning with 10k iterations takes 20 minutes and substantially improves the results.
- Qualitative comparisons: With three input images, SparseNeuS produces smoother and less noisy meshes than MVSNerf, while short per-scene fine-tuning adds finer-grained, cleaner, and more accurate details.NeuS produces distorted and incomplete surfaces with three input images, and COLMAP is less complete for weak-texture objects.
- Cross-dataset evaluation: On BlendedMVS, despite not being trained on that dataset, SparseNeuS produces cleaner and more complete results than MVSNerf and recovers subtle geometry such as a hose.COLMAP misses fine-grained geometry, while NeuS struggles with details in weak-texture scenes.
- Ablation studies: Ablations show that cascaded volumes produce more detailed geometry than a single volume, while pixel- and patch-based blending together outperform pixel-based blending alone.The consistency-aware scheme detects inconsistent regions and excludes them during fine-tuning, yielding cleaner geometry with fewer distortions.
- Per-scene optimization: Using learned priors reduces the Chamfer Distance from 1.98 without prior-based initialization to 1.65 with priors.The experiment evaluates per-scene optimization with and without the learned priors.
6 Conclusions
SparseNeuS targets high-quality surface reconstruction from sparse images while generalizing to new scenes. Its learned priors and sparse-view strategies improve reconstruction quality and computational efficiency, but SDF representation restricts outputs to closed surfaces.
- Figure 6 compares per-scene optimization with learned priors against optimization without priors.
- SparseNeuS recovers surfaces from multi-view images, generalizes to new scenes, and produces high-quality reconstructions from sparse images.
- Its geometry encoding volumes support generic geometry reasoning, while multi-level reasoning, multi-scale color blending, and consistency-aware fine-tuning address sparse-view challenges.
- SparseNeuS achieves better reconstruction quality and computational efficiency than state-of-the-art methods in experiments.
- Because SparseNeuS adopts a signed distance field, it can produce only closed-surface reconstructions.
Acknowlegements
The acknowledgements thank reviewers for valuable feedback and note fellowship support for Xiaoxiao Long.
- The authors thank reviewers for their valuable feedback.
- Xiaoxiao Long is supported by the Hong Kong PhD Fellowship Scheme.
- The acknowledgements record both reviewer feedback and fellowship support.
Supplementary Materials for SparseNeuS: Fast Generalizable Neural Surface
The supplementary materials accompany SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views and identify its authors, institutions, and arXiv version.
- The supplementary materials are titled “Reconstruction from Sparse Views.”
- The listed authors are Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang.
- The document identifies itself as arXiv:2206.05737v2, dated 2 Aug 2022.
- The authors are affiliated with The University of Hong Kong, Tencent Games, and Texas A&M University.
1 Details of Patch-based Color Blending
Patch-based color blending uses local surface geometry and homography to incorporate contextual patch information into sparse-view color prediction. This produces more reliable and consistent predictions, especially where texture is weak or intensity changes.
- Patch-based color blending jointly evaluates local and contextual radiance consistency for more reliable color predictions.
- The method estimates a local plane at sampled point q using the SDF gradient as its normal direction.
- Selected local-plane points are projected into each input view using homography, and their colors are obtained by interpolation.
- All points on the local plane share q’s blending weights, requiring only one blending-weight query for the patch.
- Patch colors are blended across views and aggregated along the query ray using SDF-based volume rendering.
- Patch-based rendering provides more global information and consistent shape context in weak-texture and changing-intensity regions.
2 More Implementation Details
SparseNeuS combines multi-scale image features, sparse 3D convolutions, and MLPs for SDF modeling and fine-tuning. Training uses specified loss weights, while DTU images receive threshold-based denoising to remove black backgrounds and noise.
- Network details: A Feature Pyramid Network extracts multi-scale image features, while sparse 3D CNNs use a U-Net-like architecture with torchsparse.The SDF fθ uses a four-hidden-layer MLP with hidden size 256.
- Network details: The SDF fθ is modeled by an MLP with 4 hidden layers of size 256, and the fine-tuning blending network fc uses 3 hidden layers.
- Training parameters: Total-loss weights are α = 0.1 and β = 0.02, with SDF scaling parameter τ = 100 for the sparseness loss.
- Training parameters: The consistency-aware color loss uses λ0 = 0.01 and λ1 = 0.015 by default, while their ratio may be tuned per scene.Decreasing λ0/λ1 keeps more regions; increasing it excludes more regions.
- Data preparation: DTU images are denoised by thresholding intensities below τ = 10 as invalid black regions, then applying dilation and erosion to reduce isolated outliers.
3 More experiments
Experiments test SparseNeuS with varying numbers of input views and compare it visually with established reconstruction methods on DTU and BlendedMVS. The method produces plausible two-view reconstructions, smoother and more complete surfaces, and strong generalization to an unseen dataset.
- Different number of views as input: SparseNeuS produces plausible geometries for an unseen DTU object using only two input views and supports an arbitrary number of views.Experiments evaluate 2–8 input views on Scan105 of DTU.
- Qualitative comparisons on DTU: Compared with MVSNerf, SparseNeuS produces smoother, less noisy meshes through fast network inference on DTU.The passage attributes MVSNerf’s noisy surfaces to density representation lacking a local surface constraint.
- Qualitative comparisons on DTU: Short per-scene fine-tuning improves SparseNeuS results with finer details and cleaner, more accurate surfaces.
- Qualitative comparisons on DTU: SparseNeuS reconstructs more complete and accurate surfaces than NeuS, whose geometries are described as incomplete and distorted.The passage attributes NeuS’s limitation to radiance ambiguity.
- Qualitative comparisons on BlendedMVS: Despite not being trained on BlendedMVS, SparseNeuS produces cleaner and more complete results than MVSNerf, demonstrating strong generalizability.For the Buddha head, Colmap recovers only sparse points while SparseNeuS produces much more complete geometry.