Source-linked AI summary
Improving neural implicit surfaces geometry with patch warping
François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, Mathieu Aubry
TL;DR
Neural implicit surfaces have limited reconstruction accuracy because neural networks struggle with high-frequency textures. NeuralWarp adds geometry-based patch warping and visibility handling to optimize photo-consistency, improving 3D metrics by 20% on average over unsupervised implicit methods on DTU and EPFL.
Problem
Neural implicit surface reconstruction remains less accurate because radiance networks struggle to represent high-frequency textures.
Method
NeuralWarp combines volumetric rendering with implicit-geometry patch warping, robust SSIM comparison, and masking of invalid reprojections.
Results
20%: average improvement in 3D reconstruction metrics over state-of-the-art unsupervised neural implicit-surface methods on DTU and EPFL.
Takeaways & Limitations
Image warping lets implicit reconstruction exploit high-frequency texture information while retaining volumetric rendering, producing strong performance improvements on DTU and EPFL.
Takeaways & Limitations
The method struggles with high-frequency geometry, increases computation and memory costs, and can produce artifacts from reflections or view-dependent effects.
Abstract
from arXiv · showhide
Neural implicit surfaces have become an important technique for multi-view 3D reconstruction but their accuracy remains limited. In this paper, we argue that this comes from the difficulty to learn and render high frequency textures with neural networks. We thus propose to add to the standard neural rendering optimization a direct photo-consistency term across the different views. Intuitively, we optimize the implicit geometry so that it warps views on each other in a consistent way. We demonstrate that two elements are key to the success of such an approach: (i) warping entire patches, using the predicted occupancy and normals of the 3D points along each ray, and measuring their similarity with a robust structural similarity (SSIM); (ii) handling visibility and occlusion in such a way that incorrect warps are not given too much importance while encouraging a reconstruction as complete as possible. We evaluate our approach, dubbed NeuralWarp, on the standard DTU and EPFL benchmarks and show it outperforms state of the art unsupervised implicit surfaces reconstructions by over 20% on both datasets.
1. Introduction
Neural implicit reconstruction struggles with high-frequency textures, so NeuralWarp adds geometry-based patch warping to directly optimize multi-view photo-consistency. Robust patch comparison and visibility handling produce substantial gains on DTU and EPFL.
- Neural implicit methods jointly optimize geometry and color but struggle to represent high-frequency textures, limiting reconstruction accuracy.
- NeuralWarp reprojects source-image patches through implicit geometry to preserve high-frequency texture information during geometry optimization.
- Patch warping uses SSIM over entire patches rather than direct color differences, accommodating non-Lambertian materials.
- The method handles invalid, invisible, or occluded reprojections while encouraging reconstruction completeness.
- 20%: average improvement in 3D reconstruction metrics over state-of-the-art unsupervised neural implicit-surface methods on DTU and EPFL.
2. Related Work
The paper bridges multi-view photo-consistency and neural implicit surfaces by combining image-based patch warping with volumetric rendering. Unlike related approaches focused on learned view synthesis or pretrained features, it optimizes geometry directly without multi-scene training.
- Multi-view reconstruction methods include classical MVS representations and neural implicit occupancy or SDF representations optimized for image rendering.
- Classical MVS often relies on multi-step depth optimization and fusion, whereas neural implicit methods jointly optimize continuous scene representations.
- Unlike MVSDF, the method does not require a pretrained network and uses image patches rather than pretrained CNN features for consistency.
- The optimization targets geometry quality directly rather than training networks across multiple scenes for new-scene view synthesis.
- The approach combines volumetric rendering with patch warping, aggregating predicted radiance or source-view patches from points sampled along camera rays.
3. Method
NeuralWarp extends volumetric neural rendering by warping source-image patches through implicit geometry, then optimizing geometry with robust photo-consistency while downweighting invalid projections and occlusions.
- 3.1. Volumetric rendering of radiance field: The method builds on volumetric rendering, where occupancy-derived weights combine radiance values sampled along each camera ray.The geometry network predicts occupancy or an SDF-derived quantity, while the radiance network predicts view-dependent color and normals come from geometry differentiation.
- 3.2. Warping images with implicit geometry: NeuralWarp replaces radiance-network color predictions with source-image colors projected from sampled 3D points, allowing geometry to enforce cross-view photo-consistency.The warped value averages projected source colors along the ray, using the implicit geometry network rather than memorized scene colors.
- 3.3. Optimizing geometry from warped patches: The warping loss compares reference and warped patches with SSIM, rather than direct color differences that can create artifacts under viewpoint-dependent intensity changes.SSIM is used as the photometric distance, with ℓ1 reserved for an ablation.
- 3.2. Warping images with implicit geometry: Entire patches are warped by approximating each sampled 3D neighborhood as a plane and applying a normal-induced homography between calibrated views.The homography maps every pixel in a reference patch to corresponding source-image locations; experiments use 11 × 11 patches.
- 3.3. Optimizing geometry from warped patches: Validity masks suppress unreliable warps caused by invalid projections or occlusions, while soft weighting reduces the influence of near-surface visibility problems.Projection validity includes out-of-image projections, opposite-sided views, and near-degenerate camera-plane configurations; the final mask combines projection and occlusion masks.
- 3.4. Optimization details: More than 20% improvement on mean DTU metrics is reported against neural implicit surface approaches without additional inputs.The comparison excludes methods using masks or supervised depth from the unsupervised neural-implicit subset, while MVSDF is noted as not directly comparable because of custom filtering.
4. Experiments
Experiments on DTU and EPFL show that NeuralWarp improves over unsupervised neural implicit surface methods, while ablations support combining volumetric rendering with patch warping. The method remains weaker than COLMAP for some full-scene or high-frequency geometry outcomes and adds computational costs.
- DTU benchmark: NeuralWarp outperforms existing unsupervised neural implicit surface methods by a large margin on DTU, with larger gains on highly textured scenes.On weakly textured scenes, its performance is comparable to other methods.
- Qualitative results: NeuralWarp’s warped images preserve high-frequency texture that volumetric rendering smooths, accompanying important accuracy improvements on DTU.Figure 4 visualizes rendering and geometric error maps against VolSDF, with blue indicating low error and yellow high error.
- EPFL benchmark: NeuralWarp improves 3D reconstruction metrics by more than 20% over other neural implicit surfaces on EPFL.COLMAP performs best on the full metric, while NeuralWarp performs best on the center metrics.
- Ablation study: The ablation compares no, pixel, and patch warping together with volumetric rendering and self-occlusion detection on all DTU scenes.The supplied ablation results state that removing volumetric rendering improves over baseline but performs worse than combining volumetric and warping losses.
- Limitations: Compared with COLMAP, NeuralWarp struggles with high-frequency geometry, while patch processing and occlusion masking increase memory use and computation.Patch similarity also does not model reflections, which can produce artifacts.
5. Conclusion
The paper presents multiview reconstruction with implicit functions by combining image warping and volumetric rendering. It reports strong performance improvements on the DTU and EPFL datasets and improved access to high-frequency texture.
- The method combines image warpings with volumetric rendering for multiview reconstruction using implicit functions.
- The approach can take advantage of high-frequency texture more easily than existing neural implicit surface methods.
- The paper reports strong performance improvements on the DTU and EPFL datasets.
Supplementary material Improving neural implicit surfaces geometry with patch warping
The supplementary material reports a failure case, an ablation study, and additional qualitative DTU results.
- The supplementary material first presents a failure case of the method.
- It then reports a detailed ablation study and additional qualitative results on the DTU dataset.
1. Failure case
The method fails on scenes with strong view-dependent effects, where multi-view consistency lacks useful signal and introduces reconstruction artifacts.
- Specularities and saturated pixels create a failure case for the method on DTU scan 65.The scene has large view-dependent effects.
- Multi-view consistency produces artifacts when view-dependent effects provide no useful reconstruction signal.
2. Detailed results ablation study
The ablation study reports overall, accuracy, and completeness metrics for each scene, highlighting where volumetric rendering is especially beneficial.
- The ablation study evaluates three quality metrics: overall, accuracy, and completeness, across individual scenes.
- Adding volumetric rendering is especially beneficial on poorly textured, highly specular scenes, including scans 63, 65, 97, and 110.
- Tables 1, 2, and 3 provide detailed overall, accuracy, and completeness metrics for the ablation study on DTU.
3. Additional DTU visualizations
Additional DTU results are presented through qualitative comparisons between VolSDF and the proposed method.
- Figures 2, 3, and 4 provide additional DTU results.
- The visualizations compare VolSDF with the proposed method across three figures.