Source-linked AI summary
Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement
Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, Gang Zeng
TL;DR
NeRF representations are difficult to manipulate and inefficient for common mesh-based rendering workflows, motivating textured surface mesh reconstruction from images. NeRF2Mesh initializes geometry and decomposed appearance with NeRF, then adaptively refines mesh geometry and appearance before baking textures for real-time rendering. The method achieves enhanced mesh quality and competitive rendering quality, while remaining limited in relighting, complex view-dependent effects, and semitransparency.
Problem
NeRF’s implicit volumetric representation lacks the practical manipulation and hardware support of polygonal meshes, limiting efficient rendering and downstream use.
Method
NeRF2Mesh extracts a coarse mesh from an appearance-decomposed NeRF, jointly refines vertex positions, face density, and appearance using re-projected rendering errors, and bakes the result into textures.
Results
NeRF2Mesh achieves enhanced surface mesh quality, relatively smaller mesh size, and competitive rendering quality, with real-time rendering and interactive editing support.
Takeaways & Limitations
The reconstructed meshes are practical assets for downstream manipulation and editing with common 3D hardware and software.
Takeaways & Limitations
Baking illumination into textures restricts relighting, the small appearance network struggles with complex view-dependent effects, and single-pass rasterization cannot handle semitransparency.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) have constituted a remarkable breakthrough in image-based 3D reconstruction. However, their implicit volumetric representations differ significantly from the widely-adopted polygonal meshes and lack support from common 3D software and hardware, making their rendering and manipulation inefficient. To overcome this limitation, we present a novel framework that generates textured surface meshes from images. Our approach begins by efficiently initializing the geometry and view-dependency decomposed appearance with a NeRF. Subsequently, a coarse mesh is extracted, and an iterative surface refining algorithm is developed to adaptively adjust both vertex positions and face density based on re-projected rendering errors. We jointly refine the appearance with geometry and bake it into texture images for real-time rendering. Extensive experiments demonstrate that our method achieves superior mesh quality and competitive rendering quality.
1. Introduction
NeRF2Mesh addresses the mismatch between efficient but difficult-to-use NeRF representations and practical polygonal meshes by refining coarse NeRF-extracted geometry into textured surface meshes. Its adaptive refinement and appearance decomposition target mesh quality, compactness, real-time rendering, and editing.
- NeRF representations render realistically but are difficult to manipulate and slow to render because implicit functions and specialized ray marching receive poor hardware support.
- Existing mesh-oriented methods can produce inaccurate surfaces, inconvenient feature-space textures, over-smoothed geometry, redundant mesh elements, or limited topology and scene-level reconstruction.
- NeRF2Mesh jointly refines geometry and appearance from a coarse mesh extracted from an appearance-decomposed NeRF.
- Its iterative refinement adaptively subdivides complex surfaces and decimates simpler ones according to re-projected 2D image errors.
- The resulting meshes offer enhanced surface quality, relatively smaller mesh size, competitive rendering quality, and real-time editing with common 3D hardware and software.
2. Related Work
Related work combines NeRF or differentiable rendering with explicit meshes, while SDF-based approaches define extractable surfaces. These approaches still face challenges involving complex topology, object-level scope, over-smoothed geometry, and redundant mesh complexity.
- NeRF-based methods have been explored to combine NeRF rendering advantages with mesh representations, but related approaches can retain inaccurate surfaces or inconvenient feature-space textures.
- Direct explicit mesh reconstruction remains difficult for complex scenes with intricate topology, and many methods assume a template mesh with fixed topology.
- SDF-based methods determine surfaces through a zero level set, commonly followed by Marching Cubes mesh extraction.
- The related approaches described are limited by over-smoothed geometry and redundant vertices and faces when preserving details.
3. Method
NeRF2Mesh uses a two-stage pipeline that initializes geometry and decomposed appearance with grid-based NeRF, then jointly refines a coarse mesh and exports textured images. Surface refinement adjusts vertices and face density from re-projected errors, while differentiable rendering and regularization support mesh quality and real-time compatibility.
- Stage 1: Efficient NeRF Training: Appearance is decomposed into view-independent diffuse color and view-dependent specular color using a color grid and two shallow MLPs.The diffuse term can become an RGB texture, while specular features and a small MLP support later view-dependent rendering.
- Stage 1: Efficient NeRF Training: Stage 1 renders rays by accumulated density weights and optimizes pixelwise color error, with specular, entropy, and optional density regularization.The entropy term sharpens surfaces, while total variation regularization reduces floaters in unbounded outdoor scenes.
- Stage 2: Surface Mesh Refinement: Stage 2 inherits the appearance model through differentiable rasterization and jointly optimizes geometry and appearance with the pixelwise color loss.The inherited appearance reduces the training needed for stage 2 to converge.
- Stage 2: Surface Mesh Refinement: Face density is adapted from accumulated re-projected rendering errors: high-error faces are subdivided, while low-error faces are decimated and remeshed.The method uses the 95th-percentile error for subdivision and the 50th-percentile error for decimation, repeating updates during training.
- Mesh Exportation: The final mesh uses UV unwrapping and separate diffuse and specular images, with a fragment shader enabling real-time view-dependent rendering.Laplacian smoothing regularizes vertex offsets to prevent abrupt geometry, and the exported mesh is compatible with common 3D software and hardware.
4. Experiment
Experiments evaluate surface quality, mesh compactness, rendering quality, efficiency, and ablations across synthetic, forward-facing, and unbounded scenes. NeRF2Mesh reconstructs thin and complex structures faithfully while retaining competitive rendering and practical export characteristics.
- Surface reconstruction: The method reconstructs thin structures and complex topology with high fidelity, while producing more orderly meshes than competing methods.Qualitative comparisons focus on dense foliage and rope-net structures, where other methods fail to recover complex geometry accurately.
- Surface reconstruction: The method achieves the best averaged Chamfer Distance across NeRF-synthetic scenes, with particularly strong results on ficus, ship, and lego.Performance is slightly worse on scenes with many non-Lambertian surfaces because the appearance network can model lighting effects as erroneous geometry.
- Mesh size: Adaptive refinement produces relatively fewer vertices and faces while preserving enhanced mesh quality.The process can add vertices for surface detail while reducing faces to control overall mesh size.
- Rendering quality: Rendering quality decreases from volumetric NeRF to mesh, but smoothness regularization balances surface smoothness against rendering quality.Removing the regularizer improves rendering quality at the expense of surface quality, while the mesh approach outperforms NVdiffrec in the reported comparison.
- Rendering quality: Exported meshes provide comparable rendering quality across datasets and yield compact, intuitive texture images.The reported visualizations include rendering comparisons and texture exports from different methods.
- Efficiency: Training and mesh export take roughly 1 hour per scene on a single 16GB NVIDIA V100, and the lightweight meshes support real-time rendering on OpenGL-enabled devices.Competing methods are reported to require several hours or days with higher hardware demands.
- Ablation: Removing iterative refinement makes face density nearly uniform, increasing mesh size and slightly reducing rendering quality.Removing smoothness regularization instead causes irregularities, self-intersections, and larger meshes despite better rendering quality.
5. Limitations and Conclusion
The method exports textured meshes for real-time rendering and downstream manipulation, but baked illumination prevents relighting and limited appearance modeling weakens reconstruction in complex view-dependent regions.
- Limitations: Baking illumination into textures restricts the method’s ability to perform relighting.The limitation arises from the difficulty of estimating unknown lighting conditions without compromising reconstruction quality.
- Limitations: A relatively small appearance network struggles with complex view-dependent effects, causing inferior surface quality in those regions.The resulting appearance errors can confound iterative surface refinement.
- Conclusion: The framework reconstructs textured surface meshes from multi-view RGB images and bakes appearance into textures for real-time rendering.The conclusion emphasizes enhanced surface quality, particularly for thin structures, and compatibility with downstream manipulation and editing.
A. Additional Implementation Details
Implementation details include texture-processing steps and adaptive face-density refinement. The refinement increases density where data indicate greater error and reduces it elsewhere.
- Texture export: Texture processing repairs connected-component borders by out-painting one pixel and quantizes diffuse and specular features into 8-bit PNG images.The implementation also explores dataset-specific hyper-parameters because object and unbounded-scene data require different settings.
- Adaptive refinement: Adaptive face-density refinement decimates faces with high error and remeshes them using a target edge length tied to the mesh bounding-box diagonal.The stated remeshing threshold uses an average edge length of 2% of the bounding-box diagonal.
B.1. Additional Qualitative Results
Additional results demonstrate relighting, level-of-detail support, and visual effects of TV-loss regularization, while also documenting limitations involving transparency, fur, and intricate view-dependent effects.
- Relighting: Relighting uses the reconstructed mesh with a rotating point light and only the diffuse texture.The experiment targets scenes captured under predominantly ambient lighting conditions.
- Levels of detail: The textured surface mesh supports level-of-detail simulation through mesh decimation.The paper links this capability to accurate surface geometry and geometric-error minimization.
- Ablation: TV-loss ablation visualizes how removing surface regions affects internal geometry and mesh quality.The visualization is presented as an additional qualitative analysis.
- Limitations: Single-layer rasterization cannot handle semi-transparent objects such as glass bottles and tends to learn opaque textures.This limitation follows from the method’s rasterization design.
- Limitations: Animal fur is difficult to emulate because mesh-surface smoothness regularization limits volumetric simulation.The limitation concerns phenomena that ordinarily require volumetric representation.
- Limitations: A relatively small appearance network cannot model intricate view-dependent effects, causing vertex manipulation, reduced smoothness, and inaccurate geometry.The paper identifies improved surface-and-material decomposition as a possible way to address this issue.
B.2. Additional Quantitative Results
The paper reports per-scene rendering-quality evaluations in Tables 7–9 and additional regularization-loss ablations in Table 6.
- Per-scene evaluation: Per-scene rendering-quality evaluation results are listed in Table 7.
- Per-scene evaluation: Per-scene rendering-quality evaluation results are listed in Table 8.