Source-linked AI summary
HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details
Yiqun Wang, Ivan Skorokhodov, Peter Wonka
TL;DR
Neural surface reconstruction methods often produce over-smoothed geometry because learning high-frequency details is difficult. HF-NeuS models surfaces with signed distance fields, decomposes them into base and displacement functions, and uses transformed transparency with adaptive optimization; it reports improved quantitative and visual reconstruction quality, while retaining limitations on thin or poorly observed structures.
Problem
Neural surface reconstruction methods struggle to learn high-frequency geometry details, leading to over-smoothed reconstructed shapes.
Method
HF-NeuS combines transformed-SDF transparency modeling, base-plus-displacement SDF decomposition, and spatially adaptive optimization.
Results
HF-NeuS outperforms previous work in quantitative reconstruction quality and visual inspection, including improved high-frequency surface details.
Takeaways & Limitations
The method reconstructs more refined surfaces by focusing optimization on regions where the implicit distance-field representation is not ideal.
Takeaways & Limitations
HF-NeuS can overfit visible radiance without accurately reconstructing thin structures, and its additional implicit function increases optimization requirements.
Abstract
from arXiv · showhide
Neural rendering can be used to reconstruct implicit representations of shapes without 3D supervision. However, current neural surface reconstruction methods have difficulty learning high-frequency geometry details, so the reconstructed shapes are often over-smoothed. We develop HF-NeuS, a novel method to improve the quality of surface reconstruction in neural rendering. We follow recent work to model surfaces as signed distance functions (SDFs). First, we offer a derivation to analyze the relationship between the SDF, the volume density, the transparency function, and the weighting function used in the volume rendering equation and propose to model transparency as transformed SDF. Second, we observe that attempting to jointly encode high-frequency and low-frequency components in a single SDF leads to unstable optimization. We propose to decompose the SDF into a base function and a displacement function with a coarse-to-fine strategy to gradually increase the high-frequency details. Finally, we design an adaptive optimization strategy that makes the training process focus on improving those regions near the surface where the SDFs have artifacts. Our qualitative and quantitative results show that our method can reconstruct fine-grained surface details and obtain better surface reconstruction quality than the current state of the art. Code available at https://github.com/yiqun-wang/HFS.
1 Introduction
Neural rendering methods model 3D scenes implicitly, but surface-oriented approaches still struggle to reconstruct high-frequency geometry. HF-NeuS addresses this with transformed-SDF transparency modeling, SDF decomposition, and adaptive optimization.
- NeRF models 3D objects with neural density and view-dependent color functions, inspiring substantial follow-up work.
- NeuS and VolSDF constrain volume rendering with signed distance fields to support meaningful surface extraction.
- HF-NeuS derives the relationship between signed distance, density, transparency, and rendering weights, then models transparency as a transformed SDF.
- A single SDF struggles to learn high-frequency details, motivating decomposition into base and displacement functions with coarse-to-fine optimization.
- HF-NeuS introduces spatially adaptive weighting so optimization focuses on problematic regions in the distance field.
- The method reports clearer surface-reconstruction improvement on the 15-scene DTU benchmark and qualitative comparisons.
2 Related Work
Related work spans classical multi-view reconstruction, neural implicit surfaces, and methods for recovering high-frequency detail from images. These approaches differ in whether they use explicit geometric representations or continuous implicit functions with differentiable rendering.
- Multi-view 3D reconstruction: Classical multi-view reconstruction uses voxel-based methods to decide surface occupancy in discretized 3D space.
- Multi-view 3D reconstruction: Point-based methods estimate depth from view correlations, fuse depth maps into point clouds, and reconstruct meshes afterward.
- Neural implicit surfaces: Neural implicit methods reconstruct continuous shape functions directly from 2D images using differentiable rendering.
- High-frequency detail reconstruction: Positional encoding and sinusoidal activations guide neural networks toward reconstructing high-frequency information.
3 Method
HF-NeuS models surface reconstruction with signed distance fields by deriving a transparency-based rendering formulation, separating coarse and high-frequency geometry, and adapting optimization near problematic surface regions.
- 3.1 Modeling transparency as transformed SDF: HF-NeuS derives criteria for transforming signed distances into transparency, linking transparency, volume density, and the rendering weighting function.The method models transparency directly as a transformed signed distance function and uses its derivative to obtain density-related terms.
- 3.1 Modeling transparency as transformed SDF: The transparency derivative is the negative weighting function, whose maximum is intended to coincide with the ray-surface intersection.For planar surfaces with a single intersection, designing the transparency derivative to reach a minimum at the surface yields a weighting maximum there.
- 3.1 Modeling transparency as transformed SDF: HF-NeuS uses a logistic sigmoid transparency with slope parameter s and simplifies discretized density computation relative to NeuS.The explicit transparency function also supports inverse-CDF sampling without VolSDF’s complex sampling scheme, while avoiding NeuS’s two-point computation.
- 3.2 Implicit displacement field without 3D supervision: The signed distance field is decomposed into a base function and a displacement function along the base surface normal to separate coarse and high-frequency geometry.This decomposition supports multi-scale fitting because a single implicit function struggles to learn low- and high-frequency information simultaneously.
- 3.2 Implicit displacement field without 3D supervision: A coarse-to-fine strategy gradually increases learned frequency content to improve high-frequency reconstruction from multiple images.The displacement function complements the base function, making high-frequency information easier to learn.
- 3.3 Modeling an adaptivate transparency function: HF-NeuS varies the transparency scale s spatially using the signed-distance gradient norm, increasing it where gradients exceed 1 to target artifacts.Larger s values demand greater distance-function precision and magnify errors, especially near the surface.
4 Experiments
HF-NeuS is evaluated on DTU, NeRF-synthetic, and BlendedMVS using quantitative surface and image metrics, qualitative comparisons, and module ablations. The experiments show improved reconstruction quality and high-frequency detail, while also exposing remaining overfitting and computational limitations.
- Datasets: Experiments evaluate HF-NeuS on DTU, NeRF-synthetic, and BlendedMVS, including scenes with sharp or high-frequency features.DTU uses 15 models, while nine additional challenging scenes come from NeRF-synthetic and BlendedMVS.
- Comparison: HF-NeuS surpasses competing methods on DTU surface reconstruction and PSNR, while recovering finer geometric details such as windows and bird feathers.Chamfer distance measures surface fidelity, whereas PSNR measures image quality.
- Comparison: On challenging high-frequency scenes, the gap between HF-NeuS and other methods widens, with qualitative and quantitative advantages over the baselines.NeRF can achieve strong PSNR while performing poorly on Chamfer distance, showing that image quality does not ensure surface fidelity.
- Ablation study: The ablations show that coarse-to-fine optimization can prevent training divergence, while directly introducing high frequencies can cause overfitting.An increase in PSNR does not necessarily imply improved surface reconstruction fidelity.
- Limitations: HF-NeuS still overfits some ground-truth radiance and can miss thin structures, such as individual ropes, when lighting or texture complicates reconstruction.The limitation is illustrated with rope grids and a DTU Bunny case.
5 Conclusion
HF-NeuS improves multi-view surface reconstruction by combining a transparency derivation, decomposed signed distance functions, and adaptive scale constraints. The method outperforms prior work in quantitative reconstruction quality and visual inspection, but requires extra computation and remains susceptible to radiance overfitting.
- Method: HF-NeuS models multi-view surfaces with a derived signed-distance-to-transparency formulation and two independent implicit functions.The decomposition uses a base function and a displacement function to support refined reconstruction.
- Method: Adaptive scale constraints focus optimization on regions where the implicit-function distribution is not ideal.This design is intended to improve reconstruction of refined surface structure.
- Results: HF-NeuS outperforms current state-of-the-art methods in quantitative reconstruction quality and visual inspection.The conclusion reports both numerical and visual improvements.
- Limitations: The method requires additional computational resources and coding complexity because it optimizes an extra implicit function.The paper also observes some overfitting to ground-truth radiance because 3D supervision is absent.
A.1 Frequency ablation study
The frequency ablation varies the coarse-to-fine parameter and the number of positional-encoding bands to study their effects on reconstructed surface quality. Too little frequency progression oversmooths surfaces, whereas too much introduces artifacts.
- Frequency settings: The ablation varies αd with L = 16 and varies L with αd = 0.5 on the DTU Buddha model.Each model is trained for 300K iterations.
- Coarse-to-fine parameter: Small αd values produce oversmoothed reconstructions, while large values introduce many artifacts.The study adopts αd = 0.5 for the reported configuration.
A.2 Adaptive transparency ablation study
The adaptive transparency ablation tests whether focusing optimization on difficult surface regions improves reconstruction. It helps recover holes, sparsely observed cheekbones, and thin belt details.
- Adaptive transparency: Adaptive transparency better reconstructs holes, cheekbones with few supporting views, and fine belt details than the non-adaptive strategy.The comparison uses Lego, DTU skull, and BlendedMVS dog models.
A.3 Additional reconstruction results
Additional comparisons evaluate HF-NeuS against NeuS and VolSDF across DTU, NeRF-Synthetic, and BlendedMVS, including an adaptive-transparency ablation.
- HF-NeuS is qualitatively compared with NeuS and VolSDF on the DTU, NeRF-Synthetic, and BlendedMVS datasets.
- The adaptive-transparency ablation compares reconstructed surfaces without the strategy against surfaces using adaptive transparency.
A.4 Qualitative comparison of different design choices
The design-choice comparisons examine transparency modeling, frequency control, regularization, few-image training, and SDF decomposition through qualitative reconstructions and one reported PSNR comparison.
- The proposed transparency model avoids division-related numerical problems and separate sampling of section points and mid-points.The OUR Base-Sigmoid variant also shows better roof geometry consistency than VolSDF and NeuS in the reported comparison.
- HF-NeuS positional encoding avoids high-frequency information at training onset, making optimization more stable than the compared Siren-based approach.The Siren variant produced high PSNR but low geometry fidelity in the reported comparison.
- Constraining the base SDF with Eikonal regularization improves reconstruction quality relative to omitting that regularization.The reported degradation without base-SDF regularization is slight.
- With 10% of the training images, HF-NeuS preserves complete object structure and reconstructs thin stripes better than NeuS.For the shown test image, PSNR is 31.77 for HF-NeuS versus 28.31 for NeuS.
- The base SDF reconstructs a smooth Buddha model, while the displacement function adds cracks and small forehead holes.
A.6 Qualitative ablation study
The ablation study shows that separating displacement-based detail learning from coarse-to-fine frequency control improves geometric fidelity and image reconstruction quality, with the full adaptive strategy performing best.
- Learning high-frequency details directly from image information can overfit images without reconstructing correct geometry.Using an implicit displacement function reduces noise but can still leave large geometric errors.
- Coarse-to-fine positional encoding explicitly controls frequency and makes training more stable without 3D supervision.
- IDF+C2F further improves geometric fidelity and PSNR compared with using only the coarse-to-fine strategy.
- The full adaptive strategy improves geometric fidelity and PSNR beyond IDF+C2F.
- The SDF decomposition visualization presents the reference image, base SDF, and combined SDF in separate views.
- The ablation figure shows synthetic images on the left and reconstructed surfaces on the right for each setting.
A.7 The visualization of local errors
Local-error heatmaps show that HF-NeuS achieves larger improvements in fine details, including the roof and the excavator shovel.
- HF-NeuS shows higher improvement in local details such as the roof and the excavator shovel.
B Additional implementation details
HF-NeuS adds implementation mechanisms for constraining displacement, scheduling detail refinement, and sampling near surfaces. Training each scene takes around 20 hours, while mesh extraction and rendering add substantial inference costs.
- Runtime: Training takes around 20 hours per scene for 300k iterations, while high-resolution mesh extraction takes about 60 seconds.Rendering a 1600x1200 image takes around 540 seconds.
- Displacement constraint: The displacement constraint is relaxed near the surface over training, allowing the model to fit high-frequency details progressively.The constraint starts tight to accelerate base-SDF convergence and is gradually relaxed as iterations increase.
- Hierarchical sampling: Hierarchical sampling first draws 64 uniform points, then 64 additional points according to the weighting function.Sampling probability is higher where rays intersect the surface and lower elsewhere.
- Hierarchical sampling: The scale parameter s controls whether adaptive samples concentrate close to the surface or spread around it.The method modulates s using a coefficient c that depends increasingly on gradients near the surface as s increases.