PaperlayerView exact revision on arXiv

Source-linked AI summary

Surface Reconstruction from Point Clouds: A Survey and a Benchmark

Zhangjin Huang, Yuxin Wen, Zihao Wang, Jinjuan Ren, Kui Jia

arXiv:2205.02413v1cs.CV

TL;DR

Surface reconstruction from imperfect point clouds is an ill-posed problem, and existing reviews lacked thorough comparisons on a common benchmark. This paper reviews classical and deep learning methods, builds a large-scale mixed synthetic and real-scanned benchmark, and systematically evaluates robustness and generalization. The studies find that some classical methods outperform deep learning methods in robustness and generalization, while misalignment, missing points, and outliers remain unsolved.

  • Problem

    Surface reconstruction from discrete point clouds is ill-posed and affected by scanning imperfections, while existing reviews lacked thorough investigation on a common benchmark.

  • Method

    The paper reviews classical and deep learning methods and benchmarks them on synthetic and real-scanned object- and scene-level data containing practical sensing imperfections.

  • Results

    Some classical methods perform better than deep learning methods in both robustness and generalization, while deep learning methods struggle with complex-shape generalization.

  • Takeaways & Limitations

    The benchmark and studies identify conditions, strengths, and limitations of existing methods for practitioners and future research.

  • Takeaways & Limitations

    Misalignment, missing points, and outliers remain less addressed and unsolved by existing surface reconstruction methods.

Abstract

from arXiv · show

Reconstruction of a continuous surface of two-dimensional manifold from its raw, discrete point cloud observation is a long-standing problem. The problem is technically ill-posed, and becomes more difficult considering that various sensing imperfections would appear in the point clouds obtained by practical depth scanning. In literature, a rich set of methods has been proposed, and reviews of existing methods are also provided. However, existing reviews are short of thorough investigations on a common benchmark. The present paper aims to review and benchmark existing methods in the new era of deep learning surface reconstruction. To this end, we contribute a large-scale benchmarking dataset consisting of both synthetic and real-scanned data; the benchmark includes object- and scene-level surfaces and takes into account various sensing imperfections that are commonly encountered in practical depth scanning. We conduct thorough empirical studies by comparing existing methods on the constructed benchmark, and pay special attention on robustness of existing methods against various scanning imperfections; we also study how different methods generalize in terms of reconstructing complex surface shapes. Our studies help identify the best conditions under which different methods work, and suggest some empirical findings. For example, while deep learning methods are increasingly popular, our systematic studies suggest that, surprisingly, a few classical methods perform even better in terms of both robustness and generalization; our studies also suggest that the practical challenges of misalignment of point sets from multi-view scanning, missing of surface points, and point outliers remain unsolved by all the existing surface reconstruction methods. We expect that the benchmark and our studies would be valuable both for practitioners and as a guidance for new innovations in future research.

1 Introduction

Surface reconstruction from point clouds is ill-posed and complicated by scanning imperfections, motivating a common benchmark for comparing classical and deep learning methods. The paper contributes such a benchmark and systematic studies that reveal robustness, generalization, metric, and unresolved-challenge patterns.

  • 1 Introduction: Surface reconstruction is a fundamental computer vision and graphics problem because depth sensors commonly produce point clouds for downstream applications.
  • 1 Introduction: The paper reviews methods by their surface-geometry priors and benchmarks both classical optimization-based and deep learning solutions.
  • 1 Introduction: The benchmark combines synthetic and real-scanned object- and scene-level data with noise, non-uniform sampling, outliers, missing points, and multi-view misalignment.These imperfections are intended to reflect challenges encountered in practical 3D scanning.
  • 1 Introduction: Misalignment, missing points, and outliers remain less addressed and unsolved by existing surface reconstruction methods.
  • 1 Introduction: Classical methods can outperform deep learning methods in both robustness and generalization, while deep models struggle to generalize to complex shapes.
  • 1 Introduction: Surface normals are key to reconstruction success, and quantitative metrics can disagree with visually pleasant results.

2 Problem Statement

The problem is to recover a continuous surface from a discrete point set produced by scanning. Because this recovery is ill-posed, the paper formulates reconstruction as regularized optimization using explicit or implicit surface representations.

  • 2 Problem Statement: Given a scanned point set P, the goal is to recover its underlying continuous surface S* despite the ill-posed nature of the problem.
  • 2 Problem Statement: Regularization imposes geometry-aware priors, such as smoothness or fairness, to obtain an approximation S from the observed points.
  • 2 Problem Statement: The abstract objective uses a data-fidelity loss L, a geometric regularizer R, and penalty weight λ, then instantiates explicit or implicit representations.

3 Surface Reconstruction with a Categorization of Geometric Priors

Surface reconstruction combines data fidelity with geometric priors because discrete point clouds underdetermine a continuous surface. The survey organizes these priors into triangulation, smoothness, template-based, modeling, learning-based, and hybrid approaches.

  • Explicit reconstruction fits a mapping from sampled surface locations, whereas implicit reconstruction learns a function whose values are constrained at observed points.The survey discusses explicit and implicit data-fidelity formulations, including point-set distances and signed-distance or occupancy functions.
  • Data fidelity alone is insufficient for geometry-plausible reconstruction, so regularization imposes geometric priors on the recovered surface.The survey categorizes these priors as triangulation-based, smoothness, template-based, modeling, learning-based, and hybrid.
  • 3.1 Triangulation-based Prior: Triangulation-based methods approximate locally differentiable surfaces with meshes and select valid triangular faces generated from the observed points.Delaunay triangulation satisfies explicit data fidelity simultaneously; Greedy Delaunay and Ball-Pivoting select faces using different procedures.
  • 3.2 Surface Smoothness Priors: Smoothness priors either locally smooth noisy observations or constrain function complexity, with Point Set Surfaces combining both strategies.Poisson Surface Reconstruction regularizes the normal field but usually produces over-smooth results; Screened Poisson adds first- and second-order regularization.
  • 3.3 Template-based Priors: Template-based reconstruction estimates weighted templates and their parameters by minimizing a distance between the formed surface and observed points.Templates may be geometric primitives or complex shapes from an auxiliary dataset, while RANSAC is commonly used for primitive fitting.

3.4 Modeling Priors

Modeling priors regularize reconstruction through the design of the function class itself, including deep networks used without training. Learning-based priors instead optimize model parameters from training shapes and may operate globally or on local shape primitives.

  • 3.4 Modeling Priors: Modeling priors arise from the design of the geometric model and restrict reconstruction to surfaces representable by that model.Constraining the hypothesis space can promote smoother surfaces, while model choice regularizes which surface types can be represented.
  • 3.4 Modeling Priors: Untrained deep networks can serve as geometric priors, with later methods extending global modeling to local implicit functions.Deep Geometric Prior, Point2Mesh, and SAIL-S3 use network architectures as priors; analyses also characterize modeling properties of MLPs and convolutional networks.
  • 3.4 Modeling Priors: Learning-based priors optimize model parameters on training pairs of point sets and ground-truth surfaces, then fit observations through latent codes or related parameters.The learned prior can support explicit or implicit reconstruction, and auto-encoder architectures can directly produce latent codes.
  • 3.4 Modeling Priors: Learned priors may encode global semantic shape patterns or local shape primitives, depending on how training shapes are organized.Local priors support reconstruction of arbitrary surfaces that cannot be semantically categorized.
  • 3.4 Modeling Priors: Recent methods combine learning-based priors with triangulation, smoothness, or template-based priors to improve surface plausibility.Examples include PointTriNet for triangle generation, IMLSNet for smooth implicit modeling, and learned networks for geometric primitive fitting.

4 A Surface Reconstruction Benchmark

The benchmark combines synthetic and real-scanned point clouds of object and scene surfaces, explicitly modeling practical scanning imperfections and varying surface complexity. Its synthetic object pipeline collects and normalizes CAD meshes, groups them by algebraic complexity, simulates scanning, estimates normals, registers views, and samples fixed-size point clouds.

  • The benchmark synthesizes and practically scans object and scene surfaces while targeting the main imperfections encountered in real scanning.These include noise, outliers, non-uniform sampling, misalignment, and missing surface points.
  • 4.1.2 Groups of Varying Surface Complexities: Object instances are grouped into low-, middle-, and high-complexity categories using algebraic surface complexity.Algebraic complexity is measured by averaged local-patch approximation errors under a fixed maximum function degree rather than unstable high-degree polynomial fitting.
  • 4.1.3 Synthetic Point Cloud Scanning: Synthetic object scanning collects normalized CAD surfaces, simulates imperfect sensing from multiple viewpoints, estimates oriented normals, registers views, and applies FPS to produce fixed-size point clouds.The benchmark uses 80k, 120k, or 160k points for low-, middle-, and high-complexity surfaces, respectively.
  • 4.1.3 Synthetic Point Cloud Scanning: The benchmark models point-wise noise with truncated Gaussian perturbations and non-uniform sampling by replacing FPS with random sampling.Noise severity uses σ_noise values of 0.001, 0.003, and 0.006; random sampling produces less uniform local densities.
  • 4.1.3 Synthetic Point Cloud Scanning: Missing-surface conditions are simulated by restricting viewpoints to narrow trajectory bands, yielding approximately 99%, 94%, or 86% surface coverage.The three coverage levels correspond to using three, two, or one trajectory bands.

5 Experimental Set-up for Benchmarking Existing Surface Reconstruction Methods

The experiments compare representative classical and learning-based reconstruction methods on benchmark subsets under standardized preprocessing and multiple metrics. The setup is designed to assess method behavior across scanning imperfections and surface conditions while making comparisons more practical.

  • The study compares representative methods from different geometric-prior groups to identify their advantages and disadvantages under varied working conditions.The selection assumes conclusions from representative methods generalize within their broader method groups.
  • The empirical study samples 22 synthetic object surfaces across low-, middle-, and high-complexity groups, with six scanning conditions per instance.
  • Methods are evaluated with and without standard preprocessing, including outlier removal, de-noising, and point re-sampling for synthetic data.The preprocessing pipeline uses statistical outlier removal, Jets smoothing, and FPS retaining 40% of the original synthetic points.
  • Reconstruction quality is measured with Chamfer Distance, F-score, Normal Consistency Score, and the proposed Neural Feature Similarity metric.CD and F-score assess overall shape similarity, NCS assesses normal consistency, and NFS targets perceptual similarity in deep feature space.
  • Classical and learning-based methods are implemented through established libraries or released author code with method-specific parameter tuning.Results without preprocessing are reported separately because their comparative qualities are similar to the preprocessed results.

6 Main Results

The benchmark shows that robustness and generalization vary substantially across methods and scanning imperfections. Classical methods can outperform deep learning on complex-shape generalization, while missing points, misalignment, and outliers remain difficult.

  • 6 Main Results: Evaluation metrics can disagree: strong quantitative scores do not always correspond to visually pleasant reconstructions.CD and F-score may remain stable under misalignment even as NCS and NFS decline.
  • 6.1 The Remaining Challenges: Non-uniform point distributions are relatively easy for most methods, whereas point-wise noise preserves coarse structure but can remove surface details, especially for triangulation-based methods.Several semantic or geometric-primitive learning methods are exceptions under non-uniform sampling.
  • 6.1 The Remaining Challenges: Misalignment preserves overall shape scores under CD and F-score but reduces NCS and NFS, reflecting thickened or multilayered surfaces that are less pleasant perceptually.Methods using smoothness or modeling priors have an advantage under misalignment.
  • 6.1 The Remaining Challenges: Missing points remain unsolved, and most methods produce unsatisfactory results for misalignment and point outliers, with only a few exceptions such as SPSR.Missing regions commonly remain incomplete, while implicit methods may fill holes with envelopes that do not represent the true surface.
  • 6.2 Optimization-based, Learning-free Methods Versus Learning-based, Data-driven Ones: Optimization-based, learning-free methods generalize better across evaluation metrics on complex synthetic object surfaces, while learning-based methods show greater robustness to severe imperfections.This pattern also appears on real-scanned data.
  • 6 Main Results: Oriented surface normals improve reconstruction, and correct inward or outward orientation matters more than normal precision.Removing camera-pose information causes substantial performance drops because local normals may be wrongly oriented.

7 Conclusion

The paper reviews classical and deep learning surface-reconstruction methods through geometric priors and evaluates them on a benchmark containing practical scanning imperfections. Its empirical studies compare robustness and generalization and identify remaining challenges for future research.

  • 7 Conclusion: The paper organizes classical and deep learning methods by the geometric priors used to regularize surface reconstruction.
  • 7 Conclusion: A large-scale benchmark with synthetic and real-scanned data tests robustness and generalization under imperfections common in practical 3D scanning.
  • 7 Conclusion: The benchmark studies identify remaining challenges and are intended to guide future surface-reconstruction research.

Appendix A Technical Details for Computing the Algebraic Surface Complexity

The appendix defines algebraic surface complexity from local approximation errors and curvature, then groups benchmark object surfaces by the resulting complexity values.

  • Appendix A Technical Details for Computing the Algebraic Surface Complexity: Local polynomial patches approximate a smooth surface with an error bound determined by patch width and derivatives of order m+1.Patches are chosen to cover the surface without holes.
  • Appendix A Technical Details for Computing the Algebraic Surface Complexity: For fixed mesh vertex counts, comparing algebraic complexities amounts to comparing constants that depend on local surface curvatures.The benchmark uses m = 1 for piecewise-linear approximation.
  • Appendix A Technical Details for Computing the Algebraic Surface Complexity: Algebraic complexity is computed from the expected integral of squared normal curvatures over points on a surface.The curvature formulation uses principal, mean, and Gaussian curvatures.
  • Appendix A Technical Details for Computing the Algebraic Surface Complexity: The benchmark divides object surfaces into low-, middle-, and high-complexity groups containing 972, 486, and 162 instances, respectively.The groups follow an approximately 6:3:1 ratio.

Appendix B Details of Evaluation Metrics

The benchmark uses four metrics to compare reconstructed surface meshes from different methods, combining geometric, normal-consistency, and neural-feature perspectives.

  • Appendix B Details of Evaluation Metrics: Surface reconstructions are evaluated with Chamfer Distance, F-score, Normal Consistency Score, and Neural Feature Similarity.NFS is described as a newly proposed neural metric.

B.1 Popular Evaluation Metrics

The paper uses Chamfer Distance, F-score, and Normal Consistency Score as established metrics for comparing reconstructed and ground-truth surfaces represented as point sets.

  • Chamfer Distance measures symmetric distance between sampled reconstructed and ground-truth surface point sets.The evaluation samples 200k points for synthetic object surfaces and 1500k points for synthetic scene surfaces.
  • F-score combines point-set precision and recall through their harmonic mean, with threshold τ controlling sensitivity.The paper sets τ to 0.005 for synthetic object surfaces, 0.03 for synthetic scene surfaces, and 0.5 for real-scanned data.
  • Normal Consistency Score evaluates normal agreement while using normals from continuous triangular facets to reduce discrete sampling error.The evaluation uses the same point-count settings for synthetic object, scene, and real-scanned data as described for the other metrics.

B.2 The Proposed Neural Evaluation Metric

The paper proposes Neural Feature Similarity to compare reconstructed and reference surfaces in a learned feature space, addressing perceptual limitations of conventional geometric metrics.

  • Neural Feature Similarity compares surfaces through cosine similarity in a deep feature space intended to better reflect human-perceived shape differences.The authors motivate it because Chamfer Distance, F-score, and Normal Consistency Score may miss shape differences aligned with human perception.
  • A feature-extracting network maps point clouds to vectors and is trained so same-surface samples become invariant while different-surface samples remain distinct.The training uses a self-supervised objective over point-cloud pairs sampled from the same or different surfaces.
  • The final similarity is computed from aligned local surface patches rather than whole surfaces, averaging differences across the local patches.The feature network is a six-layer MLP-based auto-decoder with 256 channels per layer and Leaky ReLU activations.

Appendix E Experimental Results Without Data Pre-processing

Without input pre-processing, the reported comparisons retain similar relative quality to those obtained with the pre-processing pipeline.

  • Results with and without pre-processing have similar comparative qualities for synthetic object and scene surfaces.The comparisons are reported in Tables 13 and 14; real-scanned data are excluded because the scanners automatically pre-process point clouds.

Appendix F More Results for the Real-scanned Data

Additional real-scanned results indicate that reconstruction quality depends heavily on surface material, while the broader observations remain consistent with the paper’s earlier results.

  • Scanned point-cloud quality depends heavily on surface material in the additional real-scanned qualitative results.The passage states that other observations are consistent with those in Section 6.
  • The appendix tables compare methods using surface normals, methods without normals, and methods using normals only during learning on scene and real-scanned data.For scene surfaces and real-scanned data, results distinguish settings with and without ground-truth camera poses.
  • The appendix also reports quantitative comparisons for unprocessed synthetic object and scene point clouds, highlighting the best and second-best methods per column.Scene-surface results mark methods unable to produce reasonable outputs with a dash when limited generalization prevents reconstruction.
Loading 2205.02413v1…