Source-linked AI summary
Point2Mesh: A Self-Prior for Deformable Meshes
Rana Hanocka, Gal Metzer, Raja Giryes, Daniel Cohen-Or
TL;DR
Mesh reconstruction from point clouds is ill-posed and traditionally relies on manually designed priors. Point2Mesh learns a self-prior from one input cloud while deforming an initial mesh, and is demonstrated on non-ideal point clouds. Its authors identify runtime, memory, and initial-mesh quality as limitations.
Problem
Point-cloud mesh reconstruction is ill-posed, so methods need a prior that encodes expected properties of the reconstructed mesh.
Source: §1 · p. 1Method
Results
Point2Mesh is demonstrated on shapes and real scans with noise, unoriented or unknown normals, and missing regions, and compared with other leading techniques.
Source: §1 · p. 2Takeaways & Limitations
The self-prior models recurring and correlated shape structure, while being weak at modeling non-recurring noise and outliers.
Source: §6 · p. 11The method is expensive in time and memory, and its initial mesh may be imperfect, especially under non-ideal conditions.
Source: §6 · p. 11
Abstract
from arXiv · showhide
In this paper, we introduce Point2Mesh, a technique for reconstructing a surface mesh from an input point cloud. Instead of explicitly specifying a prior that encodes the expected shape properties, the prior is defined automatically using the input point cloud, which we refer to as a self-prior. The self-prior encapsulates reoccurring geometric repetitions from a single shape within the weights of a deep neural network. We optimize the network weights to deform an initial mesh to shrink-wrap a single input point cloud. This explicitly considers the entire reconstructed shape, since shared local kernels are calculated to fit the overall object. The convolutional kernels are optimized globally across the entire shape, which inherently encourages local-scale geometric self-similarity across the shape surface. We show that shrink-wrapping a point cloud with a self-prior converges to a desirable solution; compared to a prescribed smoothness prior, which often becomes trapped in undesirable local minima. While the performance of traditional reconstruction approaches degrades in non-ideal conditions that are often present in real world scanning, i.e., unoriented normals, noise and missing (low density) parts, Point2Mesh is robust to non-ideal conditions. We demonstrate the performance of Point2Mesh on a large variety of shapes with varying complexity.
1 INTRODUCTION
Point2Mesh addresses the ill-posed task of reconstructing meshes from point clouds by learning a CNN prior from the input shape itself. Shared kernels optimized across the shape encourage geometric repetition, and the method is demonstrated on point clouds with noise, unknown or unoriented normals, and missing regions.
1 INTRODUCTION
- The CNN prior is learned automatically from a single input point cloud during inference, without training data or pre-training.This avoids requiring labeled training pairs or advance knowledge of noise and partiality distributions.Source: §1 · pp. 1–2
- The method iteratively deforms an initial watertight mesh to fit the point cloud while preserving watertightness through vertex displacements.The initial mesh can be estimated with techniques that roughly approximate the input shape and have arbitrary genus.Source: §1 · p. 2
- The CNN prior is suited to natural shapes and can remove noise and outliers, while the cited Poisson reconstruction method assumes oriented normals.The network struggles to explain noisy and unnatural shapes; unoriented normals can be difficult to obtain in real scans.Source: §1 · p. 2
- Because the same local kernels are optimized across the whole shape, the CNN prior encourages local-scale geometric self-repetition.The network applies convolutions locally but reuses kernels over the entire shape.Source: §1 · p. 2
- Point2Mesh is demonstrated on shapes and real scans, including point clouds with noise, unknown or unoriented normals, and missing regions.The paper compares it with other leading techniques, a smoothness prior, and optimization without a network prior.Source: §1 · p. 2
2 RELATED WORK
Prior surface-reconstruction work includes point-cloud enhancement, implicit-surface methods, and iterative mesh deformation. Point2Mesh is situated among neural approaches and differs from prior methods that use hand-designed priors or lack global weight sharing.
2 RELATED WORK
- Point-cloud consolidation work enhances input data through denoising, smoothing, up-sampling, or related processing.Early reconstruction work also fitted implicit functions and generated meshes over their zero-sets.Source: §2 · p. 2
- Poisson reconstruction formulates surface recovery as a Poisson system, while other approaches use Fourier coefficients, wavelets, multiscale implicit functions, or compact RBFs.Poisson reconstruction and related methods process local regions and require perfectly oriented normals to guarantee watertightness.Source: §2 · p. 2
- Mesh-deformation methods iteratively fit an initial mesh to a point cloud, but their non-convex optimizations can reach local minima and use hand-crafted priors.Sharf et al. [2006] start with an interior spherical mesh and use smoothness priors and heuristics; Point2Mesh instead learns its prior from the input shape.Source: §2 · pp. 2–3
- The CNN reconstructs natural shapes better than shapes with added noise, shuffled vertices, or uniform noise.Figure 3 plots reconstruction error against optimization iterations for the four input types.Source: Figure 3 · p. 3
- Deep Image Prior and related work motivate neural self-priors; Point2Mesh reports weight sharing as key to its self-prior advantage.The authors connect shared weights with learning repeating structures in the processed mesh.Source: Neural Self-Priors · p. 3
- Other neural reconstruction work includes differential renderers, scan-to-mesh reconstruction, and networks that deform mesh cuboids for shape synthesis.The cited works reconstruct geometry from images or scans, or address shape synthesis.Source: Geometric Deep Learning and… · p. 3
- Deep Geometric Prior trains separate MLPs for local charts and uses Poisson reconstruction afterward, unlike Point2Mesh's globally shared weights.The authors characterize Deep Geometric Prior as closer to point up-sampling approaches.Source: Geometric Deep Learning and… · p. 3
3 POINT2MESH
Point2Mesh optimizes a CNN self-prior to deform an initial mesh toward a point cloud, using shared weights to exploit recurring geometry. Its mesh representation, point-cloud distance, and beam-gap loss support this iterative reconstruction.
3 POINT2MESH
- The CNN self-prior iteratively deforms an initial mesh to shrink-wrap the point cloud, using recurring geometry to suppress noise and outliers.Reconstruction proceeds coarse-to-fine, with the mesh refined by subdivision across levels.Source: §3 · pp. 3–4
- The self-prior smooths noise while preserving recurring ridges and completes missing regions consistently with the global shape.The paper illustrates this behavior on an ankylosaurus and a missing half dome.Source: §3 · pp. 3–4, Figure 7 · p. 5
3.1 Explicit Mesh Representation
- Network-predicted edge displacements are averaged into per-vertex displacements, while changing vertex positions preserves the mesh’s connectivity and genus.Differential positions are measured relative to the input mesh vertices.Source: §3.1 · pp. 4–5, §3 · p. 4
3.2 Mesh to Point Cloud Distance
- A differentiable mesh sampler supplies points for a bidirectional Chamfer distance between the reconstructed surface and input point cloud.The sampling mechanism allows gradients to back-propagate through the network weights.Source: §3.2 · p. 5
3.3 Beam-Gap Loss
- Because Chamfer-only optimization can stall before entering narrow, deep cavities, beam-gap adds an approximate surface-distance loss using rays cast from mesh points.The beam intersection is approximated from the discrete point cloud, assuming reasonable sampling density.Source: §3.3 · p. 5
- Mutual k-nearest-neighbor matches identify well-fitted mesh points, which receive no beam-gap penalty.When multiple beam intersections are detected, the closest is selected.Source: §3.3 · pp. 5–6
4 IMPLEMENTATION DETAILS
Point2Mesh uses a coarse-to-fine mesh optimization, with initial meshes selected according to the expected genus and mesh parts used to manage memory. Runtime varies with mesh resolution and shape complexity.
4.1 Iterative Coarse-to-Fine Optimization
- Mesh subdivision enables finer geometric detail and can help optimization escape local minima, while progressive refinement avoids starting with an overly complex mesh.The mesh begins with roughly a couple thousand faces, is deformed toward the target, then subdivided and smoothed as optimization proceeds.Source: §4.1 · p. 6
4.2 Part-Based
- PartMesh divides a reconstructed mesh into sub-meshes that can be processed separately, with accumulated gradients used for back-propagation across the full mesh.Overlapping vertices are averaged when recovering their final positions.Source: §4.2 · p. 7
4.3 Initial Mesh Approximation
- For genus-0 shapes, the convex hull initializes the mesh; otherwise, a coarse octree manifold approximation of an alpha shape or Poisson reconstruction provides the initial mesh.The coarse approximation distances initialization from potentially incorrect reconstructions and closes topological holes in low-density regions.Source: §4.3 · p. 7, Figure 9 · p. 7
4.4 Run-time
- A non-optimized implementation takes 0.23 seconds per iteration for 2,000 faces and 0.59 seconds for 6,000 faces; total runtimes range from about 3 minutes to 3 hours or more.At 40,000 faces, an iteration takes 4.7 seconds using 8 parts; extremely high-resolution results can take over 14 hours.Source: §4.4 · pp. 7–8
5 EXPERIMENTS
The experiments examine real scans, imperfect normals, denoising, shape completion, and the role of the network self-prior. They report that the network helps avoid undesirable local minima and show reconstructions on scans with holes and incorrectly oriented normals.
5 EXPERIMENTS
- The experiments include real scans and point sets sampled from ground-truth meshes, with tests involving missing regions, noise, and challenging cavities.Mesh datasets include Thingi10k, COSEG, TOSCA high-resolution, and the Surface Reconstruction Benchmark.Source: §5 · p. 8
5.1 Real Scans
- On real scans, Point2Mesh produces plausible reconstructions despite large input holes and avoids artifacts seen with smooth reconstruction when normals are incorrectly oriented.The comparisons include NextEngine scans and scans from Choi et al. [2016]; the normal-orientation comparison is shown in Figure 12.Source: NextEngine 3D scans · p. 9, Figure 12 · p. 9, Figure 13 · p. 9
5.2 Accuracy Metric
- The F-score is the harmonic mean of precision and recall at threshold τ, but a high F-score does not necessarily imply better visual quality.Precision measures reconstructed-to-ground-truth distance, while recall measures ground-truth-to-reconstruction distance.Source: §5.2 · p. 9
5.3 Comparisons
- Denoising and completion evaluations use two ground-truth meshes each, with five sampled variants per shape; completion evaluates recall on the missing surface and has a correct-genus initialization advantage.The denoising and completion comparison scopes are shown in Tables 1 and 2, respectively.Source: Denoising · pp. 9–10, Table 1 · p. 9, Low Density Completion · p. 10
5.4 Network is a Prior
- The network self-prior facilitates desirable convergence: the reconstruction objective and coarse-to-fine framework alone are insufficient, while direct optimization is prone to local minima.The authors attribute this advantage to the network's weight sharing and self-similarity capabilities.Source: §5.4 · pp. 10–11, Figure 16 · p. 10
- Among 50 manually classified centaur-shape configurations, clean results were associated with more bottleneck features and learned parameters, while increased pooling was more desirable for completion.Narrower convolutional width produced noisy reconstructions; the number of residual connections did not seem to affect performance.Source: §5.4 · p. 11, Figure 17 · p. 11
6 DISCUSSION
Point2Mesh uses a CNN self-prior to regress irregular-mesh geometry, leveraging recurring structure in natural shapes. Its mesh-deformation strategy preserves genus and provides control, while the method remains costly and depends on the quality of its initial mesh.
6 DISCUSSION
- The network performs regression on an irregular mesh, optimizing its geometry for surface reconstruction.Deforming a given mesh preserves its genus and provides control that the authors say competing methods lack.Source: §6 · p. 11
- CNN weight sharing models recurring, correlated geometry across natural shapes, while remaining weak at representing non-recurring noise and outliers.The authors connect this self-similarity to modeling natural shapes and completing missing parts or removing noise and outliers.Source: §6 · p. 11
- The authors report good performance on noisy point clouds and those with missing regions in denoising and completion tasks.Source: §6 · p. 11
- The method is expensive in time and memory compared with state-of-the-art surface reconstruction techniques.For shapes with genus greater than zero, it also relies on available initial-mesh techniques, which can produce imperfect results, especially under non-ideal conditions.Source: §6 · p. 11
- Future work could detect point-cloud genus without training data by learning a self-prior during input processing.Another proposed direction is to optimize one deformable mesh to shrink-wrap multiple objects and obtain compatible triangulations.Source: §6 · p. 11