PaperlayerView exact revision on arXiv

Source-linked AI summary

Shape As Points: A Differentiable Poisson Solver

Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, Andreas Geiger

arXiv:2106.03452v2cs.CVcs.GR

TL;DR

Existing shape representations trade efficiency, topology flexibility, surface information, and inference speed. Shape-As-Points introduces a differentiable Poisson point-to-mesh bridge, achieving efficient watertight reconstruction and state-of-the-art results from noisy unoriented point clouds, while large unobserved regions remain incomplete in single-object reconstruction.

  • Problem

    Point clouds are lightweight but do not directly encode surfaces, meshes are usually topology-restricted, and neural implicit representations require slow inference and careful initialization.

  • Method

    Shape-As-Points represents shapes with oriented point samples and uses a GPU-accelerated differentiable Poisson solver to compute an indicator function and backpropagate losses to the point cloud.

  • Results

    Shape-As-Points demonstrates state-of-the-art reconstruction from noisy unoriented point clouds at a significantly reduced computational budget, with indicator computation taking 12 ms versus 330 ms for a neural implicit network at 128^3 resolution.

  • Takeaways & Limitations

    The representation is lightweight, interpretable, topology-agnostic, and produces high-quality watertight meshes efficiently in optimization-based and learning-based reconstruction settings.

  • Takeaways & Limitations

    In single-object reconstruction, the method cannot complete large unobserved regions, such as the unobserved bottom of a person’s feet.

Abstract

from arXiv · show

In recent years, neural implicit representations gained popularity in 3D reconstruction due to their expressiveness and flexibility. However, the implicit nature of neural implicit representations results in slow inference time and requires careful initialization. In this paper, we revisit the classic yet ubiquitous point cloud representation and introduce a differentiable point-to-mesh layer using a differentiable formulation of Poisson Surface Reconstruction (PSR) that allows for a GPU-accelerated fast solution of the indicator function given an oriented point cloud. The differentiable PSR layer allows us to efficiently and differentiably bridge the explicit 3D point representation with the 3D mesh via the implicit indicator field, enabling end-to-end optimization of surface reconstruction metrics such as Chamfer distance. This duality between points and meshes hence allows us to represent shapes as oriented point clouds, which are explicit, lightweight and expressive. Compared to neural implicit representations, our Shape-As-Points (SAP) model is more interpretable, lightweight, and accelerates inference time by one order of magnitude. Compared to other explicit representations such as points, patches, and meshes, SAP produces topology-agnostic, watertight manifold surfaces. We demonstrate the effectiveness of SAP on the task of surface reconstruction from unoriented point clouds and learning-based reconstruction.

1 Introduction

Shape-As-Points connects explicit oriented point clouds to watertight meshes through a differentiable Poisson solver. The representation targets lightweight storage, fast inference, topology-agnostic geometry, and reconstruction from unoriented point clouds.

  • Neural implicit representations are flexible and differentiable but suffer from slow inference and difficult initialization, motivating an explicit alternative.
  • DPSR solves an indicator function from an oriented point cloud in a few milliseconds and backpropagates losses from the mesh or indicator grid.
  • SAP stores oriented surface samples instead of volumetric quantities or large neural networks, while producing watertight, self-intersection-free, topology-agnostic meshes.
  • 12 ms at 128^3 resolution versus 330 ms for a neural implicit network demonstrates SAP's lower geometry-extraction time.The comparison uses the same 128^3 resolution.
  • SAP is presented as interpretable, lightweight, and capable of high-quality watertight meshes at low inference times, with a versatile differentiable Poisson solver.
  • The authors evaluate SAP's inference time, initialization sensitivity, topology-agnostic capacity, and reconstruction from noisy unoriented point clouds.

2 Related Work

Prior work spans explicit point, mesh, and patch representations, implicit voxel or neural functions, classical point-cloud reconstruction, and learning-based priors. These approaches differ in topology handling, surface completion, and robustness to noisy inputs.

  • Explicit representations directly parameterize geometry as point clouds, meshes, or surface patches and are generally lightweight.
  • Implicit representations encode shapes as level sets over voxel grids or as neural networks, supporting object and scene geometry.
  • Classical point-cloud reconstruction methods include convex hulls, alpha shapes, ball pivoting, and Poisson Surface Reconstruction.
  • Learning-based methods use local, object-level, or scene-level priors, but local priors can struggle when large surfaces must be completed.

3 Method

The method uses a differentiable Poisson solver to convert oriented point clouds into indicator fields and watertight meshes, supporting both optimization-based and learning-based reconstruction. Spectral computation and differentiable gradients connect point, field, and mesh representations for efficient reconstruction.

  • 3 Method: The differentiable Poisson solver supports both optimization-based and learning-based surface estimation.It bridges oriented point clouds with indicator functions and meshes.
  • 3.1 Differentiable Poisson Solver: Oriented point normals are rasterized onto a uniform voxel grid, where discretized differential operators form the Poisson system for the indicator function.The system uses the divergence and Laplacian operators under boundary and scale constraints.
  • 3.1 Differentiable Poisson Solver: Spectral methods solve the discretized PDEs on uniform grids using GPU-supported FFT operations instead of irregular finite-element structures.A Gaussian spectral smoothing kernel mitigates ringing from rasterized point normals.
  • 3.2 SAP for Optimization-based 3D Reconstruction: In optimization-based reconstruction, the solver produces an indicator grid and Marching Cubes converts it into a mesh whose Chamfer loss backpropagates to the source oriented points.Gradients through the mesh extraction are approximated using the inverse surface normal.
  • 3.2 SAP for Optimization-based 3D Reconstruction: The optimization process resamples points from the largest mesh component every 200 iterations and uses coarse-to-fine grid resolutions from 32^3 to 256^3.Resampling removes drifting outliers and promotes a more uniform point distribution.
  • 3.3 SAP for Learning-based 3D Reconstruction: In learning-based reconstruction, a network encodes unoriented point coordinates, predicts offsets and normals, and produces a clean oriented point cloud for Poisson-based mesh generation.Training uses ground-truth indicator grids derived from watertight meshes.

4 Experiments

The experiments evaluate SAP for optimization-based and learning-based reconstruction from unoriented point clouds across synthetic, scanned, noisy, and outlier-contaminated inputs. SAP provides robust reconstruction while maintaining efficient inference and benefiting from differentiable indicator-grid supervision.

  • Experimental setup: Experiments cover single-object reconstruction on Thingi10K, SRB, and D-FAUST, plus learning-based reconstruction across 13 ShapeNet classes.The optimization setting uses five objects per dataset; ShapeNet experiments follow established train, validation, and test splits.
  • Optimization-based reconstruction: SAP achieves superior performance to classical and network-based methods across challenging datasets with complex geometry, thin structures, noise, and incomplete observations.The authors report robust behavior where some baselines fail completely.
  • Optimization-based reconstruction: SAP avoids free-space artifacts and topology-dependent initialization issues affecting IGR and Point2Mesh in unoriented point-cloud reconstruction.IGR can create meshes in free space, while Point2Mesh requires an initial mesh whose topology cannot change and may rely on inaccurate SPSR initialization.
  • Learning-based reconstruction: SAP achieves similar or better results than baselines on all three ShapeNet metrics, while DPSR supervision improves reconstructions across all tested noise and outlier settings.The DPSR loss directly supervises the indicator grid that implicitly determines the surface through the Poisson equation.
  • Runtime and ablations: SAP evaluates the indicator grid more than 20× faster than ConvONet and is roughly 5× and 8× faster overall at 128^3 and 256^3 resolutions.The runtime comparison uses GPU-accelerated methods and separates encoding, grid evaluation, and Marching Cubes costs.
  • Runtime and ablations: Increasing offsets improves performance, while the 3D encoder performs best in the most challenging ShapeNet setup, possibly because it aligns with the 3D indicator grid.Additional offsets densify points near the surface and can preserve geometric detail better.

5 Conclusion

The paper concludes that Shape-As-Points is a lightweight and interpretable representation that efficiently produces watertight meshes for unoriented point-cloud reconstruction. Its current scope is limited by cubic memory growth with indicator-grid resolution, motivating sliding-window or space-adaptive extensions.

  • SAP is demonstrated in both optimization-based and learning-based surface reconstruction from unoriented point clouds.
  • Cubic memory requirements with indicator-grid resolution currently limit SAP to small scenes.The authors suggest sliding-window processing and space-adaptive structures such as octrees for larger scenes.

Supplementary Material for Shape As Points: A Differentiable Poisson Solver

The supplementary derivation rasterizes oriented point normals onto a voxel grid using trilinear interpolation. Each grid vertex aggregates contributions from neighboring points through their interpolation weights.

  • Point rasterization: Point normals are scattered to voxel-grid vertices using trilinear interpolation weights.The grid is defined by its origin and voxel size, and each point contributes according to nearby vertices.
  • Point rasterization: For each point, neighboring voxel-grid indices are computed from its location and grid geometry.The formulation uses point coordinates, normals, grid origin, and voxel size.
  • Point rasterization: Each grid value is obtained by summing contributions from points in the vertex’s neighborhood.N_j denotes the set of point indices neighboring vertex j.

A.2 Spectral Methods for Solving PSR

The solver uses spectral methods to transform the rasterized point-normal field into the Poisson solution. It computes differential operators in frequency space, smooths rasterization artifacts, then transforms and normalizes the indicator field.

  • Spectral solution: The Poisson PDEs are solved with multidimensional Fourier and inverse Fourier transforms.Spatial coordinates are mapped to frequency variables, where derivatives can be computed analytically.
  • Spectral solution: The divergence of the rasterized point normals and the Laplacian are represented in the spectral domain.These operators form the ingredients for the unnormalized Poisson solution.
  • Indicator field: The unnormalized indicator solution is computed spectrally and converted to the physical domain with an inverse Fourier transform.
  • Indicator field: A Gaussian spectral smoothing kernel mitigates ringing caused by Gibbs effects when rasterizing point normals.The kernel bandwidth depends on σ and the grid resolution r.
  • Indicator field: The indicator field is normalized so point locations have value zero and interior and exterior regions have values ±0.5.

B Implementation Details

The appendix specifies implementation settings for optimization-based and learning-based reconstruction, including baselines, resolutions, iteration schedules, and training configurations.

  • Implementation settings: The experiments cover optimization-based and learning-based reconstruction, with implementation details provided for baselines and SAP.The optimization-based setting uses official implementations of IGR and Point2Mesh; the learning-based setting trains AtlasNet, ConvONet, PSGN, 3D-R2N2, SAP, and an ablated variant.
  • Optimization-based reconstruction: IGR is optimized for 15000 iterations per object, while Point2Mesh uses 6000 iterations and an initialized mesh based on the input geometry.For genus-zero objects, Point2Mesh uses the convex hull of the input point cloud as its initial mesh.
  • Optimization-based reconstruction: SAP follows a coarse-to-fine and resampling strategy while increasing Gaussian smoothing across grid resolutions.The listed settings use σ = 2 at 32^3 and 64^3 resolutions and σ = 3 at 128^3; the 256^3 setting uses σ = 3 for detailed objects.
  • Learning-based reconstruction: Table 5 evaluates Chamfer distance on the ShapeNet test set with 3K input points and noise level 0.005 to compare training progress.The comparison concerns SAP's geometric initialization and ConvONet's convergence.
  • Learning-based reconstruction: Learning-based baselines train for at least 300000 iterations with Adam at learning rate 10^-4, whereas SAP uses 300000 iterations and learning rate 5 × 10^-4.SAP training uses batch size 32 and approximately two days on two GTX 1080Ti GPUs.

C Discussions on “Easy Initialization” Property of SAP

SAP can start from simple or coarse geometry: coarse initialization converges faster in optimization, while direct geometric initialization accelerates learning convergence.

  • Optimization-based setting: Starting SAP from a coarse shape converges faster than starting from a sphere, while both achieve equivalent accuracy.The comparison reports no better local minimum for either initialization.
  • Learning-based setting: In learning-based reconstruction, the input point cloud serves as geometric initialization because SAP predicts offset and normal fields for it.This contrasts with neural-implicit weight initialization, which is analytically derived only for simple shapes such as spheres.
  • Learning-based setting: SAP converges much faster than ConvONet on ShapeNet test data with 3K input points and noise level 0.005.Table 5 reports Chamfer distance at different training iterations to evaluate this convergence difference.
  • Resampling strategy: Without resampling, SAP still reconstructs reasonable shapes, but unevenly distributed optimized points produce noisier surfaces.The resampling strategy regularizes the point cloud toward a uniform distribution, improving surface reconstruction.

D.3 Analysis on the Gaussian Smoothing Parameter σ

The Gaussian smoothing parameter balances detail preservation against noise suppression, while spectral-domain filtering provides an efficient regularization mechanism.

  • Role of Gaussian smoothing: The Gaussian acts as a smoothness regularizer for the solved implicit function, with σ = 0 corresponding to no Gaussian smoothing.The ablation studies the parameter at 256^3 resolution.
  • Spectral implementation: Spectral Gaussian filtering is efficient because physical-domain convolution becomes a spectral-domain dot product, reducing complexity from O(N^2) to O(N log N).Here n is the regular-grid resolution and N = n^3.
  • Effect of σ: Lower σ preserves more details but is more prone to noise, whereas higher σ produces smoother shapes while losing details.The practical choice of σ can follow the noise level of the target point cloud.

D.4 Preliminary Results on Multi-view Reconstruction with Differentiable Rendering

SAP's differentiable point-to-mesh layer can be combined with differentiable rendering for multi-view reconstruction, using image-color and silhouette supervision.

  • Method: The multi-view extension rasterizes the reconstructed mesh, predicts pixel colors with an MLP, and applies RGB and silhouette losses.PyTorch3D provides mesh rasterization and differentiable silhouette rendering.
  • Results: Preliminary results reconstruct detailed geometry from only 2D supervision on one synthetic object with 20 images and two real-world objects with about 50 images.The real-world examples come from the DTU dataset.
  • Efficiency: Compared with neural implicit methods, SAP outputs an explicit mesh each iteration and avoids expensive ray marching for surface finding.The cited comparison presents this as the source of faster inference.
  • Noise and outliers: During noisy-input handling, SAP predicts seven offsets per input point, densifying the points and placing them roughly on the surface.The passage connects this surface placement with enhanced reconstruction quality.
  • Noise and outliers: For outliers, SAP moves many points near the surface while assigning small normal magnitudes to some points that remain outliers.The arrow lengths visualize predicted normal magnitudes.

E.2 Additional Results for Reconstruction from Noisy Point Clouds

Additional ShapeNet results combine per-category evaluation across 13 classes with a focused noisy-point-cloud comparison on lamp. SAP is also qualitatively compared with reconstruction baselines.

  • Quantitative evaluation: Table 7 compares baselines and SAP variants per category across all 13 ShapeNet classes.All methods are trained on the full 13-class subset.
  • Comparison to Points2Surf: The Points2Surf comparison uses ShapeNet’s lamp class with noise level 0.005, covering roughly 2,000 training and 450 testing objects.The comparison is restricted to lamp because Points2Surf requires very long training and inference time.
  • Comparison to Points2Surf: 3 orders of magnitude faster inference: SAP outperforms Points2Surf on all metrics after 30K training iterations.SAP uses two GTX 1080Ti GPUs for less than four hours, while Points2Surf takes over 50 hours on eight GPUs to reach 150 epochs.
  • Qualitative comparison: Figure 10 reports that SAP estimates normals robustly under noise, ignores outliers selectively, reconstructs fine details, and achieves higher quality than ConvONet with less inference time.The qualitative comparison includes SPSR, 3D-R2N2, AtlasNet, and ConvONet.
Loading 2106.03452v2…