PaperlayerView exact revision on arXiv

Source-linked AI summary

POCO: Point Convolution for Surface Reconstruction

Alexandre Boulch, Renaud Marlet

arXiv:2201.01831v2cs.CVcs.CGcs.LG

TL;DR

Implicit reconstruction methods face scalability and surface-localization limitations when representing whole objects or scenes with coarse or global features. POCO computes latent vectors at input points with point convolutions and learns neighborhood interpolation weights. The paper reports state-of-the-art performance for object and scene reconstruction, while noting limits when large shape regions are missing and challenges with orientation under high density.

  • Problem

    Implicit reconstruction methods struggle to scale to arbitrary scenes and may place latent information away from surface points, where difficult occupancy decisions occur.

  • Method

    POCO computes implicit-function features at input points using point convolutions and learns interpolation weights for query-relative neighboring features.

  • Results

    POCO reaches the state of the art for object and scene reconstruction, with or without oriented normals, and produces finer details than competing methods.

  • Takeaways & Limitations

    Point-based feature placement and learned interpolation support reconstruction across both single objects and whole scenes while preserving fine geometric detail.

  • Takeaways & Limitations

    POCO cannot complete shapes when large parts are missing, and high point-cloud density can cause orientation failures and artifacts.

Abstract

from arXiv · show

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes. The code is available at https://github.com/valeoai/POCO.

1. Introduction

POCO addresses scalability and surface-localization limitations in implicit reconstruction by using point convolutions and learned interpolation tied directly to input points. It reports finer details and strong performance on both object and scene datasets.

  • Limitations of prior methods: Existing implicit methods often struggle to scale from individual objects to arbitrary scenes because they encode shape information in limited latent representations.Fully connected architectures and a few latent vectors cannot readily capture detailed geometry across variable-sized scenes.
  • Limitations of prior methods: Voxel-based convolutional methods lose direct connections to surface samples and distribute features broadly instead of concentrating them near difficult surface decisions.Their voxel vertices may be far from the input point cloud, while occupancy information is more or less uniformly distributed in space.
  • POCO contributions: POCO attaches implicit-function features to input points, preserving point positions and concentrating learned information near the surface.This avoids abstracting away the input geometry too early.
  • POCO contributions: POCO uses point convolution to compute features, providing natural coverage and scalability to scenes of arbitrary size.The approach relies on a general point-convolution backbone rather than a task-specific architecture.
  • POCO contributions: The method learns query-relative interpolation weights rather than relying on hand-designed averaging of global features.The interpolation is applied to relative features associated with neighboring input points.
  • Reported outcomes: POCO outperforms other methods on object and scene datasets, producing finer details and supporting efficient reconstruction of large or dense inputs.The contributions also include a test-time augmentation designed for high-density or large point clouds.

2. Related work

Prior reconstruction methods trade off scalability, resolution, and direct use of surface samples. POCO instead uses point convolutions and point-attached features to retain surface information while supporting interpolation and larger scenes.

  • Existing representations: Voxel representations incur cubic memory costs, while point and mesh representations avoid grid discretization but remain limited for surface and volume reconstruction.Implicit representations avoid discretization until mesh extraction, but fully connected architectures generally struggle with arbitrary scenes.
  • Patch and local methods: Patch-based methods such as LIG optimize grid latent vectors from scratch for each input and require costly decoding, while Points2Surf combines local and globally sampled points through MLPs.LIG also requires oriented normals; Points2Surf uses a larger global component for sign information.
  • POCO: POCO computes latent features directly on input points with point convolution, then interpolates occupancy decisions from nearest neighbors using learned weights.This avoids voxel discretization and attaches information near the observed surface.
  • Grid-based implicit methods: Grid-based convolutional methods improve scalability but lose the direct connection to input surface points and distribute latent information uniformly in space.Their grid vertices may be far from the sampled surface, weakening surface-focused representation.
  • POCO: Compared with related convolutional approaches, POCO avoids voxel grids, learns interpolation, and uses point-attached features rather than adaptive or regular voxel features.AdaConv uses an adaptive voxel grid, while IF-Net and NDF use multi-scale voxel encodings with basic interpolation.

3. Our method

POCO learns a continuous occupancy function by encoding input points, forming query-relative neighbor features, weighting them, and decoding their aggregate into occupancy probabilities.

  • Goal: Given surface points P, POCO learns a continuous function ω: R3 → [0, 1] that predicts occupancy at arbitrary query points.The surface is extracted as an isosurface of the learned implicit function.
  • Encoding: A point-convolution encoder maps every input point p ∈ P to a latent vector z_p, optionally using augmented surface normals.The experiments use FKAConv with latent dimension n = 32.
  • Query-relative features: For query point q, the method selects neighboring input points, augments each latent vector with q − p, and re-encodes the result with an MLP.The neighborhood contains k = 64 nearest neighbors in the experiments, and relative latent vectors have size n = 32.
  • Weighting and interpolation: An attention mechanism infers significance weights for relative neighbor embeddings, which are combined into the query feature z_q by weighted summation.The weights are normalized over the query neighborhood before interpolation.
  • Decoding: A linear decoder maps z_q to two occupancy logits, which softmax converts into occupied and empty-space probabilities.The network is trained with cross-entropy loss; binary cross-entropy gives identical results in the reported experiments.

4. Refinements

POCO includes test-time strategies for dense inputs, large point clouds, and scale changes. These refinements average latent encodings across subsamples, process large inputs in chunks, and normalize test scale.

  • Adapting to high density: At high test-point density, the fixed training receptive field may lack enough global context, while downsampling can remove details.Oriented normals can provide the missing side-of-surface information without broadening the receptive field.
  • Adapting to high density: Test-time augmentation averages latent vectors from multiple subsamples instead of aggregating final query predictions.This avoids multiplying query-time decoding by the number of subsamples; experiments use Nview = 10 for dense-input augmentation.
  • Evaluation setting: Figure 6 compares methods trained and tested on 3k noisy ShapeNet points.The comparison concerns reconstruction under noisy, medium-density inputs.
  • Adapting to large size: For large point clouds, POCO processes overlapping chunks or subsamples of at most Ntest = 100k points because GPU memory limits backbone processing.The method remains convolutional and can therefore accept point clouds of arbitrary size in principle.
  • Scene scaling: At inference, POCO rescales point clouds so their average nearest-neighbor distance matches the training scale.This addresses sensitivity of point-based backbones to scale and density variation.

5. Experiments

POCO is evaluated across object and scene datasets, varying density, noise, normals, and generalization settings. It generally achieves finer-detail reconstructions and strong cross-domain performance, while remaining applicable to whole scenes.

  • 5.1. Datasets, baselines and metrics: Experiments cover objects and scenes across different point densities, noise settings, normal availability, and generalization configurations.Evaluations use ShapeNet, Synthetic Rooms, ABC, Famous, Thingi10k, SceneNet, and MatterPort3D, with baselines evaluated in their applicable settings.
  • 5.1. Datasets, baselines and metrics: POCO uses state-of-the-art baselines, including methods with oriented-normal requirements such as SPR.The default configuration uses FKAConv, 32-dimensional features, 64 neighbors, 64 interpolation heads, and no normals or test-time augmentation.
  • 5.2. Alternative and ablation studies: FKAConv is more efficient than the PointNet-based alternative, while relative-feature interpolation and multi-head attention improve the selected configuration.The chosen k = 64 setting is faster and has better IoU than k = 128, despite slightly worse CD and NC.
  • 5.3. Reconstruction: POCO outperforms competing methods on ShapeNet and Synthetic Rooms, producing finer details and reconstructing thin surfaces more readily than ConvONet.The ShapeNet comparison reports superiority on all metrics with a significant margin; Synthetic Rooms results also show finer details.
  • 5.3. Reconstruction: POCO generalizes from ShapeNet or ABC training to scenes and other shape domains, outperforming Points2Surf in most settings but losing slightly on very noisy or dense inputs.On SceneNet it captures finer details and does not erase thin objects; on Famous and Thingi10k, Points2Surf wins only under very noisy or dense inputs with a small margin.
  • 5.4. Discussion and limitations: POCO reaches state-of-the-art performance for object and scene reconstruction with or without oriented normals, but cannot complete shapes when large regions are missing.Its generalization extends to shapes and scenes substantially different from the training set, while missing-part completion remains constrained.

A. Implementation details

The implementation uses a standard point-convolution backbone and a compact occupancy decoder. Training reuses existing dataset samplings and follows fixed Adam optimization settings.

  • Framework and hardware: The released implementation uses PyTorch and was evaluated on a single NVIDIA RTX 2080 Ti GPU with 11GB memory.
  • Backbone: FKAConv is the convolutional backbone, with latent vectors of size 32 matching the decoder’s linear-layer width.The final decoder layer outputs occupancy, while ConvONet and LIG also use latent vectors of size 32.
  • Architecture: The architecture uses 35-dimensional relative-encoder inputs, 32-unit linear layers, 64 significance-weight outputs, and two final occupancy classes.The 35 inputs combine 32-dimensional latent vectors with three-dimensional point coordinates; activations are ReLUs and batch normalization is confined to the backbone.
  • Training data: POCO does not include point sampling during training; it reuses samplings from ConvONet and Points2Surf, while other datasets are used only for inference.
  • Training settings: Training uses Adam with learning rate 10^-3, batch sizes 16 for 3k points and 8 for 10k points, over 600k iterations.

B. Meshing for occupancy

POCO extracts occupancy surfaces with Marching cubes variants designed to balance query efficiency and thin-detail preservation. Region growing improves scene quality but increases runtime relative to refinement-based meshing.

  • Mesh generation: Marching cubes extracts implicit surfaces by evaluating occupancy on a regular 3D grid.
  • MC-refin: MC-refin starts from a coarse grid and subdivides cubes whose corners disagree on occupancy.Each selected cube is subdivided into eight subvoxels, concentrating queries near predicted surface transitions.
  • MC-regro: MC-regro starts from fine resolution and grows queried regions from input-point seeds near empty/full transitions, reducing unnecessary queries elsewhere.Its design targets locally complex surfaces and thin volumes that refinement heuristics might miss.
  • Vertex placement: Dichotomic edge searches reduce discretization effects caused by sharp occupancy transitions during vertex placement.Linear interpolation can place vertices near segment midpoints when opposite endpoints have occupancy values close to 0 and 1.
  • Comparison: MC-regro is generally qualitatively better than MC-refin on scenes but similar on objects, while remaining slower because it captures thin details.ShapeNet accuracy is similar for both variants, likely because thin details have little effect on the reported metrics.
  • Runtime: For 50k-point real-world models, latent-vector inference takes 38 seconds and full single-thread reconstruction takes 4 minutes 27 seconds.

C. Running times

POCO’s encoding is fast, but decoding occupancy queries for Marching cubes dominates runtime, especially because nearest-neighbor computation is not optimized. Its receptive field varies with scene geometry and reached thousands of points in a SceneNet analysis.

  • Running-time breakdown: Decoding dominates POCO’s runtime because Marching cubes requires many occupancy queries and nearest-neighbor searches.Nearest-neighbor computation currently requires GPU–CPU communication and is identified as the bottleneck.
  • Comparisons: POCO remains significantly faster than Points2Surf even with test-time augmentation over subsampled point clouds.With Nview = 10, feature-level augmentation adds only about 5% relative to Nview = 1.
  • Comparisons: On ShapeNet with 3k points, POCO ranks second in speed behind ConvONet, while LIG becomes faster on denser scenes.POCO’s kd-tree-based kNN search slows as density increases.
  • Receptive field: The receptive field depends on scene geometry and can increase with network depth, but mutual-neighbor groups may prevent expansion.
  • Receptive field: 29k points receive non-zero gradients in the measured SceneNet receptive field, or 16k when requiring gradient norm greater than 10^-7.

E.1. Choice of compared methods and datasets

The evaluation compares methods across heterogeneous datasets and configurations, while explicitly defining the surface and volume metrics used for reconstruction quality. Existing literature is difficult to compare directly because methods are often tested under different conditions and availability constraints.

  • Compared methods and datasets: Published reconstruction methods are difficult to compare because datasets, point densities, noise, normals, task scope, and evaluation coverage vary widely.Code, pretrained models, scripts, and parameters are also inconsistently available.
  • Compared methods and datasets: The study compares selected cited or recent methods in their reported settings rather than imposing a single configuration.This also tests POCO across varied configurations.
  • Compared methods and datasets: The experiments use dataset-specific normal and noise settings to enable comparisons with methods such as LIG, SA-ConvONet, and ConvONet.
  • Metrics: Surface metrics approximate distances between continuous surfaces using nearest sampled points on each surface.The evaluation samples points from both reconstructed and ground-truth surfaces.
  • Metrics: Chamfer distance averages bidirectional nearest-point distances using the L1 norm, and reported CD values are scaled by 10^2.
  • Metrics: IoU is a volume metric defined as TP divided by TP + FP + FN, whereas F-Score combines precision and recall at threshold t = 0.01.

E.3. More qualitative results

Qualitative comparisons show that POCO preserves sharper details across object and scene reconstructions, remains robust at low point densities, and adapts across shape domains without retraining.

  • ShapeNet: POCO reconstructs sharper and more robust surfaces than LIG across ShapeNet input densities.LIG uses part size 0.20 for 512 and 2048 points, and 0.10 for 8192 points.
  • SceneNet: On SceneNet, POCO is more robust at low point densities and recovers more details at high point densities than SPR and LIG.
  • Generalization: POCO adapts out of the box to new shape domains without retraining, especially when normals are provided.SPR performs well only on high-density point clouds.
  • Synthetic Rooms: On Synthetic Rooms, POCO produces more and sharper details than ConvONet and better handles thin surfaces such as bookcase shelves and a coffee table.

E.4. More quantitative results

Classwise ShapeNet results show POCO outperforming PointConv, ONet, and ConvONet across categories, particularly those with complex geometric details. The accompanying materials document the compared methods, datasets, and implementation dependencies.

  • Classwise ShapeNet results: POCO largely outperforms PointConv, ONet, and ConvONet across ShapeNet categories.The comparison uses models trained on 3k noisy points.
  • Classwise ShapeNet results: The largest advantages occur on detail-rich classes including lamp, rifle, and vessel, with additional gains on airplane, car, chair, and loudspeaker.Lamp and car remain among the most difficult classes for all methods, including POCO.
  • Experimental materials: The qualitative comparisons include ShapeNet and Synthetic Rooms reconstruction figures, while the experiments use multiple freely available research datasets.
  • Implementation: POCO’s implementation relies on FKAConv, PyTorch, and PyTorch-Geometric, and its code is released under the Apache License 2.0.

F.3. Methods

The evaluation compares POCO with established reconstruction methods, reusing released implementations or previously reported numbers where available. Some methods could not be compared in practice because code, models, or usage instructions were incomplete or unavailable.

  • The study compares POCO with several reconstruction methods using code released by their authors.
  • Previously reported results are reused for AtlasNet, DeepSDF, DP-ConvONet, and ONet rather than rerunning those methods.
  • Some comparisons were unavailable because repositories lacked pretrained models, training or testing instructions, preprocessing procedures, or runnable data pipelines.
  • The paper also discusses applications in heritage preservation, infrastructure maintenance, and augmented or virtual reality.
Loading 2201.01831v2…