PaperlayerView exact revision on arXiv

Source-linked AI summary

PMP-Net++: Point Cloud Completion by Transformer-Enhanced Multi-step Point Moving Paths

Xin Wen, Peng Xiang, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, Yu-Shen Liu

arXiv:2202.09507v3cs.CV

TL;DR

Point cloud completion must recover missing regions, but unordered point sets make generative methods struggle with detailed topology and structure. PMP-Net++ instead deforms each input point along uniquely regularized, multi-step paths, enhanced by transformer-based features; experiments report superiority on completion benchmarks and good up-sampling performance.

  • Problem

    Unordered point clouds make generative completion difficult to optimize for detailed topology and structure, leaving completion performance unsatisfactory.

  • Method

    PMP-Net++ formulates completion as point deformation, using Earth Mover’s Distance-inspired moving-distance regularization, point-level correspondence, multi-step path searching, and transformer-enhanced features.

  • Results

    Experiments show PMP-Net++ superiority over other methods on Completion3D and PCN, with good performance also demonstrated for point cloud up-sampling.

  • Takeaways & Limitations

    The deformation framework consistently refines detailed structure and topology while establishing point-level correspondence between incomplete and complete shapes.

  • Takeaways & Limitations

    PMD loss lacks supervision for point-moving directions or final destinations, so the network may struggle to learn optimal solutions in some examples.

Abstract

from arXiv · show

Point cloud completion concerns to predict missing part for incomplete 3D shapes. A common strategy is to generate complete shape according to incomplete input. However, unordered nature of point clouds will degrade generation of high-quality 3D shapes, as detailed topology and structure of unordered points are hard to be captured during the generative process using an extracted latent code. We address this problem by formulating completion as point cloud deformation process. Specifically, we design a novel neural network, named PMP-Net++, to mimic behavior of an earth mover. It moves each point of incomplete input to obtain a complete point cloud, where total distance of point moving paths (PMPs) should be the shortest. Therefore, PMP-Net++ predicts unique PMP for each point according to constraint of point moving distances. The network learns a strict and unique correspondence on point-level, and thus improves quality of predicted complete shape. Moreover, since moving points heavily relies on per-point features learned by network, we further introduce a transformer-enhanced representation learning network, which significantly improves completion performance of PMP-Net++. We conduct comprehensive experiments in shape completion, and further explore application on point cloud up-sampling, which demonstrate non-trivial improvement of PMP-Net++ over state-of-the-art point cloud completion/up-sampling methods.

1 INTRODUCTION

PMP-Net++ reframes point cloud completion as point deformation rather than direct generation, using point-level moving paths to capture detailed topology and structure. It combines unique path regularization, coarse-to-fine multi-step searching, recurrent path aggregation, and transformer-enhanced point features.

  • Point cloud completion addresses sparse, incomplete scans caused by limited scanner view angles or occlusions for downstream 3D vision tasks.
  • Generative completion struggles to capture detailed topology and structure because unordered points admit multiple target arrangements under CD or EMD.The resulting ambiguity makes the learned correspondence between incomplete and complete shapes difficult to establish.
  • PMP-Net++ moves source points toward the target and learns point-level correspondences, rather than directly predicting all output coordinates.This deformation formulation locally relates each predicted displacement to a source point and target position.
  • Earth Mover’s Distance regularizes total point-moving distances so the network learns a unique, meaningful path arrangement instead of multiple moving solutions.
  • Multi-step coarse-to-fine path searching refines movements across decreasing radii, while Recurrent Path Aggregation uses previous paths and current locations to select the next move.The method repeatedly searches and aggregates moving-path history during deformation.
  • A transformer-enhanced framework improves per-point features over the PointNet++ backbone, aiming to predict more accurate displacements.The paper evaluates PMP-Net++ on Completion3D and PCN and also explores point cloud up-sampling, reporting non-trivial improvement over state-of-the-art methods.

2 RELATED WORK

Prior point-cloud completion methods include volumetric-aided and direct point-cloud approaches, but generative methods struggle to reconstruct detailed topology from unordered points. PMP-Net++ instead uses multi-step deformation with point-moving-distance regularization.

  • Volumetric-aided methods use 3D CNNs to predict coarse-to-fine voxel shapes that are later converted into representations such as meshes or point clouds.
  • Direct point-cloud methods reduce storage and processing costs by completing the representation produced by many 3D scanners.
  • Generative methods such as TopNet, PCN, and SA-Net encode incomplete inputs and decode complete point clouds, with architectural differences in tree generation and attention.
  • Because unordered points hinder detailed topology reconstruction, PMP-Net++ moves input points rather than directly generating the final point cloud.
  • Multi-step searching and point-moving-distance regularization help PMP-Net++ infer detailed missing-region geometry while keeping multi-step inference efficient.

3 ARCHITECTURE OF PMP-NET

PMP-Net++ combines point-feature extraction, per-point path prediction, and recurrent path-history integration. Its path search repeatedly refines movements using progressively smaller radii.

  • The RPA module recurrently learns and forgets path-searching information from previous steps.
  • Path searching moves points through multiple coarse-to-fine steps, reducing the search radius at each step.
  • The architecture contains an encoder, a feature propagation module that predicts each point’s moving path, and an RPA module that fuses current features with previous path information.

3.1 Point Displacement Prediction

Point displacement prediction moves each input point toward its target through repeated refinement, while transformer-enhanced features provide spatially informed per-point representations for predicting displacements.

  • Multi-step framework: PMP-Net++ predicts displacement vectors that move each input point into the corresponding target point position.
  • Multi-step framework: Each point is moved for K = 3 steps, with every step taking the preceding deformed point cloud as input for continued refinement.
  • Transformer-enhanced displacement prediction: Transformer modules are inserted between PointNet++ set-abstraction layers to enhance local point features.
  • Transformer-enhanced displacement prediction: Learnable position encoding uses the 3D coordinates of point pairs to guide spatial-relation learning.
  • Transformer-enhanced displacement prediction: Per-point features are indexed by step and propagation level, then concatenated with random noise before an MLP and tanh produce a 3-dimensional displacement vector.
  • Multi-step framework: The RPA unit is placed between propagation steps and levels to retain previous path information and infer the next movement.

3.2 Recurrent Path Aggregation

Recurrent Path Aggregation adapts gated-recurrent processing to point-motion histories, selectively retaining past paths while emphasizing current-step information when deciding the next movement.

  • RPA treats previous point movements as sequential data and uses update and reset gates to encode or forget path information.
  • The gate calculations use weight matrices, biases, sigmoid activation, and feature concatenation.
  • Unlike a standard GRU, RPA emphasizes preserving current input information when producing the current-step feature.
  • The intermediate feature preserves selected information from earlier steps according to the current input feature.
  • Setting the update gate to a zero-vector allows RPA to forget all history and focus fully on the current-step input.

3.3 Optimized Searching for Unique Paths

PMP-Net++ resolves ambiguity in point-cloud deformation by enforcing shortest overall and per-step moving paths, while searching paths from coarse to fine.

  • The unordered point-cloud representation permits multiple deformations, so direct shape constraints cannot guarantee unique input-target correspondence.
  • Minimizing moving distance: Earth Mover’s Distance defines a bijection that minimizes average corresponding-point distance and yields a unique correspondence under the paper’s formulation.
  • Minimizing moving distance: PMP-Net++ minimizes Point Moving Distance loss across all three predicted steps to regularize the paths between corresponding source and target points.
  • Minimizing moving distance: Eq.(9) additionally constrains each step’s path to be shortest, encouraging subsequent moves to follow previous directions and reducing redundant decisions.
  • Multi-scaled searching radius: Coarse-to-fine searching reduces each step’s maximum stride by a power of 10, limiting later searches from overturning earlier decisions.
  • Multi-scaled searching radius: Minimizing point moving distance encourages consistent source-to-target paths, reducing redundant searching and improving efficiency.

3.4 Extension to Dense Point Cloud Completion

To extend deformation-based completion to dense outputs, PMP-Net++ injects noise into each step’s input and overlaps multiple deformation results.

  • Deformation preserves the input-output point count, so it cannot directly increase the number of points.
  • For dense completion, each step concatenates its input point cloud with noise sampled from a standard normal distribution N(0, 1).
  • Repeated deformations produce varied outputs that are overlapped to form a dense point cloud with more points than the original input.
  • Duplicated points receive feature noise so they move to different locations instead of producing identical displacements.

3.5 Training Loss

Training combines shape-level Chamfer-distance supervision with point-moving-distance regularization, using an approximate Earth Mover correspondence.

  • The deformed shape is regularized against the complete ground-truth point cloud using Chamfer distance and Earth Mover’s Distance.
  • The total training loss includes LCD(P k, P ′) + LPMD, combining output-shape discrepancy with point-moving-distance regularization.
  • P k denotes the point cloud output at step k and P ′ the target complete point cloud; the optimal correspondence φ is approximated computationally.

4 EXPERIMENTS

Experiments evaluate PMP-Net++ on PCN and Completion3D completion benchmarks, dense completion, visual comparisons, up-sampling, cross-dataset transfer, and ablations.

  • 4 EXPERIMENTS: Experiments cover PCN, Completion3D, point-cloud up-sampling, and comprehensive ablation studies.
  • Implementation: The implementation uses PointNet++ SSG with feature propagation, Adam optimization, a 10−3 initial learning rate, and 150 training epochs.
  • Dense completion: PMP-Net++ trained on 2,048-point sparse inputs can generate 16,384-point dense shapes on PCN by repeating point-moving processes.
  • PCN completion: PMP-Net++ ranks first on PCN and achieves performance comparable to the state-of-the-art method, while PMD loss improves its performance.
  • Visual comparisons: Visual comparisons report better preservation of detailed geometries than competing methods on PCN and Completion3D, including chair beams, chair legs, and table structures.
  • Completion3D completion: On Completion3D, PMP-Net++ achieves the best average Chamfer distance across categories and the best results in 7 of 8 categories.
  • Completion3D completion: PMP-Net++ improves Completion3D average CD over VRCNet by 0.17 and reduces PMP-Net’s average CD loss by 13.8%.
  • Cross-dataset completion: On ScanNet chairs, PMP-Net++ completes shapes with less noise than GRNet, while its point-moving formulation can preserve residual input points.

4.4 Point Cloud Up-sampling

PMP-Net++ is adapted from dense completion to point cloud up-sampling, where increasing point density should reveal finer geometric information. It achieves the best reported comparison performance and produces more complete, evenly distributed surfaces than PU-GAN in visual examples.

  • Task and settings: PMP-Net++ applies its dense-completion version to point cloud up-sampling, which increases point count to reveal finer 3D geometric information.Evaluation follows PU-GAN’s dataset and settings, using 120 training models and 27 testing models from 147 total models.
  • Quantitative comparison: PMP-Net++ achieves the best performance among the compared point cloud up-sampling methods.The authors use this result to support the network’s generalization across completion and up-sampling tasks.
  • Visual comparison: PMP-Net++ distributes up-sampled points more evenly than PU-GAN on the bird-head example and avoids the incomplete surface holes visible in PU-GAN’s result.The comparison is based on the visual examples in Figure 13.

4.5 Model Analysis

Model analyses examine the recurrent path aggregation module, transformer-enhanced features, deformation steps, searching-radius schedules, deformation inputs, efficiency, and geometric behavior. These studies report benefits from RPA, transformer features, multi-step coarse-to-fine refinement, and geometric correspondence, while also identifying design trade-offs and computational advantages.

  • RPA and PMP loss: RPA-based variations achieve the best performance across both PMP-Net and PMP-Net++ backbones, supporting RPA’s effectiveness across network structures.The ablation compares RPA with additive and recurrent alternatives and includes PMP loss analysis.
  • Transformer enhancement: PMP-Net++ outperforms PMP-Net across all tested unit variations, with its worst performance still better than PMP-Net’s best.The comparison is used to justify the effectiveness of the newly added transformer unit.
  • Deformation steps: At step 2, PMP-Net++ achieves CD 8.36 versus PMP-Net’s 11.90, while the preceding step reports 9.45 versus 12.26.The experiments compare one, two, and four deformation steps with a fixed tenfold searching-radius ratio.
  • Geometric correspondence: The learned deformation uses edge proximity for missing-part completion and one-to-one geometric correspondence for analogous structures such as chair legs.The authors attribute the main improvement to geometric correspondence rather than border-information constraints in PMD loss.
  • Grid-point deformation: Grid-point deformation obtains average CD 9.98, lower but relatively comparable to PMP-Net++, with ordered predictions helping while incomplete-shape deformation uses more geometric information.The alternative deforms 512 points arranged as an 8 × 8 × 8 grid into a complete shape.
  • Searching radius: Decreasing searching radius improves performance over the non-decreasing [1.0, 1.0, 1.0] strategy, while larger decreases better prevent overturning earlier decisions.The default schedule is [1.0, 0.1, 0.01], compared with [1.0, 0.5, 0.25] and the non-decreasing schedule.
  • Visual deformation analysis: Coarse-to-fine searching produces a more complete early shape and enables later refinement of detailed structure.Figure 17 compares deformation outputs under different searching-radius strategies; Figure 15 similarly shows cleaner chair-back completion with multi-step searching.
  • Noise and efficiency: Larger noise standard deviation improves completion until its influence weakens around 10^-1, while PMP-Net++ is more efficient than GRNet and comparable with PCN.The efficiency comparison considers both parameter count and FLOPs on Completion3D.

5 CONCLUSIONS

The paper concludes that PMP-Net++ completes point clouds through multi-step deformation, refining topology and structure while establishing point-level correspondence. Experiments report superiority on completion benchmarks and good performance for up-sampling, but the deformation process remains limited by insufficient constraints.

  • Conclusions: PMP-Net++ performs point cloud completion through multi-step shape deformation from source to target point clouds.The conclusion identifies multi-step deformation as the paper’s central formulation.
  • Conclusions: PMP-Net++ consistently refines predicted topology and detailed structure while establishing point-level correspondence between incomplete and complete shapes.These are the principal structural properties reported for the predicted completions.
  • Conclusions: Experiments report superiority on the Completion3D benchmark and PCN dataset, along with good performance on point cloud up-sampling.The conclusion summarizes the paper’s cross-task experimental findings.
  • Limitations: The method is limited by insufficient deformation constraints because PMD loss does not supervise point-moving directions or final destinations.The authors state that this may make learning the optimal solution difficult in some examples.
Loading 2202.09507v3…