PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive Review of Deep Learning-Based 3D Point Cloud Completion Processing and Analysis

Ben Fei, Weidong Yang, Wenming Chen, Zhijun Li, Yikang Li, Tao Ma, Xing Hu, Lipeng Ma

arXiv:2203.03311v3cs.CV

TL;DR

Point cloud completion must recover complete shapes from partial observations while improving quality enough for practical use. This paper systematically surveys deep-learning approaches, datasets, applications, comparisons, and research directions. It concludes that substantial progress exists, but practical completion still requires better structural information use and fine-grained shape prediction.

  • Problem

    Point cloud completion remains challenging because unstructured point clouds provide insufficient structural information for recovering complete 3D shapes.

  • Method

    The paper conducts a comprehensive survey and taxonomy of deep-learning point cloud completion methods, comparing approaches, datasets, metrics, applications, and research directions.

  • Results

    Point-based, GAN-based, and transformer-based methods have achieved completion effects, while SnowflakeNet is identified among the state-of-the-art methods.

  • Takeaways & Limitations

    Future progress should focus on exploiting structural information, predicting fine-grained complete shapes, and developing novel loss functions.

Abstract

from arXiv · show

Point cloud completion is a generation and estimation issue derived from the partial point clouds, which plays a vital role in the applications in 3D computer vision. The progress of deep learning (DL) has impressively improved the capability and robustness of point cloud completion. However, the quality of completed point clouds is still needed to be further enhanced to meet the practical utilization. Therefore, this work aims to conduct a comprehensive survey on various methods, including point-based, convolution-based, graph-based, and generative model-based approaches, etc. And this survey summarizes the comparisons among these methods to provoke further research insights. Besides, this review sums up the commonly used datasets and illustrates the applications of point cloud completion. Eventually, we also discussed possible research trends in this promptly expanding field.

I. INTRODUCTION

Point cloud completion addresses sparse and partial scans caused by acquisition and processing limitations, supporting downstream 3D vision applications. This survey reviews recent deep-learning methods, compares them across public datasets, and discusses future research.

  • Motivation: Raw point clouds are sparse and partial because of occlusions, reflections, transparency, and scanner resolution or angle limits.Subsequent denoising, smoothing, registration, and fusion can further increase missing points and cause topology and reconstruction errors.
  • Applications: Completing point clouds supports 3D map reconstruction, robotic scene reconstruction, underground mining safety monitoring, 3D detection, and shape classification.In robotics, completed scenes can assist route planning and decision-making; sparse distant vehicles are difficult to detect without complete point clouds.
  • Survey scope: The survey reviews rapid deep-learning progress from 2017–2022 because few existing surveys covered the latest point cloud completion developments and downstream tasks.It aims to stimulate development in both industry and academia through comparisons among deep-learning techniques.
  • Method landscape: Recent point cloud completion methods include point-based, view-based, convolution-based, graph-based, generative model-based, and transformer-based approaches.Earlier approaches transferred voxelization and 3D convolution from 2D completion, but their computational costs increase with spatial resolution.
  • Survey contributions: The review systematically compares existing deep-learning methods on public datasets and summarizes their methods, contributions, and limitations.It also discusses future research directions intended to stimulate improvement in point cloud completion.

III. CHALLENGES

Point cloud completion must reconstruct fine-grained structure from disordered inputs while capturing geometric and relational information. The review examines existing solutions and organizes relevant datasets for evaluating shape completion.

  • Challenges: Point cloud completion is challenging because required structural information conflicts with the disordered, unstructured nature of point clouds.Real-world shapes contain surfaces, semantic parts, and geometrical elements that completion methods must represent.
  • Challenges: Fine-grained completion should exploit geometrical symmetry, regular arrangements, surface smoothness, and other relational structure information.Existing strategies include iterative refinement, global-local feature integration, skip-connections, and residual connections.
  • Challenges: The review investigates state-of-the-art completion performance and the solutions used to address these significant challenges.
  • Datasets: Shape-completion datasets are categorized as artificial or real-world, with commonly studied resources including PCN, KITTI, ModelNet40, and Completion3D.Additional datasets increase object variety, viewpoints, and defect diversity toward real-world conditions.

V. METRICS

The review describes several metrics for evaluating point cloud completion, covering geometric distance, reconstruction fidelity, consistency, density distribution, surface accuracy, and uniformity. These metrics assess complementary properties of completed point clouds, including proximity to ground truth, input preservation, typical-shape reconstruction, and point distribution.

  • Distance metrics: Chamfer Distance measures the sum of average closest-point distances in both directions between reconstructed and complete point clouds.CD has CD-T (CD-ℓ1) and CD-P (CD-ℓ2) variants.
  • Distance metrics: Earth Mover’s Distance finds a bijection between two point sets that minimizes average distances between corresponding points.Unlike CD, EMD requires the two point sets to have the same size.
  • Specialized metrics: Fidelity measures input preservation, MMD measures reconstruction of a typical car, and consistency measures output stability against input variations.These metrics were proposed by PCN for evaluating different properties of completion outputs.
  • Density and surface metrics: Density-aware Chamfer Distance detects density-distribution disparities while attending to overall structure and local geometric details.DCD is derived from CD.
  • Density and surface metrics: F-score is the harmonic mean of precision and recall for evaluating object-surface distance.Precision measures reconstructed points near ground truth, while recall measures ground-truth points near the reconstruction; threshold d controls strictness.
  • Distribution metrics: Uniformity evaluates the distribution uniformity of completed point clouds using global imbalance and local clutter terms.Point subsets are cropped from output patches using farthest sampling and ball queries; the expected subset size is n̂ = √p|S1|.

VI. METHODS

Point cloud completion architectures are categorized by their employed network structures into point-based, view-based, convolution-based, graph-based, transformer-based, generative model-based, and other methods. Because many works are hybrid, a method may belong to multiple categories according to its stated highlights.

  • Architecture taxonomy: Existing completion architectures are categorized as point-based, view-based, convolution-based, graph-based, transformer-based, generative model-based, and other methods.The taxonomy is based on network structures used for point cloud completion and generation.
  • Architecture taxonomy: Hybrid methods may belong to several categories when their stated highlights use multiple architectural approaches.The review discusses works according to the methods they utilize.

A. Point-based methods

Point-based methods process point sets directly, using point-wise features, global aggregation, folding, attention, and coarse-to-fine decoders. Their central difficulty is recovering local geometric details and complex topologies from incomplete observations.

  • Point-based networks process points independently with MLPs and aggregate global features through symmetric functions, preserving permutation invariance but incompletely modeling whole-group geometry.
  • Preliminary works: PointNet++, TopNet, and related hierarchical architectures incorporate geometric information through multi-level aggregation or structured point generation without assuming a particular topology.
  • End-to-end mechanism: Encoder-decoder point-based methods extract global and regional features, then generate and refine completions through end-to-end or coarse-to-fine pipelines.
  • Folding-derived methods: Folding-based decoders reconstruct point clouds by deforming 2D grids or surface elements into 3D shapes, while AtlasNet and MSN use multiple parametric surface elements.
  • Folding-derived methods: FoldingNet samples the same 2D grid for every parent point, overlooking local shape characteristics and motivating alternative decoder designs.

B. View-based methods

View-based completion integrates partial point-cloud information with global shape information from single-view images. Its performance depends substantially on the angle and number of available views.

  • View-based methods combine pose and regional details from incomplete point clouds with global shape information from single-view images.
  • ViPC retrieves missing global structural information from an additional single-view image and uses a Dynamic Offset Predictor to refine coarse outputs.
  • View-based inputs may be RGB-D images or depth images rather than point clouds alone.
  • Performance depends largely on the angle and number of views because different views provide different information.

C. Convolution-based methods

Convolution-based methods use grid, voxel, or continuous-space operators to learn point-cloud representations and reconstruct completions. They retain convolutional processing benefits but face quantization, memory, resolution, and neighborhood-modeling challenges.

  • 3D CNN approaches voxelize point clouds, but quantization causes detail loss and insufficient representation of fine-grained information.
  • Preliminary works: PointCNN applies X-conv on transformed point neighborhoods, while KPConv and PointConv define learnable or dynamic convolution operators in continuous space.
  • Hierarchical convolution recursively aggregates neighborhoods into fewer representative points with richer information, illustrated by the 9 →5 →2 progression.
  • Convolutional encoder: Convolutional encoders and decoders include IF-Nets, SVCN, GRNet, VE-PCN, MRAC-Net, SoftPoolNet, and grid-transformation methods for completion and detail recovery.
  • Most convolution-based completion methods voxelize point clouds before applying 3D convolution, creating volumetric-representation limitations.
  • Voxel or grid representations include ineffective occupied and non-occupied regions, increasing storage demands; grid size also affects scale and spatial relationships.
  • Volumetric computation and memory requirements grow cubically with resolution.
  • Graph-based methods: Graph-based alternatives represent points as vertices and use adjacent-point edges to model non-Euclidean spatial relationships.

E. GAN-based methods

GAN-based point cloud completion uses adversarial training to generate completed shapes, but 3D point distributions create distinctive challenges. The review covers refinements, multi-view representations, latent-space methods, and integrations with reinforcement learning or graph convolution.

  • Challenges: 3D GAN completion can produce inhomogeneous point distributions, creating unwanted holes that undermine predicted shape integrity.Unlike image grids, point clouds are unstructured, making evenly distributed generation difficult.
  • Challenges: Point-cloud disorder makes correspondence-based consistency comparisons difficult, and GAN inversion can therefore yield poor reconstruction.Partial and complete 3D shapes may place corresponding regions at different spatial positions.
  • Challenges: Simple GANs generally generate only 1024 or 2048 points because complicated distributions make training difficult.Researchers consequently developed modified GAN architectures for point cloud completion.
  • GAN architecture: GANs pair a generator that produces synthetic points from random samples with a discriminator that distinguishes synthetic samples from real data.This adversarial architecture implicitly learns to estimate point collections produced by the generator.
  • GAN variants: Other GAN variants use latent-space transformations, hierarchical sampling, feature alignment, reinforcement learning, or graph convolution to improve completion.Examples include Cycle4Completion, ShapeInversion, PC-GAN, RL-GAN-Net, and graph-convolutional generators.
  • Multi-view methods: Multi-view GAN completion uses information shared across views, completing multiple depth maps before back-projecting them into a completed shape.MVCN uses a conditional GAN to leverage global and regional information from all views.

F. Variational autoencoders (VAEs)-based methods

VAE-based methods learn latent representations from complete shapes and use conditional generative modeling to complete partial inputs. The review highlights their stability and controllable diversity alongside lower completion quality than GAN-based methods.

  • VAE framework: VAE completion models learn the mode distribution of complete shapes and generate completions for partial inputs in conditional generative settings.Classic autoencoders and VAEs determine latent representations from complete 3D objects before completing incomplete data.
  • VAE methods: HyperPocket disentangles latent representations to generate multiple completed point-cloud variants, while VRCNet adds VAE-based relational enhancement.HyperPocket uses two unconnected data streams and a hypernetwork to fill spaces left by missing parts.
  • Advantages: VAE methods provide more stable training than other 3D generative models.The review identifies training stability as a principal advantage of VAE-based approaches.
  • Advantages: VAE latent representations can be manipulated to control generated point-cloud shapes.This controllability supports producing different shape variants from the learned latent space.
  • Trade-offs: VAE-based methods offer greater diversity but lower quality than GAN-based methods, with injected noise and squared-error measures potentially producing nonsmooth shapes.The review contrasts VAE diversity with GAN quality and identifies smoothness as a limitation.
  • Future directions: AutoSDF is identified as a VQ-VAE-like enhancement that may advance VAE-based point cloud completion.The review connects this direction to the success of VQ-VAE in image generation.

G. Transformer-based methods

Transformer-based methods apply attention to represent irregular point clouds and generate missing geometry. The review describes encoder-decoder, decoding, and point-generation designs while noting deployment and interpretability limitations.

  • Background: Transformers entered point-cloud processing after successful applications in natural language processing and 2D computer vision.PCT, Pointformer, and PointTransformer pioneered transformer use for point clouds.
  • Encoder-decoder methods: Transformer encoder-decoder completion treats point clouds as set-to-set translation and uses position embeddings with point proxies.A geometry-aware block explicitly simulates local geometric relations to incorporate 3D inductive bias.
  • Decoding methods: SnowflakeNet applies transformer structures during decoding and models completion as snowflake-like point growth through parent-child splitting.Snowflake Point Deconvolution progressively generates child points from parent points.
  • Point-generation methods: PCTMA-Net uses attention to extract local context and incomplete structure details before a morphing-atlas network predicts missing regions.Transformer-enhanced representation learning is also used in PMP-Net++.
  • Limitations: Transformer models are relatively large for device deployment, and their performance-enhancing mechanism is generally difficult to interpret.The review notes visual interpretation of attention as an exception in SA-Net.

VII. COMPARISON

The review compares recent completion methods across synthetic and real-world datasets, including performance, complexity, and generalization. It reports strong results from several model families while emphasizing dataset and deployment trade-offs.

  • Evaluation setup: The comparison aggregates results from original papers, whose resolutions and dataset settings vary across methods.Performance comparisons are directly comparable under the same resolution when CD type or dataset settings are unspecified.
  • Synthetic benchmarks: PCN is the most commonly used dataset among the synthetic PCN, ModelNet, and Completion3D benchmarks.These datasets are synthetic benchmarks used for completion evaluation.
  • Method patterns: Point-based models commonly use MLPs as basic units to learn point-wise information.This is a recurring architectural pattern in the benchmarked methods.
  • Method patterns: Graph-based and GAN-based networks achieve excellent completion results, motivating further attention to combining the two approaches.The review presents this combination as a research direction rather than a quantified conclusion.
  • Real-world evaluation: KITTI lacks ground truth and contains sparse real-world scans, making completion difficult, although point-based, GAN-based, and transformer-based methods achieve completion effects there.Some works specifically target real-world point clouds, which the review identifies as needing more attention.
  • Complexity: FoldingNet has the fewest parameters, PMPNet the lowest computational cost, and GRNet the highest computational cost in the reported comparison.GRNet and PF-Net have relatively many parameters, while SnowflakeNet and PoinTr also have relatively high parameter counts because of attention.
  • Generalization: PoinTr shows the most remarkable generalization on ShapeNet55 and ShapeNet34, trained on 34 seen categories and evaluated on 21 unseen categories.These datasets measure performance on unknown categories.

VIII. APPLICATIONS

Point cloud completion supports applications spanning object-level reconstruction, industrial measurement, mining, autonomous driving, and robotics. The review organizes these application contexts and highlights how completion addresses sparse or incomplete 3D data.

  • Object-level completion: Object-level completion reconstructs individual objects in real-world scenes, while relationship-based methods condition each object’s completion on another object’s partial scan.The review distinguishes single-object completion from pairwise scene completion based on inter-object information.
  • Industry: Industrial point-cloud sensing offers higher measurement accuracy and efficiency than traditional manual or equipment-based approaches in manufacturing and construction contexts.The cited example concerns point-cloud acquisition at a precast concrete manufacturing plant.
  • Mining and robotics: Completion supports underground mining by enabling 3D mapping and reconstruction for monitoring, while robot applications use completed scenes for route planning, localization, and decision-making.The review describes high-precision underground-road mapping and completed-scene construction for robot navigation.
  • Autonomous driving: In autonomous driving, completion can reduce sparse LiDAR problems in semantic scene completion, although voxelization-based convolution methods may lose scene details.Point-based completion is presented as a possible way to preserve more detail in sparse LiDAR sweeps.

E. Robotics

Robotics applications require accurate, robust, and fast completion for localization and mapping in unknown environments. The review identifies real-time efficiency, structural representation, realistic reconstruction, data realism, and generative modeling as continuing challenges and research directions.

  • Robotics: Robotic localization and mapping require accurate 6-DoF poses and efficient operation in unknown 3D environments.RGB-D-based localization can require substantial orientation time, motivating completion methods for robotic navigation settings.
  • Open questions: Point clouds lack corresponding point pairs unlike images, limiting direct transfer of image-generation methods to point cloud completion.Solving correspondence is identified as a prerequisite for applying more image-generation techniques.
  • Generative and view-based methods: Future work includes combining view-based methods with NeRF to reconstruct more realistic 3D shapes from images and integrating diffusion models into completion.Diffusion-based sample generation remains constrained by the need to improve generation speed.
  • Data: Existing networks are trained mainly on CAD-derived datasets, creating an urgent need for real-world datasets that improve robustness in unconstrained scenes.The review also calls for further attention to unsupervised methods because captured real-world point clouds remain challenging.
  • Representation and architecture: Point-cloud disorder makes voxelization computationally costly and can discard effective information, so feature learning and decoder design remain active issues.The review notes PointNet and GCN for feature extraction and identifies fully connected, FoldingNet, and transformer-based decoders as existing designs.
  • Future directions: Transformer-based methods are expected to receive increased attention, while practical completion still requires stronger structural-information use and finer-grained shape prediction.The conclusion characterizes these as two central challenges despite significant progress in deep-learning models.
  • Challenges: Few networks currently support robust real-time completion, while network training remains time-consuming, motivating lightweight and compact architectures.The review frames fast operation speed and high accuracy as joint requirements for practical deployment.
Loading 2203.03311v3…