Source-linked AI summary
PCN: Point Completion Network
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, Martial Hebert
TL;DR
Shape completion must infer complete object geometry from incomplete, noisy observations. PCN directly maps partial raw point clouds to dense complete point clouds using a compact coarse-to-fine decoder, and experiments report strong completion results across categories, unseen objects, and real-world LiDAR data.
Problem
Incomplete observations from sparsity and occlusion lose geometric and semantic information, creating the need to recover complete object shapes.
Method
PCN uses an encoder-decoder network operating directly on raw point clouds, with coarse points expanded into detailed local patches through a compact multistage decoder.
Results
Experiments show dense, complete, realistic completions across incompleteness and noise levels, with strong generalization to unseen objects and real-world KITTI LiDAR data.
Takeaways & Limitations
Point-based completion preserves fine geometry, handles extremely sparse inputs without fine tuning, and is suitable for real-time applications.
Takeaways & Limitations
PCN can incorrectly connect disconnected object parts and may fail to recover very thin structures such as wires.
Abstract
from arXiv · showhide
Shape completion, the problem of estimating the complete geometry of objects from partial observations, lies at the core of many vision and robotics applications. In this work, we propose Point Completion Network (PCN), a novel learning-based approach for shape completion. Unlike existing shape completion methods, PCN directly operates on raw point clouds without any structural assumption (e.g. symmetry) or annotation (e.g. semantic class) about the underlying shape. It features a decoder design that enables the generation of fine-grained completions while maintaining a small number of parameters. Our experiments show that PCN produces dense, complete point clouds with realistic structures in the missing regions on inputs with various levels of incompleteness and noise, including cars from LiDAR scans in the KITTI dataset.
1. Introduction
PCN addresses incomplete 3D observations by directly mapping partial point clouds to complete shapes. Its point-based, coarse-to-fine design targets dense, fine-grained outputs without voxelization.
- Real-world sensor sparsity and occlusion remove geometric and semantic information, making objects such as LiDAR cars difficult to recognize.
- Existing voxel and distance-field representations limit output resolution through cubically growing memory costs and lose geometric detail.
- PCN operates directly on raw point clouds, avoiding intermediate voxelization while supporting more fine-grained completions.
- Point-cloud completion requires permutation-invariant processing, methods for handling unclear local neighborhoods, and generators capable of producing detailed outputs.
2. Related Work
Prior shape-completion methods use geometric cues, retrieved templates, or learned mappings, with learning-based approaches commonly relying on voxel representations. PCN instead uses point clouds to preserve geometry and reduce memory demands while generating higher-resolution outputs efficiently.
- Geometry-based methods infer missing regions from local reconstruction cues, symmetry, or repeated structures and therefore assume moderately complete inputs.
- Alignment-based methods retrieve complete shapes or parts from databases and may deform templates or assemble geometric primitives to match partial inputs.
- Learning-based methods directly map partial inputs to complete shapes, offering fast inference and better generalization.
- Most learning-based methods voxelize shapes, whereas PCN uses point clouds that preserve geometric information while remaining memory efficient.
- PointNet-style encoders provide permutation invariance and perturbation robustness, and can be incorporated into PCN.
- Earlier point-set decoders generated no more than 2048 points, motivating PCN’s efficient higher-resolution output design.
3. Problem Statement
The paper formulates shape completion as predicting a dense point set covering observed and unobserved object surfaces from observed sensor points. PCN learns this mapping from synthetic partial-complete point-cloud pairs without assuming symmetry or planarity.
- Given observed surface points X, shape completion predicts a dense point set Y sampled from both observed and unobserved object surfaces.
- Because X and Y are independently sampled, the observed input need not be a subset of the target and has no explicit point correspondence with it.
- PCN uses supervised learning on large-scale synthetic pairs of partial and complete point clouds to predict Y directly from X.
- The network is generic across object categories and does not assume structural properties such as symmetry or planarity.
4. Point Completion Network
PCN encodes partial point clouds into features and generates complete outputs through a coarse-to-fine decoder. Its permutation-invariant loss supervises coarse and detailed predictions while using computationally practical distance choices.
- Architecture: PCN encodes an input point cloud into a feature vector, then predicts coarse and detailed complete point clouds for training against ground truth.The model learns a projection from partial observations to complete shapes rather than explicitly retaining input points.
- Architecture: The PointNet-based encoder summarizes geometric information into a 1024-dimensional feature vector while remaining permutation invariant and tolerant to noise.It uses shared multilayer perceptrons and point-wise max pooling to form global features.
- Decoder: The decoder first predicts s coarse points and then folds a local patch of t = u^2 points around each coarse point, producing n = st detailed points.This multistage design combines global geometry prediction with local surface approximation.
- Decoder: The coarse-to-fine decoder generates dense outputs with fewer parameters than a fully connected decoder and more flexibility than a folding-based decoder.The fully connected stage captures global geometry, whereas folding generates local geometry patches.
- Loss Function: Because point clouds are unordered, PCN uses permutation-invariant distances: CD and EMD for the coarse term, but only CD for the detailed term.EMD is too expensive at large point counts, while CD admits more efficient nearest-neighbor computation.
- Loss Function: The loss compares coarse output with a matching-size subsampled ground truth and detailed output with the full ground truth using two weighted terms.The two terms are d1 and d2, weighted by hyperparameter α.
5. Experiments
Experiments evaluate PCN on synthetic ShapeNet completions, novel categories, noise and occlusion, decoder and encoder variants, distance fields, and real LiDAR data. PCN generally outperforms strong baselines, remains robust to severe input degradation, and improves downstream registration.
- Experimental setup: The experiments compare PCN with strong baselines and ablations on synthetic ShapeNet data, then evaluate novel-shape generalization, noise and occlusion robustness, and real-world point clouds.The evaluation also includes downstream point-cloud registration.
- Generalizability to novel objects: PCN outperforms all baselines on novel categories, including visually dissimilar objects, indicating a general shape prior.The result holds for categories excluded from training, including pistol, guitar, motorbike, and skateboard.
- Comparison to volumetric methods: PCN outperforms 3D-EPN by a large margin on both CD and EMD, with the largest gains on instances where 3D-EPN has high error.The point-cloud results improve on the majority of test instances and are strongest on challenging cases.
- Comparison to volumetric methods: PCN yields lower distance-field L1 error than 3D-EPN, with more significant improvements at higher resolutions.The distance-field comparison evaluates average L1 distance on occluded voxels.
- Architecture comparisons: The multistage decoder outperforms fully connected and folding-based alternatives, while the stacked PN encoder outperforms PN2.The decoder combines coarse generation with local folding to improve global geometry and local point density.
- Efficiency and robustness: PCN has an order of magnitude fewer parameters than 3D-EPN and FC while achieving significantly better performance, and its CD and EMD errors increase only gradually with occlusion.The model was not trained on the noisy and occluded examples used in the robustness evaluation.
- Real-world data and downstream use: Despite LiDAR inputs averaging 440 points, with some below 10, PCN transfers from ShapeNet without fine-tuning and produces consistent completions.Complete PCN outputs make registration more accurate for both rotation and translation, especially when partial-input errors are large.
6. Discussion
PCN’s prominent failures arise when objects contain disconnected parts or very thin structures, reflecting mismatches between learned priors, sparse observations, and the folding-based decoder.
- Disconnected object parts are sometimes incorrectly connected because the training data predominantly contains connected objects.
- Sparse points and non-grid-like local geometry make thin structures such as wires difficult to recover.
- Figure 8 visualizes the two reported failure modes: thin structures and disconnected parts.
7. Conclusion
PCN completes shapes directly as point clouds, using a coarse-to-fine architecture to produce dense, high-resolution outputs with few parameters and strong generalization across categories and sensors.
- PCN performs shape completion without voxelization by generating dense point clouds in a coarse-to-fine fashion.
- The method achieves high-resolution completion with much fewer parameters than voxel-based models.
- PCN works across multiple object categories and inputs from different sensors, including unseen objects and real-world data.
- The supplementary material provides technical details and additional quantitative and qualitative results, including ShapeNet and KITTI analyses.
- The local folding operation deforms a u × u grid around each coarse output point to generate t = u^2 local points.
C. Network Architecture Details
The compared architectures use stacked point-cloud encoders and distinct decoders, while PCN’s multistage decoder combines fully connected generation with local folding.
- 3D-EPN is evaluated as the 3D-EPN-unet-class model, using different output resolutions for seen and unseen categories.
- The stacked PN encoder uses two PN layers with progressively wider shared MLPs, while PN2 follows the SSG architecture.
- The FC decoder uses three fully connected layers, whereas the Folding decoder uses two folding layers with u = 128 and r = 0.5.
- PCN’s multistage decoder uses three fully connected layers followed by one local folding layer with u = 4 and r = 0.05.
D. Additional Results on ShapeNet
On ShapeNet, PCN’s outputs most closely match ground truth in both global geometry and local point density, while baseline decoders exhibit distinct artifacts.
- PCN’s outputs best match ground truth in global geometry and local point density across the reported ShapeNet comparisons.
- 3D-EPN outputs often contain missing or extra parts, while FC outputs are accurate but overly concentrated in certain regions.
- Folding produces many floating points, whereas PN2 produces blurry outputs.
E. Additional Results on KITTI
On KITTI, PCN usually produces valid car completions that match the input, but a few high-error failures can distort mean performance. These failures arise when non-car points enter the car bounding box.
- A few KITTI completions have very high fidelity error, potentially biasing the mean reported in Section 5.3.Figure 11 shows the distributions of fidelity error and minimal matching distance.
- Most completions produce valid car shapes matching the input while differing from the closest ShapeNet model.The paper illustrates this behavior with successful qualitative examples.
- Incorrect segmentation causes failures when ground or nearby-object points fall inside the car bounding box.The authors suggest adding a segmentation step before PCN as a possible remedy.
F. More Architecture Analysis
Architecture analyses show that two stacked PN layers provide most of the encoder benefit, while larger bottlenecks improve performance subject to GPU memory limits. The learned representations also develop coarse-to-fine geometric structure and semantic clustering without supervision.
- Effect of stacked PN layers: Two stacked PN layers substantially outperform one, while additional layers provide almost negligible benefit on ShapeNet.The authors retain two layers because they sufficiently mix local and global geometry, although more layers could further improve performance.
- Effect of bottleneck size: PCN performance improves as bottleneck size increases, but the chosen size of 1024 is constrained by single-GPU memory.A bottleneck of 2048 cannot fit into the memory of one GPU.
- Keypoint visualization: The two PN layers summarize shapes coarsely to finely: the first selects outline points, while the second delineates more visible surfaces.This structure emerges without explicit supervision.
- Feature space visualization: The 1024-dimensional global features cluster ShapeNet test instances by semantic category despite training without category labels.The clustering is observed after embedding the features into two dimensions with t-SNE.