Source-linked AI summary
PU-GAN: a Point Cloud Upsampling Adversarial Network
Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, Pheng-Ann Heng
TL;DR
Sparse, noisy, and non-uniform scanned point clouds require upsampling that also fills gaps and preserves surface geometry. PU-GAN uses adversarial learning with specialized feature expansion, self-attention, and compound losses; experiments report stronger uniformity, surface proximity, and reconstruction quality than state-of-the-art methods, within patch-level scope.
Problem
Scanned point clouds are sparse, noisy, and non-uniform, creating a need for dense, complete, uniform, surface-faithful representations that also fill small gaps.
Method
PU-GAN combines patch-based point upsampling with adversarial learning, using up-down-up feature expansion, self-attention, and adversarial, uniform, and reconstruction losses.
Results
Extensive experiments report that PU-GAN outperforms state-of-the-art methods in distribution uniformity, proximity-to-surface, and 3D reconstruction quality.
Takeaways & Limitations
PU-GAN combines upsampling with data amendment to produce uniformly distributed point sets while implicitly penalizing outputs that deviate from expected targets.
Takeaways & Limitations
Patch-level analysis can complete tiny holes but has limited ability to fill large gaps or holes because it lacks a global view of overall shape.
Abstract
from arXiv · showhide
Point clouds acquired from range scans are often sparse, noisy, and non-uniform. This paper presents a new point cloud upsampling network called PU-GAN, which is formulated based on a generative adversarial network (GAN), to learn a rich variety of point distributions from the latent space and upsample points over patches on object surfaces. To realize a working GAN network, we construct an up-down-up expansion unit in the generator for upsampling point features with error feedback and self-correction, and formulate a self-attention unit to enhance the feature integration. Further, we design a compound loss with adversarial, uniform and reconstruction terms, to encourage the discriminator to learn more latent patterns and enhance the output point distribution uniformity. Qualitative and quantitative evaluations demonstrate the quality of our results over the state-of-the-arts in terms of distribution uniformity, proximity-to-surface, and 3D reconstruction quality.
1. Introduction
PU-GAN addresses sparse, noisy, and non-uniform scanned point clouds by combining upsampling with data amendment, using adversarial learning to improve point distributions and surface fidelity.
- Raw depth-camera and LiDAR point clouds are often sparse, noisy, and non-uniform, limiting their use for rendering, analysis, and processing.
- The target is a dense, complete, and uniform point cloud that faithfully represents the underlying surface, including filled small holes and gaps.
- Existing learned methods improve point-cloud upsampling but may produce implausible results from especially sparse and non-uniform inputs.
- PU-GAN combines upsampling with data amendment and uses adversarial learning to generate diverse point distributions while penalizing outputs that deviate from the target.
- Its generator uses up-down-up expansion with error feedback and self-correction, self-attention for feature integration, and a compound adversarial, uniform, and reconstruction loss.
- Extensive evaluations report improvements over state-of-the-art methods in distribution uniformity, proximity-to-surface, and 3D reconstruction quality.
2. Related Work
Prior point-cloud upsampling progressed from optimization-based shape priors to learned local-patch methods, while GAN research supplied a framework for learning diverse output patterns.
- Optimization-based upsampling: Optimization-based methods use geometric operators, shape priors, and iterative procedures to insert or resample points while handling surfaces, noise, edges, and nonuniformity.
- Deep-learning-based upsampling: Deep networks learn features directly from point sets, and point-cloud upsampling improves local-patch uniformity rather than generating an entire object from partial input.
- Deep-learning-based upsampling: PU-Net expands point sets by mixing and blending point features in feature space, illustrating the learned approach to patch-based upsampling.
- GAN-based 3D shape processing: GANs use a discriminator to implicitly evaluate generated point sets, benefiting generative tasks where the generator should produce diverse output patterns.
- GAN-based 3D shape processing: Earlier GAN-based 3D shape methods often operated on probabilistic spaces, images, volumes, or voxel representations rather than directly addressing raw point-cloud upsampling.
3.1. Overview
PU-GAN maps an unordered sparse point set to a denser, surface-faithful, uniformly distributed output and organizes generation around a generator–discriminator architecture trained on patches.
- Given N input points, PU-GAN generates rN output points, where r is the upsampling rate.
- The output should preserve the input’s underlying geometry, cover the target surface, and remain uniformly distributed even for sparse and non-uniform inputs.
- The generator produces dense output Q from sparse input P, while the discriminator identifies generated point sets.
- The architecture includes generator and discriminator components, up-down-up expansion and self-attention building blocks, and patch-based training with a compound loss.
3.2. Network Architecture
PU-GAN extracts point features, expands them into diverse latent representations, samples separated candidates, and uses self-attention-enhanced discrimination to support point-set generation.
- Feature extraction: The generator extracts features from input point attributes, using dense connections to integrate information across layers.
- Feature expansion: Its feature expansion component uses an up-down-up unit to enhance feature variations and enable more diverse point distributions.
- Point-set generation: The point-set generator regresses 3D coordinates from expanded features and uses farthest sampling to retain rN mutually separated points.
- Discriminator: The discriminator extracts global features, combines local and global information, and adds self-attention after feature concatenation.
3.3. Up-down-up Expansion Unit
PU-GAN’s up-down-up expansion unit expands point features, feeds them back through downsampling, and uses feature differences for self-correction and finer feature variation.
- Design motivation: PU-GAN’s expansion design targets the similarity problem of duplicated features that can impair upsampling quality.It is presented as an alternative to single-step duplication and progressive multi-step expansion.
- Up-down-up expansion unit: The unit upsamples features, downsamples them back, computes their difference, and upsamples that difference to self-correct the expanded features.This feedback strategy is intended to facilitate fine-grained features without tedious multi-step training.
- Up-feature operator: The up-feature operator duplicates each feature map, appends a unique 2D grid vector to each copy, and processes the result with self-attention and MLPs.The grid vectors increase variation among duplicated features and push new points away from the inputs.
- Down-feature operator: The down-feature operator reshapes expanded features and uses MLPs to regress the original features.
3.4. Self-attention Unit
PU-GAN uses self-attention in both generator and discriminator modules to integrate features through long-range context dependencies.
- Attention construction: The self-attention unit transforms input features into G and H with separate MLPs and derives attention weights W from them.The attention weights are generated using a softmax operation.
- Network placement: The unit is used in the generator and discriminator to enhance feature integration and subsequent feature extraction.
- Feature integration: It computes weighted features from K and adds them to the input features to produce the output features.K contains features extracted from the input through another MLP.
3.5. Patch-based End-to-end Training
PU-GAN trains on normalized surface patches, evaluates generated distributions with adversarial and uniform losses, and adds reconstruction loss to keep points near target surfaces.
- 3.5.1 Training data preparation: Training samples 200 geodesic patches per normalized mesh, each covering about 5% of the surface, then forms inputs by subsampling Poisson-disk targets.Each target set ˆQ contains rN points, while P contains N points selected on the fly.
- 3.5.2 Loss functions: The adversarial loss trains G to fool D while D learns to distinguish generated Q from target ˆQ using least-squares objectives.
- 3.5.2 Loss functions: The uniform loss combines local clutter and nonlocal imbalance terms to evaluate generated patch uniformity and encourage better point coverage.Local subsets are sampled around farthest-sampled seeds and assessed using point counts and nearest-neighbor distances.
- 3.5.2 Loss functions: Luni distinguishes point-distribution uniformity among point sets with the same number of points, unlike NUC in the illustrated case.Figure 6 uses 625-point sets and computes Luni with p=1%.
- 3.5.2 Loss functions: The reconstruction loss uses Earth Mover’s distance because adversarial and uniform losses alone do not encourage generated points to lie on the target surface.Generator and discriminator losses are minimized end-to-end with weighted adversarial, reconstruction, and uniform terms.
4. Experiments
PU-GAN is evaluated on synthetic and real-scanned point clouds using quantitative metrics, visual comparisons, ablations, robustness tests, and a classification application. It consistently improves point distribution and reconstruction-related outcomes, while remaining effective on noisy, sparse, and small inputs.
- Datasets and implementation: The evaluation uses 147 models, with 120 for training and the remainder for testing, producing 24,000 training patches.Training uses 200 patches per training model, with default settings N = 256, r = 4, and M = 50.
- Evaluation protocol: PU-GAN is compared against EAR, PU-Net, and MPU using uniformity, P2F, Chamfer, and Hausdorff metrics.Testing uses 8,192 Poisson-disk ground-truth points and 2,048 randomly selected input points; lower metric values are better.
- Quantitative comparisons: PU-GAN achieves the lowest values for all evaluation metrics, with the lowest uniformity across all tested scales.This indicates more uniform generated points than the compared methods over varying neighborhood sizes.
- Qualitative comparisons: Visual comparisons show that PU-GAN produces less noisy, more uniform point sets, fewer reconstruction artifacts, and finer details than competing methods.Examples include the elephant’s nose and tiger’s tail in reconstructed surfaces.
- Real-scanned data: On LiDAR point clouds, PU-GAN fills some holes and outputs more uniform points despite sparse and non-uniform inputs.The limitation is that patch-level processing has limited ability to fill large gaps or holes.
- Ablation and baseline comparison: Removing any component reduces performance, with discriminator removal causing the largest drop; a basic GAN alone is insufficient.The ablation evaluates the GAN framework, expansion unit, self-attention, uniform loss, and farthest sampling.
- Robustness tests: PU-GAN remains robust to increasing Gaussian noise and stable with inputs containing only 512 points.These tests assess performance under varying noise levels and input sizes.
- Application: Using PU-GAN before PointNet classification increases accuracy from 82.4% to 83.8% on ModelNet40.Both sparse training and testing point clouds are upsampled from 512 to 2,048 points in the improved setting.
5. Conclusion
PU-GAN combines GAN-based upsampling with data amendment, using specialized feature expansion, attention, and compound losses. Extensive experiments show state-of-the-art performance, while patch-level processing remains limited for large gaps and holes.
- 5. Conclusion: PU-GAN combines point cloud upsampling with data amendment capabilities through a GAN-based framework.The generator produces uniformly distributed point sets, while the discriminator penalizes deviations from the expected target.
- 5. Conclusion: The up-down-up unit expands point features with error feedback and self-correction, while self-attention improves feature fusion.
- 5. Conclusion: A compound loss guides the learning of both the generator and discriminator.
- 5. Conclusion: Extensive experiments show that PU-GAN outperforms state-of-the-art methods across various metrics and supports upsampling of real-scanned LiDAR inputs.
- 5. Conclusion: PU-GAN has limited ability to fill large gaps or holes because it completes tiny holes at the patch level.The authors identify multi-scale training as a future direction for incorporating global structures.