PaperlayerView exact revision on arXiv

Source-linked AI summary

PUFA-GAN: A Frequency-Aware Generative Adversarial Network for 3D Point Cloud Upsampling

Hao Liu, Hui Yuan, Junhui Hou, Raouf Hamzaoui, Wei Gao

arXiv:2203.00914v1cs.CVcs.MMeess.IV

TL;DR

Point-cloud upsampling seeks dense, evenly distributed points that remain consistent with the input surface, while existing deep-learning methods can leave noise, particularly in high-frequency regions. PUFA-GAN combines hierarchical feature units with a graph-filter-based, frequency-aware discriminator and identity distribution loss; the authors report better visual quality than state-of-the-art methods.

  • Problem

    Deep-learning upsampling methods focus on generating uniform high-resolution clouds but can leave noisy points that affect reconstruction, especially at corners.

    Source: §I · p. 1

  • Method

    PUFA-GAN uses hierarchical residual generator units, a graph-filter-based frequency-aware discriminator, and an identity distribution loss for point-cloud upsampling.

    Source: §I · p. 2, §V · p. 12

  • Results

    PUFA-GAN produces smoother point clouds attached to the underlying surface and clearer, finer-grained edges than the compared methods.

    Source: C. Comparison with the state-of-the-art · p. 9

  • Takeaways & Limitations

    The authors report that PUFA-GAN outperforms state-of-the-art methods and introduce HF CD and HF HD to evaluate regularity in high-frequency regions.

    Source: B. Evaluation metrics · p. 8, §V · p. 12

Abstract

from arXiv · show

We propose a generative adversarial network for point cloud upsampling, which can not only make the upsampled points evenly distributed on the underlying surface but also efficiently generate clean high frequency regions. The generator of our network includes a dynamic graph hierarchical residual aggregation unit and a hierarchical residual aggregation unit for point feature extraction and upsampling, respectively. The former extracts multiscale point-wise descriptive features, while the latter captures rich feature details with hierarchical residuals. To generate neat edges, our discriminator uses a graph filter to extract and retain high frequency points. The generated high resolution point cloud and corresponding high frequency points help the discriminator learn the global and high frequency properties of the point cloud. We also propose an identity distribution loss function to make sure that the upsampled points remain on the underlying surface of the input low resolution point cloud. To assess the regularity of the upsampled points in high frequency regions, we introduce two evaluation metrics. Objective and subjective results demonstrate that the visual quality of the upsampled point clouds generated by our method is better than that of the state-of-the-art methods.

I. INTRODUCTION

Point-cloud upsampling must produce dense, uniform points consistent with the input geometry while preserving sharp details. PUFA-GAN targets noise in high-frequency regions through specialized generator units, a frequency-aware discriminator, and an identity distribution loss.

I. INTRODUCTION

  • Existing deep-learning upsamplers prioritize uniformity but can leave noisy points that degrade reconstruction, especially at object corners.The paper frames upsampling as producing a uniform, dense, noise-free high-resolution cloud consistent with the low-resolution input geometry.Source: §I · p. 1
  • DGHRA extracts multiscale point-wise features, while HRA uses hierarchical residuals to capture details in upsampled features.The units serve feature extraction and expansion, respectively.Source: §I · pp. 1–2
  • A graph filter extracts high-frequency points for a frequency-aware discriminator, which learns to distinguish real from generated high-frequency points.The discriminator receives both the high-frequency points and the whole upsampled cloud to learn high-frequency and global features.Source: §I · pp. 1–2
  • An identity distribution loss constrains the upsampled cloud’s geometry distribution to match that of the input low-resolution cloud.Source: §I · p. 2
  • HF CD and HF HD are introduced to assess the regularity of upsampled points in high-frequency regions.Source: §I · p. 2

II. RELATED WORK

Prior upsampling approaches include optimization-based methods and deep-learning methods, with different constraints on geometry, local detail, and uniformity. The paper positions PUFA-GAN against these approaches and defines its input-output geometry and patch-based processing.

II. RELATED WORK

  • Optimization-based methods use input geometry priors and can upsample flat regions efficiently, but are limited on complex sharp regions such as corners and edges.Source: §II · p. 2
  • Deep-learning upsamplers include multiscale feature expansion, surface parameterization, graph convolutions, and cascaded refinement, with some approaches requiring normals or risking error accumulation.PU-Net uses multiscale features and shared MLPs; MPU progressively upsamples, while PU-Geo requires normal vectors.Source: §II · pp. 2–3
  • Self-Net emphasizes placing points near high-frequency regions, whereas PUFA-GAN distinguishes high-frequency points to limit noise without pushing points toward those regions.The paper notes that Self-Net’s objective may affect uniformity, especially for sparse clouds.Source: §II · p. 3
  • Compared with PU-GAN’s global discriminator, PUFA-GAN adds DGHRA-based feature extraction, HRA-based feature expansion, and a frequency-aware discriminator to address local detail and high-frequency noise.The paper also proposes an identity distribution loss.Source: §II · pp. 3–4

A. Overview

  • The task maps a non-uniform sparse input cloud to a uniform dense output that preserves its underlying surface, even distribution, and high-frequency details.PUFA-GAN processes patches, with a generator producing an upsampled patch and a discriminator distinguishing generated from ground-truth patches.Source: Figure 2 · p. 3, A. Overview · p. 4

B. Generator

The generator extracts point-wise features through densely connected feature blocks that combine dynamic graph construction with hierarchical residual aggregation. These units capture multiscale neighborhood information and refine features through gated residuals.

1) Feature Extraction:

  • Each feature extraction block concatenates one DGHRA with several HRA units to connect preceding multiscale features.The feature extraction module produces point-wise representative features from the input point cloud.Source: 1) Feature Extraction · p. 4
  • DGHRA dynamically constructs a multiscale feature graph from neighborhood information and captures hierarchical pure residuals for fine-grained representation.Its dynamic neighborhoods are formed by KNN search based on feature similarity.Source: 1) Feature Extraction · p. 4
  • HRA uses four residual blocks and a gate to fuse and recalibrate hierarchical residuals before adding them to identity features.A max-pooling operation then selects maximum point-wise feature responses across nearest neighbors; cascaded HRA units also appear in feature expansion.Source: 1) Feature Extraction · pp. 4–5

2) Feature Expansion:

Feature expansion duplicates input features, adds distinguishing 2D vectors, and refines them with cascaded HRA units before geometry reconstruction. A frequency-aware discriminator evaluates both the full upsampled cloud and graph-filtered high-frequency points.

2) Feature Expansion:

  • Feature expansion duplicates each input feature r times, appends a 2D vector to distinguish the copies, and uses cascaded HRA units to produce refined upsampled features.The 2D grid strategy is used to increase feature diversity.Source: 2) Feature Expansion · p. 5

3) Geometry Generation:

  • Geometry reconstruction maps expanded features to 3D coordinates, then applies farthest point sampling to select rN points from (r + 2)N candidates.The sampling strategy is used to ensure upsampled point-cloud uniformity.Source: 3) Geometry Generation · p. 5

C. Frequency-Aware Discriminator

  • A graph filter extracts high-frequency points from the upsampled cloud; global and high-frequency discriminator inputs support global-context and local-noise assessment, respectively.The global-only discriminator can miss local detail in dense regions, allowing noise to distract the generator when producing high-frequency points.Source: C. Frequency-Aware Discriminator · p. 5, Figure 5 · p. 5

1) HF Points Extraction Using Graph Filter:

The method uses a Haar-like high-pass graph filter to identify points with strong local variation. It ranks filtered point responses by magnitude and selects the highest-ranked points as high-frequency candidates.

1) HF Points Extraction Using Graph Filter:

  • The graph represents points as nodes and uses a symmetric adjacency matrix A to encode edge dependencies, with a Gaussian function and threshold defining the adjacency.Source: 1) HF Points Extraction Using Graph… · p. 6
  • The Haar-like filter H(A)=I−A suppresses low-frequency components and amplifies high-frequency components; graph filtering operates in the graph Fourier domain before inverse transformation.The filter response is H(λ_i)=1−λ_i.Source: 1) HF Points Extraction Using Graph… · p. 6
  • The L2 norm of each filtered point response measures local variation, and the top 256 points by descending norm are selected as high-frequency candidates.The selection count is set empirically.Source: 1) HF Points Extraction Using Graph… · p. 6

2) Discriminator structure:

The discriminator has global and high-frequency heads with matching backbones whose feature responses are combined to produce a confidence value.

2) Discriminator structure:

  • The global and high-frequency heads use the same backbone to process their respective inputs.Source: 2) Discriminator structure · p. 6
  • Each head applies two MLP layers and max pooling, concatenates a duplicated response for feature enhancement, then uses self-attention to strengthen point-feature interactions.Source: 2) Discriminator structure · p. 6
  • Two further MLP layers and max pooling produce head responses that are concatenated and passed through three fully connected layers for confidence prediction.The discriminator assumes confidence near 1 for ground truth and near 0 for generator output.Source: 2) Discriminator structure · p. 6

D. Loss Functions

PUFA-GAN trains its generator and discriminator jointly with adversarial, reconstruction, uniformity, and identity-distribution losses. These terms target realistic outputs, surface fit, even point spacing, and consistency with the low-resolution input.

D. Loss Functions

  • The least-squares adversarial loss trains the generator to confuse the discriminator and the discriminator to distinguish generated points from ground truth.The generator minimizes its prediction error toward 1 for generated points, while the discriminator learns to distinguish generated and target clouds.Source: D. Loss Functions · p. 7
  • Earth Mover’s distance encourages the upsampled cloud to lie close to the underlying surface of the ground-truth cloud.The paper uses an approximate sub-optimal bijection because computing the optimal one is expensive.Source: D. Loss Functions · p. 7
  • The uniformity loss combines global point-count balance across local subsets with local constraints on nearest-neighbor spacing.The global term equalizes subset counts, while the local term encourages similar nearest-neighbor distances.Source: D. Loss Functions · p. 7
  • The identity distribution loss maps the HR output back to the LR input, reducing the network’s solution space and helping it find an optimum quickly.FPS selects an LR-sized subset of the output, and a bijection relates it to the original input.Source: D. Loss Functions · p. 7
  • PUFA-GAN is trained using both the generator loss LG and discriminator loss LD in a joint loss; the named terms wgen, wrec, wuni, wid, and wdis are weights.Source: D. Loss Functions · p. 7

A. Experimental setup

Experiments use PU147 for training and testing, with patch-based training and a patch-fusion inference procedure. The setup also specifies the network configuration, training schedule, and test-time filtering.

A. Experimental setup

  • The PU147 dataset contains 147 models; 120 were used for training and the remainder for testing, yielding 24,000 randomly cropped training patches.The dataset draws on released models from PU-Net, MPU, and the Visionair repository.Source: A. Experimental setup · p. 7
  • The generator uses four feature-extraction blocks, each with one DGHRA and two HRA units, followed by three cascaded HRA units for feature expansion.The setup uses 16 nearest neighbors, feature dimension 128, and upsampled feature dimension 256.Source: A. Experimental setup · p. 7
  • Training ran for 130 epochs with batch size 28, Gaussian perturbation, random rotation and scaling, and separate generator and discriminator learning rates.The learning rates were 0.0009 for the generator and 0.0006 for the discriminator.Source: A. Experimental setup · p. 7
  • At inference, local patches generated around FPS-selected seeds are fused, then FPS selects the final cloud; a graph filter removes potential high-frequency noise before fusion.The authors say the filtering step can eliminate noise while retaining more detailed textures in high-frequency regions.Source: A. Experimental setup · p. 8

B. Evaluation metrics

Evaluation combines conventional geometric and uniformity metrics with two measures focused on high-frequency regions. The high-frequency metrics compare graph-filtered points from the output and ground truth.

B. Evaluation metrics

  • The evaluation uses Chamfer distance, Hausdorff distance, point-to-surface distance, and Uniformity to assess shape difference, surface deviation, and point distribution.For these metrics, the ground truth has 8,192 Poisson-disk-sampled points and the low-resolution input has 2,048 Monte Carlo-sampled points.Source: B. Evaluation metrics · pp. 8–9
  • HF CD and HF HD measure Chamfer and Hausdorff distances, respectively, between high-frequency points extracted from the output and ground truth using a graph filter.Both filtered point sets contain 2,048 points for these computations.Source: B. Evaluation metrics · pp. 8–9

C. Comparison with the state-of-the-art

PUFA-GAN leads or ranks second on the reported test-set metrics and produces visually cleaner edges and finer detail than comparison methods. Robustness tests report favorable outcomes on unseen, scanned, differently sized, and noisy point clouds, alongside higher training and inference times.

C. Comparison with the state-of-the-art

  • With upsampling ratio r=4, PUFA-GAN performed best in CD (0.258), HD (3.571), HF CD (2.081), and HF HD (19.744), and ranked second in P2F and Uniformity.The comparison included PU-Net, MPU, PU-GAN, PU-Geo, PU-GCN and Dis-PU.Source: C. Comparison with the state-of-the-art · p. 9
  • Compared with other methods, PUFA-GAN produces smoother surface-attached clouds, clearer edges, and finer texture; its high-frequency points are cleaner and closer to ground truth.The visual comparisons highlight a duck’s mouth, a cow’s hoof, a statue’s leg, and the m60 fish example.Source: C. Comparison with the state-of-the-art · pp. 9–10
  • PUFA-GAN required 28 h to train and 0.94 s for inference; the reported fastest times were 2 h for PU-GCN training and 0.13 s for PU-Net inference.The authors attribute PUFA-GAN’s higher time complexity mainly to graph-filter extraction of high-frequency points.Source: C. Comparison with the state-of-the-art · p. 10

D. Robustness Analysis

  • In the reported comparison, Monte-Carlo random sampling produced test inputs with 2,048 points, and PUFA-GAN achieved the best performance for all evaluation metrics compared with the other methods.Source: D. Robustness Analysis · p. 10
  • On real KITTI LiDAR scans, PUFA-GAN generated dense, uniform scenes with rich geometric detail despite sparse, noisy, and occluded inputs.The comparison is shown for zoomed-in car and wall views.Source: D. Robustness Analysis · p. 10, Figure 8 · p. 10
  • With input sizes of 256, 1,024, and 4,096 points, PUFA-GAN recovered approximate shapes even from 256-point inputs.The tests used ModelNet40 and the paper’s test datasets.Source: D. Robustness Analysis · p. 10, Figure 9 · p. 10
  • Under gradually increased Gaussian noise, PUFA-GAN preserved natural output shapes and avoided noise interference.The tests covered unseen ModelNet40 and test point clouds.Source: D. Robustness Analysis · p. 10

E. Ablation Study

PUFA-GAN outperforms state-of-the-art methods in extensive experiments, and its ablation study finds that removing any proposed component reduces performance. The conclusion describes the network’s components and identifies several directions for future work.

E. Ablation Study

  • The ablation study quantitatively and qualitatively evaluates five components: feature extraction, feature expansion, patch-fusion GF, frequency-aware discrimination, and identity distribution loss.Fig. 11 compares PUFA-GAN with versions omitting DGHRA, HRA, GF, the high-frequency discriminator, or IDL.Source: E. Ablation Study · p. 10, Figure 11 · p. 11
  • The ×4 noisy-cloud visualizations use Gaussian noise levels of 0, 0.5%, and 1%; the examples include an unseen ModelNet40 Guitar and a Sculpture from the test dataset.The ablation table is scoped to the PU147 dataset.Source: Figure 10 · p. 11, Table IV · p. 11
  • Removing any proposed component decreases performance; removing HRA causes the largest degradation, while the patch-fusion GF has the weakest influence.The authors attribute HRA’s impact to its role in feature detail and upsampling quality; they describe GF as a postprocessing step that does not affect the entire network performance.Source: E. Ablation Study · p. 11, Table IV · p. 11

V. CONCLUSION

  • Extensive experiments show that PUFA-GAN outperforms state-of-the-art methods.The conclusion also reports a generator with DGHRA-based feature extraction, HRA-based feature expansion, and geometry generation, plus global and high-frequency discriminator heads.Source: §V · pp. 11–12
  • Future work will explore arbitrary upsampling ratios, point attributes, point-cloud completion, and unsupervised or semisupervised learning to reduce reliance on paired, scarce training samples.The paper currently addresses geometry information only.Source: §V · p. 12
Loading 2203.00914v1…