Source-linked AI summary
PU-Net: Point Cloud Upsampling Network
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, Pheng-Ann Heng
TL;DR
Sparse, irregular point clouds make it difficult to generate denser points that both follow underlying surfaces and remain uniformly distributed. PU-Net learns multi-level point features, expands them implicitly in feature space, and reconstructs an upsampled set; experiments show improved uniformity and smaller surface deviation.
Problem
Point cloud upsampling must generate denser points from sparse, unordered data while preserving surface geometry and avoiding clustered distributions.
Method
PU-Net learns multi-level features on point patches, expands point sets through a multi-branch feature-space convolution, and reconstructs the expanded features as points.
Results
PU-Net produces better distribution uniformity and lower deviation from underlying surfaces than the evaluated baselines and optimization-based comparison.
Takeaways & Limitations
The experiments support patch-level deep-network upsampling for generating denser, more uniform point sets without requiring an input-point order.
Takeaways & Limitations
PU-Net is trained using collected 3D objects as prior information, so its training scope depends on that object collection.
Abstract
from arXiv · showhide
Learning and analyzing 3D point clouds with deep networks is challenging due to the sparseness and irregularity of the data. In this paper, we present a data-driven point cloud upsampling technique. The key idea is to learn multi-level features per point and expand the point set via a multi-branch convolution unit implicitly in feature space. The expanded feature is then split to a multitude of features, which are then reconstructed to an upsampled point set. Our network is applied at a patch-level, with a joint loss function that encourages the upsampled points to remain on the underlying surface with a uniform distribution. We conduct various experiments using synthesis and scan data to evaluate our method and demonstrate its superiority over some baseline methods and an optimization-based method. Results show that our upsampled points have better uniformity and are located closer to the underlying surfaces.
1. Introduction
PU-Net addresses point-cloud upsampling by learning to generate denser points that follow an underlying surface and remain uniformly distributed despite unordered, irregular inputs. It introduces a patch-level, data-driven network and evaluates it against baseline and optimization-based methods.
- Problem: Point-cloud upsampling must handle unordered, irregular data while placing generated points on the surface and distributing them uniformly.These requirements make simple interpolation inadequate.
- Approach: PU-Net learns multi-level per-point features and expands the point set through a multi-branch convolution operation in feature space.Expanded features are split and reconstructed into the upsampled point set.
- Approach: The network operates on patches, learning geometry semantics from 3D models and applying them to upsample a given point cloud.Unlike earlier point-set networks, its input and output point counts differ.
- Evaluation: The evaluation uses distribution uniformity and distance deviation from underlying surfaces on synthetic and real-scanned data.The method is compared with baselines and optimization-based methods.
- Related work: Optimization-based methods may assume smooth underlying surfaces or depend heavily on accurate normals and parameter tuning.These constraints restrict their scope or affect result quality.
2. Network Architecture
PU-Net builds patch-level point features by combining hierarchical and multi-level information, then expands features to produce rN output coordinates. Its design uses interpolation, convolution, reshaping, and separate feature processing to support diverse reconstructed points.
- Patch extraction: PU-Net extracts multiscale patches from prior 3D models to learn local geometry patterns for upsampling.Patches are sampled at varying geodesic scales and use Poisson disk samples as reference distributions.
- Point feature embedding: Hierarchical feature learning captures growing spatial scales, using relatively small grouping radii because upsampling relies strongly on local context.The hierarchy is adapted from PointNet++ for point-cloud upsampling.
- Point feature embedding: Multi-level feature aggregation directly combines features from different levels so the network can learn their relative importance.The paper reports that top-down skip-connection propagation was inefficient for this upsampling problem.
- Point feature embedding: Interpolated features restore all original points, convolution reduces each level to dimension C, and concatenation forms the embedded feature.Interpolation uses inverse-distance weights from three nearest neighbors.
- Feature expansion: Feature expansion converts an N × ˜C representation into rN × ˜C2, treating expanded features as an implicit increase in point count.The operation addresses the difficulty of applying image-style upsampling to unordered point clouds.
- Feature expansion: Separate convolutions process the generated feature sets to reduce correlations that could place reconstructed points too close together.The separate weights allow the r feature sets to contain more diverse information.
- Coordinate reconstruction: Fully connected layers regress each expanded feature to 3D coordinates, producing rN × 3 upsampled point coordinates.This is the coordinate reconstruction stage.
3. End-to-End Network Training
PU-Net addresses ill-posed point-cloud upsampling by generating varied training inputs and optimizing a joint loss that balances surface reconstruction with point uniformity.
- 3.1. Training Data Generation: Randomly sampling training inputs from fixed ground-truth patches simulates multiple feasible outputs and enlarges the effective training dataset.The input points are resampled each epoch at downsampling rate r.
- 3.2. Joint Loss Function: The joint loss combines reconstruction and repulsion terms to place generated points on object surfaces while distributing them uniformly.This objective is trained end-to-end.
- 3.2. Joint Loss Function: Earth Mover’s distance compares predicted and referenced point clouds and is chosen over Chamfer Distance because it better captures shape.The reconstruction loss uses a bijection between predicted and ground-truth points.
- 3.2. Joint Loss Function: Reconstruction loss alone can cluster generated points near original points, so repulsion loss penalizes points that are too close to neighboring points.The repulsion term operates over k-nearest neighbors and uses a fast-decaying weight function.
- 3.2. Joint Loss Function: The network minimizes the joint objective end-to-end, with α balancing reconstruction and repulsion losses and β controlling weight decay.θ denotes the network parameters.
4. Experiments
Experiments evaluate PU-Net on collected, synthetic, and real-scanned data using surface deviation and distribution uniformity, comparing it with baselines, EAR, and design variants. PU-Net generally achieves more uniform outputs, lower surface deviation, stronger reconstruction results, and robustness to noisy and scanned inputs.
- Evaluation metrics: Surface deviation measures distances from predicted points to closest mesh points, while NUC measures normalized variation in point counts across equal-size surface disks.NUC uses D=9000 disks in the experiments and can assess local or global uniformity depending on disk-area percentage p.
- Comparisons with other methods: Compared with EAR, PU-Net better balances surface deviation and uniformity without requiring careful radius tuning.EAR produces nonuniform added points at small radius and more outliers at large radius; the comparison uses 1024 input points and 4X upsampling.
- Comparisons with other methods: PU-Net achieves the lowest surface deviation and best distribution uniformity among the deep-learning baselines on the collected and SHREC15 datasets, especially for local uniformity.PointNet performs worst, while PointNet++(MSG) slightly improves uniformity but requires more parameters and substantially longer training and testing.
- More experiments: Visual comparisons show PU-Net points better matching the underlying surface with lower deviations, and its reconstructed surfaces are closest to ground truth among the compared methods.Other methods can miss structures such as a Horse leg or overfill holes.
5. Conclusion
PU-Net generates denser, more uniform point sets through patch-level multi-level feature aggregation and context-aware upsampling. The authors report effectiveness while identifying completion and severe undersampling of tiny structures as limitations.
- 5. Conclusion: PU-Net generates denser and more uniform point sets from sparser inputs using patch-level multi-level feature aggregation.The network captures local and global information while bypassing any prescribed point order through features containing non-local geometry.
- 5. Conclusion: The method is not designed for completion and cannot fill large holes or missing parts.
- 5. Conclusion: Severely undersampled tiny structures may not receive meaningful added points.
- 5. Conclusion: Future work targets irregular and sparse data, including downsampling methods that preserve information in decimated point sets.
A. Overview
The supplementary material provides additional dataset details and describes the network architectures and baseline networks used in the experiments.
- A. Overview: The supplement first details the collected dataset and then presents the network architecture and experimental baseline networks.
B. Details of our Collected Dataset
The collected datasets contain 60 varied 3D models spanning smooth surfaces and sharp corners or edges, with the complete model names listed separately.
- B. Details of our Collected Dataset: The training and testing datasets contain 60 different 3D models.
- B. Details of our Collected Dataset: The collected models vary substantially in geometry, including smooth surface regions and sharp corners and edges.
- B. Details of our Collected Dataset: Table 4 provides the complete names of the 3D models in the training and testing datasets.
C. Details of Network Architectures
The supplement specifies PU-Net’s hierarchical feature-learning, feature-aggregation, expansion, and coordinate-reconstruction components, while separately documenting baseline architectures.
- C. Details of Network Architectures: The architecture details are organized around hierarchical feature learning, multi-level feature aggregation, feature expansion, and coordinate reconstruction.
- C. Details of Network Architectures: Hierarchical feature learning uses four levels with progressively larger ball radii and feature widths.The listed levels use (N, 0.05, [32, 32, 64]), (N/2, 0.1, [64, 64, 128]), (N/4, 0.2, [128, 128, 256]), and (N/8, 0.3, [256, 256, 512]).
- C. Details of Network Architectures: Multi-level aggregation restores each level’s features by interpolation and reduces them to 64 dimensions, yielding ˜C = 259.
- C. Details of Network Architectures: Feature expansion uses output channel sizes ˜C1 = 256 and ˜C2 = 128.
- C. Details of Network Architectures: Coordinate reconstruction uses two fully connected layers with 64 and 3 output channels.
- C. Details of Network Architectures: PointNet, PointNet++, and PointNet++(MSG) are documented as baseline architectures, with ReLU after layers except the final coordinate regression layer.