PaperlayerView exact revision on arXiv

Source-linked AI summary

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas

arXiv:1706.02413v1cs.CV

TL;DR

PointNet does not capture metric-induced local structure in point sets, while non-uniform sampling complicates neighborhood-scale selection. PointNet++ recursively applies PointNet over nested metric-based regions and adaptively aggregates multi-scale features, achieving state-of-the-art performance on challenging 3D point-cloud benchmarks.

  • Problem

    PointNet’s global point-set processing does not capture metric-induced local structure, while non-uniform point densities complicate learning across neighborhood scales.

  • Method

    PointNet++ recursively applies PointNet to nested, overlapping metric-based regions and adaptively aggregates multi-scale features according to local point densities.

  • Results

    PointNet++ achieves state-of-the-art performance on challenging benchmarks of 3D point clouds and processes point sets efficiently and robustly.

  • Takeaways & Limitations

    Hierarchical metric-aware features and density-adaptive multi-scale aggregation make PointNet++ effective for point sets with non-uniform sampling.

  • Takeaways & Limitations

    The MSG and MRG layers remain computationally expensive, motivating future work to accelerate inference by sharing computation across local regions.

Abstract

from arXiv · show

Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

1 Introduction

PointNet++ addresses PointNet’s inability to capture metric-induced local structure by hierarchically learning features from overlapping neighborhoods. It further handles non-uniform point densities through adaptive multi-scale feature combination, improving robustness and detail capture.

  • Motivation: Point clouds must be permutation-invariant, while metric-defined local neighborhoods can have different properties such as density.These characteristics distinguish geometric point sets from regularly gridded data.
  • Motivation: PointNet encodes individual points and aggregates them globally, but does not capture local structure induced by the metric.The passage identifies local structure as important for convolutional architectures.
  • PointNet++: PointNet++ hierarchically partitions points into overlapping metric-based regions, extracts fine geometric features, and groups them into larger units for higher-level features.The architecture applies local processing at progressively larger contextual scales.
  • PointNet++: PointNet++ uses PointNet as its local feature learner, enabling shared processing of unordered local sets while retaining robustness to input corruption.Shared local learners require partitions to produce common structures across regions.
  • PointNet++: Farthest point sampling selects neighborhood centroids to evenly cover the point set, with receptive fields determined by both input data and the metric.Each partition is a neighborhood ball whose parameters include centroid location and scale.
  • Multi-scale learning: PointNet++ combines multi-scale neighborhoods and random input dropout to adaptively weight patterns across scales, improving robustness and detail capture for variable-density inputs.The passage reports efficient and robust point-set processing and results significantly better than state-of-the-art.

2 Problem Statement

The problem considers learning set functions on discrete point sets modeled as metric spaces inherited from Euclidean space, potentially with nonuniform point density. These functions take the point set and per-point features as input and produce semantically meaningful information.

  • Problem formulation: The input is a discrete metric space X = (M, d), where M ⊆ Rn and d is inherited from the ambient Euclidean space.The metric is defined over the set of points in Euclidean space.
  • Problem formulation: The point density of M may vary across the ambient Euclidean space.The formulation explicitly allows density to be nonuniform rather than assuming uniform sampling.
  • Problem formulation: The goal is to learn set functions f that map X, together with additional features for each point, to semantically relevant information.The passage gives classification functions as one practical example, though the example is truncated.

3 Method

PointNet++ extends PointNet with hierarchical grouping that progressively abstracts larger local regions, enabling local feature learning across contextual scales. It further adapts feature aggregation to non-uniform point densities through multi-scale and multi-resolution architectures.

  • Hierarchical feature learning: PointNet++ replaces PointNet’s whole-set max pooling with hierarchical grouping that progressively abstracts larger local regions.Each set abstraction level produces a smaller point set whose features summarize local context.
  • Set abstraction levels: Each abstraction level samples centroids, groups neighboring points, and applies PointNet to convert local regions into fixed-length feature vectors.The three layers are Sampling, Grouping, and PointNet; grouping supports flexible neighborhood sizes while producing fixed-length local features.
  • Sampling and grouping: Iterative farthest point sampling provides better coverage of the entire point set than random sampling given the same number of centroids.The method repeatedly selects the point farthest from previously selected centroids in metric distance.
  • Local feature learning: Relative coordinates within each local region, combined with point features, capture point-to-point relations for local pattern learning.Coordinates are translated into a local frame relative to each centroid before applying PointNet.
  • Density adaptive layers: PointNet++ addresses non-uniform density by combining features from multiple neighborhood scales according to local point densities.Multi-scale grouping applies PointNets at different scales and concatenates their features into a multi-scale representation.
  • Density adaptive layers: Multi-resolution grouping reduces the computational cost of multi-scale grouping while preserving adaptive aggregation based on point distribution.It concatenates features summarized from lower-level subregions with features obtained by direct grouping at the current level.

4 Experiments

PointNet++ is evaluated across 2D and 3D object datasets, real 3D scenes, and non-Euclidean shape classification. It outperforms PointNet and other baselines while demonstrating robustness to sampling-density variation and benefits from intrinsic features and geodesic neighborhoods.

  • Datasets and evaluation: Experiments cover MNIST, ModelNet40, SHREC15, and ScanNet, spanning 2D objects, 3D rigid and non-rigid shapes, and real 3D scenes.Object classification uses accuracy, while semantic scene labeling uses average voxel classification accuracy.
  • Object classification: PointNet++ achieves significantly lower error than PointNet on MNIST and is remarkably stronger than PointNet on ModelNet40 using the same 1024-point coordinate-only input.The paper reports relative MNIST error-rate reductions of 60.8% and 34.6% versus PointNet variants.
  • Robustness to sampling density variation: MSG+DP performance drops by less than 1% from 1024 to 256 test points, demonstrating robustness to sampling-density variation.MRG+DP is also described as very robust across sampling densities.
  • ScanNet scene labeling: On ScanNet semantic labeling, PointNet++ outperforms all baseline methods by a large margin by learning directly from point clouds and using data-dependent sampling.Hierarchical feature learning captures geometry at different scales for understanding scenes and labeling objects of various sizes.
  • Robustness to sampling density variation: For virtually scanned scenes, SSG performance greatly falls under density shift, whereas MRG is more robust because it switches to coarser features when sampling is sparse.The evaluation compares SSG, MSG+DP, MRG+DP, and a baseline approach.
  • Non-Euclidean shape classification: On SHREC15, PointNet++ with a non-Euclidean metric space and intrinsic features achieves the best performance in all settings and outperforms the prior method by a large margin.Intrinsic features are important for non-rigid classification, and geodesic neighborhoods outperform Euclidean neighborhoods.

5 Related Work

Prior deep-learning approaches either rely on representations or architectures that do not directly handle unordered point sets with metric-space structure, limiting local-context modeling. PointNet++ explicitly incorporates distances and learns point features under noisy, non-uniform sampling densities.

  • Point-set learning: Convolutional neural networks do not apply directly to unordered point sets with distance metrics.The paper identifies this mismatch as the motivation for its metric-space point-set design.
  • Point-set learning: Recent deep-learning methods for unordered sets ignore underlying distance metrics, preventing local-context capture and causing sensitivity to global translation and normalization.PointNet++ explicitly considers the underlying distance metric to address these issues.
  • Sampling density: Metric-space point samples are noisy and non-uniformly dense, making effective feature extraction and scale selection difficult.Earlier approaches addressed scale selection in geometry processing, photogrammetry, and remote sensing, whereas this approach learns to extract point features.
  • Alternative representations: Volumetric grids and geometric graphs are popular 3D deep-learning representations, but neither explicitly considers non-uniform sampling density.These representations are presented as alternatives to point-set processing in 3D metric space.

6 Conclusion

PointNet++ processes point sets in metric spaces by recursively learning hierarchical features over nested partitions. It also addresses non-uniform sampling through multi-scale aggregation, while motivating future work on faster inference and higher-dimensional applications.

  • Contributions: PointNet++ recursively partitions input point sets to learn hierarchical features with respect to the distance metric.The architecture is designed for point sets sampled in a metric space.
  • Contributions: Two set abstraction layers aggregate multi-scale information according to local point densities to address non-uniform sampling.The layers are proposed to handle varying point sampling density.
  • Future work: Future work includes accelerating inference for MSG and MRG layers through shared computation across local regions.The proposed direction targets especially the MSG and MRG layers.
  • Future work: Applications in higher-dimensional metric spaces are another future direction where CNN-based methods may be computationally infeasible.The passage suggests PointNet++ may scale well in such spaces.

Supplementary · A Overview · B Details in Experiments

The supplementary material expands the main-paper experiments with implementation details and additional validation, including benchmark results and analyses of neighborhood queries, sampling randomness, and complexity. It also defines the network-architecture notation used for these experiments, covering set abstraction, multi-scale processing, fully connected layers, and feature propagation.

  • Supplementary: The supplementary material provides additional experimental details and further experiments validating and analyzing the proposed method.These materials extend the experiments reported in the main paper.
  • A Overview: Section B specifies the network architectures, data preparation, and training procedures used in the main-paper experiments.Section C adds part-segmentation benchmark results and analyses of neighborhood queries, sampling randomness, and time-space complexity.
  • B Details in Experiments: SA(K,r,[l1, ..., ld]) denotes a set-abstraction level with K local regions of ball radius r and a d-layer PointNet of widths li.The notation describes local regions and the fully connected layers applied within each abstraction level.
  • B Details in Experiments: SA([l1, ...ld]) denotes a global set-abstraction level that converts a set into a single vector.This is the global counterpart to the local set-abstraction notation.
  • B Details in Experiments: The multi-scale notation represents MSG using m scales, with separate radii and fully connected-layer configurations for each scale.MSG is written using SA(K, [r(1), ..., r(m)], [[l(1) ...], ..., [l(m) ...]]).
  • B Details in Experiments: FC(l,dp) denotes a fully connected layer with width l and dropout ratio dp.This notation specifies the layer width and dropout setting.
  • B Details in Experiments: FP(l1, ..., ld) denotes a feature-propagation level with d fully connected layers that update features concatenated from interpolation and skip links.All fully connected layers use batch normalization and ReLU except the final score-prediction layer.

B.1 Network Architectures · B.2 Virtual Scan Generation

The paper specifies classification architectures for its Ours SSG, MSG, and MRG networks, alongside a ScanNet procedure for generating labeled virtual scans with non-uniform sampling density. The architectures use set-abstraction hierarchies, multi-scale or multi-resolution branches, and fully connected classification heads.

  • B.1 Network Architectures: Ours SSG uses three set-abstraction layers followed by fully connected layers FC(512, 0.5) →FC(256, 0.5) →FC(K).Its set-abstraction layers are SA(512, 0.2, [64, 64, 128]), SA(128, 0.4, [128, 128, 256]), and SA([256, 512, 1024]).
  • B.1 Network Architectures: The PointNet++ MSG architecture applies multi-scale grouping at 512 and 128 sampled points before shared set abstraction and classification layers.The grouping radii are [0.1, 0.2, 0.4] and [0.2, 0.4, 0.8], followed by SA([256, 512, 1024]) and the same fully connected head.
  • B.1 Network Architectures: Semantic scene labeling uses a network whose last two fully connected layers in FP are followed by dropout layers with drop ratio 0.5.The passage identifies this as the network for semantic scene labeling.
  • B.1 Network Architectures: Semantic and part segmentation use a network whose last two fully connected layers in FP are followed by dropout layers with drop ratio 0.5.The passage identifies this as the network for semantic and part segmentation.
  • B.2 Virtual Scan Generation: Virtual scans are generated from ScanNet scenes by placing the camera 1.5m above the floor-plane centroid and rotating it evenly in 8 horizontal directions.This setup produces labeled virtual scans with non-uniform sampling density.
  • B.2 Virtual Scan Generation: For each direction, rays pass from the camera through a 100px by 75px image plane to select visible scene points.The passage describes this ray-casting process as the basis for generating virtual scans.

B.3 MNIST and ModelNet40 Experiment Details

The experiments preprocess MNIST digits as 2D point clouds with thresholded, normalized pixels and fixed cardinality, while training uses Adam, augmentation, and GPU-implemented TensorFlow models.

  • MNIST preprocessing: MNIST pixels are normalized to [0, 1], with intensities above 0.5 retained as valid digit points.The resulting 2D point-cloud coordinates lie within [−1, 1], centered at the image origin.
  • MNIST preprocessing: Augmented points increase each MNIST point set to a fixed cardinality of 512.
  • Training setup: All experiments use Adam with learning rate 0.001, while augmentation randomly scales objects, perturbs object and point-sample locations, and rotates ModelNet40 objects.
  • Training setup: Models are implemented in CUDA with TensorFlow and trained on GTX 1080 or Titan X GPUs, requiring around 20 hours to converge.

B.4 ScanNet Experiment Details … C.2 Neighborhood Query: kNN v.s. Ball Query.

The paper reports implementation details for ScanNet and SHREC15, then presents additional experiments on part segmentation and neighborhood selection. These experiments specify data construction, feature processing, evaluation settings, and comparisons between ball-query and kNN neighborhoods.

  • B.4 ScanNet Experiment Details: ScanNet training data uses 1.5m × 1.5m × 3m scene cubes filtered by voxel occupancy and annotation validity.Cubes are retained when at least 2% of voxels are occupied and at least 70% of surface voxels have valid annotations.
  • B.4 ScanNet Experiment Details: ScanNet cubes are sampled on the fly, randomly rotated around the up-right axis, and augmented to a fixed 8192-point cardinality.The passage states that augmented points are added to make the point set have fixed cardinality during training-data generation.
  • B.5 SHREC15 Experiment Details: SHREC15 experiments sample 1024 points per shape and reduce 300-dimensional intrinsic features to 64 dimensions using PCA.The 300 dimensions comprise 100-dimensional WKS, HKS, and multiscale Gaussian-curvature features.
  • B.5 SHREC15 Experiment Details: An 8-dimensional embedding is used to mimic geodesic distance for describing the non-Euclidean metric space during neighborhood selection.The embedding follows.
  • C More Experiments: The additional-experiments section provides results intended to validate and analyze the proposed network architecture.The supplied passage introduces the section without reporting specific findings.
  • C.1 Semantic Part Segmentation: The semantic part-segmentation experiment uses 16,881 point-cloud shapes from 16 classes with 50 annotated parts and follows the official train-test split.Category labels are assumed known, and the task predicts a part label for every point.
  • C.1 Semantic Part Segmentation: For part segmentation, each point includes its normal direction, and mean point IoU across part classes is the evaluation metric.The framework is compared with traditional learning-based techniques and deep-learning approaches [20] [33], while training minimizes cross-entropy loss.
  • C.2 Neighborhood Query: kNN v.s. Ball Query.: On uniformly sampled ModelNet40 shapes with 1024 points, radius-based ball query performs slightly better than kNN neighborhood search.The experiment also varies the search radius and k; the passage notes a speculation about kNN behavior on highly non-uniform point sets but does not provide its conclusion.

C.3 Effect of Randomness in Farthest Point Sampling. · C.4 Time and Space Complexity.

PointNet++ remains stable under randomness in farthest point sampling, with low feature and classification variation on ModelNet40. Its efficiency depends on the architecture: the model is compact and reasonably fast, while multi-scale feature extraction increases cost and MRG improves efficiency relative to MSG.

  • C.3 Effect of Randomness in Farthest Point Sampling.: PointNet++ evaluates FPS randomness on ModelNet40 through repeated feature and classification stability tests.FPS subsampling depends on which point is selected first, so the model was tested across different random seeds.
  • C.3 Effect of Randomness in Farthest Point Sampling.: 0.021 feature-norm difference corresponds to a 2.1% deviation after averaging standard deviations across feature dimensions.Global features were extracted for every test sample 10 times with different random seeds, then compared with per-shape mean features.
  • C.3 Effect of Randomness in Farthest Point Sampling.: 0.17% standard deviation in test accuracy shows classification is robust to FPS sampling randomness on ModelNet40.The variation is measured across all ModelNet40 test shapes.
  • C.4 Time and Space Complexity.: PointNet++ without density-adaptive layers achieves the smallest model size with fair speed, although vanilla PointNet has the best time efficiency.Forward time was recorded with batch size 8 in TensorFlow 1.1 on a single GTX 1080, excluding the first batch.
  • C.4 Time and Space Complexity.: MSG costs 2x as much as SSG because it performs multi-scale region feature extraction.The higher cost accompanies MSG’s good performance on non-uniformly sampled data.
  • C.4 Time and Space Complexity.: MRG is more efficient than MSG because it uses regions across layers.This comparison concerns the relative efficiency of the two density-adaptive architectures.
Loading 1706.02413v1…