PaperlayerView exact revision on arXiv

Source-linked AI summary

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas

arXiv:1612.00593v2cs.CV

TL;DR

Point clouds are irregular, so converting them to grids or images can make them unnecessarily voluminous and introduce quantization artifacts. PointNet directly processes unordered point sets through a unified architecture and achieves faster, state-of-the-art-level performance across 3D recognition benchmarks.

  • Problem

    Converting irregular point clouds to voxel grids or images creates unnecessarily voluminous data and quantization artifacts that can obscure natural invariances.

  • Method

    PointNet directly consumes unordered point clouds using shared point processing and max pooling for classification, part segmentation, and scene semantic parsing.

  • Results

    PointNet is faster and achieves performance on par with or better than state-of-the-art approaches across shape classification, part segmentation, and scene segmentation benchmarks.

  • Takeaways & Limitations

    Theoretical analysis indicates that PointNet summarizes point clouds with sparse key points and remains robust to small perturbations, outliers, and missing data.

  • Takeaways & Limitations

    PointNet must respect permutation invariance in unordered point sets, while invariance to rigid motions also requires consideration.

Abstract

from arXiv · show

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds and well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

1. Introduction

PointNet directly processes unordered 3D point clouds with a unified architecture for classification, part segmentation, and scene semantic parsing. Its symmetric max-pooling design supports permutation invariance while enabling efficient, robust learning and theoretical analysis.

  • Motivation: PointNet consumes point clouds directly instead of converting them to regular 3D voxel grids or image collections.Point clouds avoid the combinatorial irregularities and complexities of meshes and provide a simpler input representation.
  • Architecture: The architecture respects permutation invariance because point clouds are sets whose point ordering should not affect the computation.Rigid-motion invariance is also considered, and spatial transformer networks can canonicalize transformed inputs.
  • Architecture: Each point is initially processed identically and independently, using its three coordinates or additional dimensions such as normals.The network outputs either a class label for the whole input or per-point segment and part labels.
  • Architecture: A single symmetric max-pooling function selects informative points and aggregates learned values into a global shape descriptor or per-point predictions.The network learns optimization criteria that identify interesting or informative points in the cloud.
  • Analysis: PointNet can approximate any continuous set function and summarizes point clouds with a sparse set of key points roughly corresponding to object skeletons.The theoretical analysis also explains robustness to small perturbations of input points.
  • Evaluation and Contributions: Experiments cover shape classification, part segmentation, and scene segmentation, finding PointNet much faster and competitive with or better than state-of-the-art multi-view and volumetric approaches.The paper also provides empirical and theoretical analysis of stability and efficiency and visualizes learned 3D features.

2. Related Work

Prior 3D learning methods use handcrafted features, volumetric representations, or multiview images, each creating task-specific limitations. Point clouds are also unordered sets, but prior set-processing work addressed generic sets without incorporating geometric structure.

  • Point Cloud Features: Handcrafted point-cloud features encode statistical properties and target intrinsic or extrinsic transformation invariance, but selecting the optimal task-specific combination is nontrivial.Features are also categorized as local or global.
  • Deep Learning on 3D Data: Volumetric CNNs operate on voxelized shapes, but resolution limits, data sparsity, and 3D convolution cost constrain their scalability.FPNN and Vote3D address sparsity, yet still operate on sparse volumes and struggle with very large point clouds.
  • Deep Learning on 3D Data: Multiview CNNs render point clouds or shapes into 2D images and achieve dominating shape-classification and retrieval performance, but are difficult to extend to scene understanding and other 3D tasks.The cited limitations include point classification and shape-related tasks.
  • Deep Learning on Unordered Sets: Point clouds are unordered sets of vectors, whereas most deep-learning methods target regular sequences, images, or volumes, leaving point-set learning relatively underexplored.A read-process-write attention network can consume unordered sets and sort numbers, but its generic-set and NLP focus omits geometry.

3. Problem Statement

PointNet directly consumes unordered 3D point sets, representing each point by coordinates and optionally additional feature channels. It supports object classification with class-level scores and semantic segmentation with per-point category scores.

  • Input Representation: Point clouds are represented as unordered sets of 3D points, each containing (x, y, z) coordinates and optionally channels such as color or normals.Unless otherwise noted, the framework uses only (x, y, z) coordinates.
  • Object Classification: For object classification, PointNet processes point clouds sampled from shapes or pre-segmented from scenes and outputs k scores for k candidate classes.The input may be a complete sampled shape or an extracted object from a scene point cloud.
  • Semantic Segmentation: For semantic segmentation, the model processes either an object or a 3D-scene sub-volume and outputs n × m scores for n points across m semantic subcategories.This formulation covers part-region segmentation and object-region segmentation.

4. Deep Learning on Point Sets

PointNet is designed around point sets’ unordered structure, local interactions, and transformation invariance. Its max-pooling architecture combines global and local information, aligns inputs and features, and admits theoretical approximation and robustness analyses.

  • Point Set Properties: Point clouds are unordered, so PointNet must remain invariant to all N! permutations of the input points.The paper contrasts point sets with ordered pixel or voxel arrays.
  • Joint Alignment Network: PointNet includes joint alignment networks for input points and point features, with an orthogonalizing regularization term that stabilizes optimization and improves performance.The alignment goal is invariance of learned representations under transformations such as rigid transformations.
  • Symmetry Function for Unordered Input: PointNet uses a symmetric aggregation function, approximating per-point transformations with a multilayer perceptron followed by a single-variable function and max pooling.The formulation applies a symmetric function to transformed set elements, while multiple learned functions capture different set properties.
  • Local and Global Information Aggregation: For segmentation, PointNet concatenates the global point-cloud feature with each point feature, producing per-point representations aware of local geometry and global semantics.This design supports per-point quantities such as normals and is reported to achieve state-of-the-art shape part and scene segmentation.
  • Theoretical Analysis: Theorem 1 states that any continuous set function under Hausdorff distance can be arbitrarily approximated by the network with sufficiently large max-pooling dimension K.The theorem uses a continuous function h and a symmetric function g formed by a continuous function composed with vector MAX.
  • Bottleneck Dimension and Stability: The max-pooling bottleneck summarizes shapes through sparse key points, explaining robustness to point perturbations, corruption, and extra noise points.The paper relates this robustness to a sparsity principle and reports that key points form an object’s skeleton.

5. Experiment

Experiments show PointNet performs competitively across 3D object classification, part segmentation, and semantic scene segmentation while remaining robust to missing or corrupted points. The network also offers strong computational and memory efficiency, with linear scaling in the number of input points.

  • 3D Object Part Segmentation: PointNet achieves a 2.3% mean IoU improvement on ShapeNet part segmentation and beats baseline methods in most categories.The evaluation metric is mIoU on points, comparing PointNet with two traditional methods and a 3D CNN baseline.
  • 3D Object Part Segmentation: On simulated Kinect scans, PointNet loses only 5.3% mean IoU when evaluated on incomplete point clouds generated from six random viewpoints.The model is trained on complete shapes and partial scans using the same architecture and training setting.
  • Semantic Segmentation in Scenes: For semantic scene segmentation, PointNet significantly outperforms a handcrafted-feature MLP baseline and produces smooth predictions robust to missing points and occlusions.The Stanford dataset contains scans from 6 areas, including 271 rooms, with 13 semantic categories.
  • Robustness Analysis: When 50% of points are missing, accuracy drops only 2.4% with furthest sampling and 3.8% with random sampling; with 20% outliers, accuracy remains above 80%.Robustness to outliers holds when the network has seen outliers during training.
  • Efficiency and Scalability: PointNet is 141x more FLOPs-efficient than MVCNN and 8x more efficient than Subvolume, uses 17x fewer parameters than MVCNN, and scales as O(N).It processes more than one million points per second for classification or semantic segmentation on a 1080X GPU with TensorFlow.

6. Conclusion

PointNet directly consumes point clouds and unifies several 3D recognition tasks, achieving results on par with or better than state-of-the-art methods while providing theoretical analysis and visualizations of the network.

  • 6. Conclusion: PointNet directly consumes point clouds and unifies object classification, part segmentation, and semantic segmentation in one approach.The network is presented as a unified solution for multiple 3D recognition tasks.
  • 6. Conclusion: PointNet obtains on-par or better results than state-of-the-art methods on standard benchmarks.The conclusion reports comparative performance across the described 3D recognition tasks.
  • 6. Conclusion: The work provides theoretical analysis and visualizations to support understanding of the PointNet network.These analyses accompany the proposed architecture and its benchmark results.

Supplementary · A. Overview · B. Comparison between PointNet and VoxNet (Sec 5.2)

The supplementary material adds quantitative results, technical details, qualitative examples, and further analyses. It also compares PointNet and VoxNet under missing-input conditions, where PointNet is substantially more robust.

  • Supplementary: The supplementary document provides additional quantitative results, technical details, and qualitative test examples beyond the main paper.It also includes further architecture, training, detection, application, analysis, theory, and visualization material.
  • A. Overview · B. Comparison between PointNet and VoxNet (Sec 5.2): Section B extends the Sec 5.2 robustness test by comparing PointNet with VoxNet on incomplete point-cloud inputs.VoxNet is described as a representative volumetric-representation architecture.
  • B. Comparison between PointNet and VoxNet (Sec 5.2): Both networks use the same train-test split and receive 1024 input points.This establishes a shared experimental setting for the robustness comparison.
  • B. Comparison between PointNet and VoxNet (Sec 5.2): VoxNet voxelizes point clouds into 32 × 32 × 32 occupancy grids and uses rotation and jittering augmentation.The rotations are random and performed around the up-axis.
  • B. Comparison between PointNet and VoxNet (Sec 5.2): At test time, points are randomly dropped by a specified ratio to evaluate robustness to missing data.Because VoxNet is sensitive to rotations, its prediction averages scores from 12 point-cloud viewpoints.
  • B. Comparison between PointNet and VoxNet (Sec 5.2): 86.3% to 46.0%: VoxNet’s accuracy drops by 40.3% when half the input points are missing.The passage contrasts this with PointNet’s 3.7% performance drop under the same missing-point test.
  • B. Comparison between PointNet and VoxNet (Sec 5.2): 3.7%: PointNet’s performance drop is much smaller than VoxNet’s when half of the input points are missing.The passage states that this result is explained by the paper’s theoretical analysis, though the provided text is truncated.

C. Network Architecture and Training Details (Sec 5.1) · D. Details on Detection Pipeline (Sec 5.1)

The paper specifies PointNet’s alignment networks, optimization settings, and segmentation extensions, then builds detection from semantic segmentation and classification outputs. The detection pipeline combines connected components with category-specific sliding-window classification when nearby objects are difficult to separate.

  • C. Network Architecture and Training Details (Sec 5.1): The classification PointNet uses a mini-PointNet to regress a 3 × 3 input transformation matrix from raw points.It applies shared MLP layers of 64, 128, and 1024 units, max pooling, and fully connected layers of 512 and 256 units; the matrix starts as identity.
  • C. Network Architecture and Training Details (Sec 5.1): A second transformation network mirrors the first but outputs a 64 × 64 feature transformation matrix, with an orthogonality regularizer weighted 0.001.The matrix is identity-initialized, and the regularization loss is added to the softmax classification loss.
  • C. Network Architecture and Training Details (Sec 5.1): Classification training uses Adam with learning rate 0.001, momentum 0.9, batch size 32, and learning-rate halving every 20 epochs.The model uses 0.7 dropout keep ratio on the final 256-dimensional fully connected layer; ModelNet training converges in 3–6 hours on TensorFlow with a GTX1080 GPU.
  • C. Network Architecture and Training Details (Sec 5.1): The segmentation network concatenates each point’s local feature with the global max-pooled feature and uses the classification network’s training parameters without dropout.Shape part segmentation additionally supplies a one-hot class vector, increases neurons in some layers, and uses skip links to combine local features from different layers.
  • C. Network Architecture and Training Details (Sec 5.1): For part segmentation, PointNet trains across categories using a one-hot category indicator, while evaluation predicts labels only for the specified object category.The semantic segmentation task uses the architecture from the main paper; training takes around 6–12 hours on ShapeNet part and around half a day on Stanford semantic parsing.
  • C. Network Architecture and Training Details (Sec 5.1): The volumetric CNN baseline converts each point cloud to a 32 × 32 × 32 occupancy grid and applies five 3D convolution operations with 32 output channels.Its receptive field is 19 per voxel, followed by 1 × 1 × 1 convolutions for voxel-wise segmentation; it trains across categories but scores only the given category during comparison.
  • D. Details on Detection Pipeline (Sec 5.1): The detection system combines semantic segmentation results with the object-classification PointNet to generate scene object proposals.Connected components use BFS from random scene points, grouping nearby points with the same predicted label within a 0.2 meter search radius; clusters exceeding 200 points yield bounding-box proposals.
  • D. Details on Detection Pipeline (Sec 5.1): For crowded chair scenes, category-specific binary classifiers perform sliding-window detection, after which non-maximum suppression prunes the resulting boxes.Object-detection precision-recall curves aggregate six models, each trained on five areas and tested on the unseen sixth area.

E. More Applications (Sec 5.1)

PointNet’s global shape signatures support retrieval of geometrically similar point clouds, while its learned point features can also be used to compute correspondences between similar shapes.

  • Model Retrieval from Point Cloud: PointNet computes a global signature for each query point cloud and retrieves geometrically similar shapes from the ModelNet training split.The signature is taken from the layer before score prediction in the classification network.
  • Object Detection: 3D point-cloud object detection is evaluated across all six areas for table, chair, sofa, and board categories at an IoU threshold of 0.5 in volume.The evaluation is summarized with precision-recall curves.
  • Model Retrieval from Point Cloud: Top-5 retrieval examples are shown for chair, plant, nightstand, and bathtub queries, with wrong-category results marked in red.The examples use point clouds from the ModelNet test split.
  • Shape Correspondence: PointNet’s learned point features can compute shape correspondences by matching points that activate the same dimensions in the global features.The method computes correspondences between the critical point sets of two shapes.
  • Shape Correspondence: Correspondences are demonstrated between pairs of similar chairs and tables.The detections are presented in Figures 13 and 14.

F. More Architecture Analysis (Sec 5.2)

The analysis shows that PointNet benefits from more input points and a larger bottleneck, while retaining reasonable performance on 2D point sets. Additional experiments demonstrate local normal reconstruction, segmentation robustness, and generalization to unseen shape categories.

  • Effects of Bottleneck Dimension and Number of Input Points: Performance saturates at around 1K input points, while increasing the max-layer size from 64 to 1024 yields a 2−4% performance gain.The results indicate that sufficient point feature functions are needed to cover 3D space and discriminate different shapes.
  • Effects of Bottleneck Dimension and Number of Input Points: Even with 64 input points obtained by furthest point sampling on meshes, PointNet achieves decent performance.
  • MNIST Digit Classification: On MNIST represented as 2D point sets, PointNet achieves reasonable performance, although well-engineered CNNs remain best with less than 0.3% error rate.The comparison includes an ordered-vector MLP, a pixel-sequence RNN, and a vanilla CNN.
  • Normal Estimation: A modified segmentation PointNet produces reasonable point-normal reconstructions, with predictions smoother and more continuous than mesh-derived ground truth containing flipped directions.The model regresses per-point normals using absolute cosine distance as the loss.
  • Segmentation Robustness: Segmentation robustness is tested by comparing results for full input clouds, critical point sets, and upper-bound shapes.Part labels combine per-point features with the learned global shape feature.
  • Network Generalizability to Unseen Shape Categories: Critical point sets and upper-bound shapes for unseen face, house, rabbit, and teapot categories show that learned per-point functions generalize.Because training emphasizes man-made objects with planar structures, reconstructed upper-bound shapes in novel categories also contain more planar surfaces.

G. Proof of Theorem (Sec 4.3)

The section proves that continuous set functions can be approximated by composing a continuous function with a symmetric MAX-based representation. It also establishes that this representation yields critical-point and upper-bound sets preserving the network output under certain point-set changes.

  • Theorem 1: Any continuous set function under Hausdorff distance can be approximated arbitrarily by a continuous function composed with a symmetric MAX-based function.The symmetric function has the form g(x1, . . . , xn) = γ◦MAX, with continuous γ and element-wise vector MAX.
  • Theorem 1: The proof constructs a finite interval occupancy representation, then maps its binary occupancy vectors through a continuous function γ matching f on the corresponding discretized sets.The construction uses K = ⌈1/δϵ⌉ intervals and MAX aggregation of soft interval indicators.
  • Theorem 1: The resulting γ◦MAX representation is symmetric, so its output does not depend on the ordering of the input points.The proof states directly that γ ◦MAX is a symmetric function.
  • Theorem 2: For every input set S, there exist critical-point and upper-bound sets CS and NS such that f(T) = f(S) whenever CS ⊆T ⊆NS.The critical set is formed from points attaining the MAX value in each dimension; adding points whose h values remain below u(S) does not change the representation or output.

H. More Visualizations

The visualizations show that PointNet learns semantically organized global shape features, detects spatially distinct point patterns, and produces segmentation and scene-parsing predictions across complete, partial, and unseen environments.

  • Classification Visualization: t-SNE embeddings of 1024-dimensional global signatures cluster similar ModelNet40 test shapes by semantic category.The embedding is computed from the classification PointNet’s global signatures.
  • Segmentation Visualization: Additional segmentation visualizations cover complete CAD models and simulated Kinect partial scans, including failure cases and error analysis.Figure 23 summarizes six common segmentation error types.
  • Scene Semantic Parsing Visualization: Scene semantic-parsing examples show input, predictions, and ground truth for semantic segmentation and object detection in two office rooms and one conference room unseen during training.The visualizations include both point-level semantic segmentation and object detection.
  • Point Function Visualization: The classification network uses K = 1024 per-point features and max pooling to form a single global shape descriptor.Each point receives K-dimensional features before aggregation.
Loading 1612.00593v2…