Source-linked AI summary
PointCNN: Convolution On $\mathcal{X}$-Transformed Points
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, Baoquan Chen
TL;DR
Point clouds have spatially local structure but are irregular and unordered, so direct convolution can lose shape information and vary with point ordering. PointCNN learns an X-transformation that weights and permutes neighborhood features before applying typical convolution. It performs on par with or better than state-of-the-art point-cloud networks across multiple challenging datasets and tasks.
Problem
Point clouds are irregular and unordered, limiting direct convolution because it can discard shape information and remain sensitive to point ordering.
Method
PointCNN learns an X-transformation from local point coordinates to weight and permute input features before typical convolution.
Results
PointCNN performs on par with or better than state-of-the-art point-cloud networks across multiple challenging benchmark datasets and tasks.
Takeaways & Limitations
PointCNN generalizes CNN-style feature learning to point clouds by using X-Conv to process spatially local correlations.
Takeaways & Limitations
The K-nearest-neighbor implementation assumes a more or less uniform input-point distribution; non-uniform clouds require radius search followed by random point selection.
Abstract
from arXiv · showhide
We present a simple and general framework for feature learning from point clouds. The key to the success of CNNs is the convolution operator that is capable of leveraging spatially-local correlation in data represented densely in grids (e.g. images). However, point clouds are irregular and unordered, thus directly convolving kernels against features associated with the points, will result in desertion of shape information and variance to point ordering. To address these problems, we propose to learn an $\mathcal{X}$-transformation from the input points, to simultaneously promote two causes. The first is the weighting of the input features associated with the points, and the second is the permutation of the points into a latent and potentially canonical order. Element-wise product and sum operations of the typical convolution operator are subsequently applied on the $\mathcal{X}$-transformed features. The proposed method is a generalization of typical CNNs to feature learning from point clouds, thus we call it PointCNN. Experiments show that PointCNN achieves on par or better performance than state-of-the-art methods on multiple challenging benchmark datasets and tasks.
1 Introduction
Point clouds contain spatially local structure but are irregular and unordered, making direct convolution lose shape information and depend on point ordering. PointCNN addresses this with a learned X-transformation that weights and permutes local features before convolution.
- Motivation: Direct convolution is ill-suited to point clouds because their irregular, unordered representation complicates exploiting spatially-local correlations.Unlike regular grids, point-cloud neighborhoods lack a canonical ordering.
- Motivation: Direct convolution can discard shape information while remaining sensitive to the ordering of input points.The paper’s example contrasts equal outputs for differently shaped neighborhoods with unequal outputs caused by reordered points.
- Method: PointCNN learns a K × K X-transformation from the coordinates of K input points using a multilayer perceptron.The learned transformation is applied to local point neighborhoods rather than relying on a predefined grid order.
- Method: X-Conv simultaneously weights and permutes input features, after which a typical convolution operates on the transformed features.The transformation can adapt to different point shapes and learn permutations that recover ordering invariance.
- Results: With ideal X-transformations, X-Conv considers point shapes while remaining invariant to ordering, and PointCNN performs on par with or better than state-of-the-art point-cloud networks.The paper reports this pattern across multiple challenging benchmark datasets and tasks.
2 Related Work
Prior work extends feature learning to regular, irregular, and graph-like domains through voxelization, sparse structures, symmetric aggregation, projection, and learned kernels. PointCNN instead uses a latent regular domain formed by neighborhood-specific X-transformations.
- Regular Domains: CNNs exploit regular-grid structure effectively, but higher-dimensional voxel approaches can incur substantial computation and memory costs.Octrees, kd-trees, hashing, and sparse activations are described as ways to reduce computation or preserve sparsity.
- Irregular Domains: PointNet and Deep Sets pursue input-order invariance with symmetric functions, while PointNet++ and SO-Net apply PointNet hierarchically to capture local structures.Other methods improve PointNet-like models through kernel correlation and graph pooling.
- Irregular Domains: Some approaches project or interpolate point-cloud features into predefined regular domains before applying conventional CNNs.PointCNN differs by keeping the regular domain latent rather than predefined.
- Irregular Domains: Other point-cloud methods parameterize kernels individually for each point, whereas PointCNN learns X-transformations from each neighborhood.The paper suggests neighborhood-level transformations could be more adaptive to local structures.
- Graphs and Meshes: Graph and mesh representations support additional feature-learning methods, including spectral graph convolution on local graphs for point clouds.The related work also directs readers to a broader survey of these approaches.
- Invariance vs. Equivariance: PointCNN’s X-transformations are related to spatial transformer mechanisms and are ideally capable of realizing equivariance.This positions the method as a transformation-based alternative within work on invariance and equivariance.
3 PointCNN
PointCNN generalizes hierarchical CNN construction to point clouds through X-Conv, which aggregates local neighborhoods into fewer representative points with richer features. X-Conv uses local coordinates, lifted features, and a learned transformation to weight and permute neighborhood information before convolution.
- 3.1 Hierarchical Convolution: PointCNN generalizes hierarchical CNNs by recursively applying X-Conv to point neighborhoods, producing fewer representative points with richer features.The point-cloud hierarchy parallels grid CNNs while replacing grid patches with neighboring points around representative points.
- 3.2 X-Conv: X-Conv converts neighboring coordinates into local coordinate systems, lifts them with an MLP, and combines the lifted representations with associated point features.The local-coordinate construction makes the output depend on relative rather than absolute positions.
- 3.2 X-Conv: A learned K × K transformation weights and permutes the combined neighborhood features before a typical convolution produces the representative point feature.The transformation is learned from the neighborhood coordinates, and the full operator is differentiable for backpropagation.
- 3.3 PointCNN Architectures: PointCNN architectures reduce point counts while enriching features, then use receptive-field growth and fully connected layers for shape-level classification.With two X-Conv layers, the final representative point can aggregate information from all points in the preceding layer.
- 3.3 PointCNN Architectures: Dilated X-Conv keeps more representative points in upper layers while expanding receptive fields by sampling K points from K × D neighbors.The receptive field increases from K/N to (K × D)/N without increasing the actual neighboring point count or kernel size.
4 Experiments
PointCNN is evaluated across classification, segmentation, sketch, image, ablation, visualization, and efficiency experiments. It achieves top or competitive performance across several point-cloud benchmarks, while X-Conv improves feature concentration and is identified as central to performance.
- Classification and segmentation: PointCNN achieves top performance on both ModelNet40 and ScanNet classification benchmarks.
- Classification and segmentation: PointCNN outperforms all compared methods on ShapeNet Parts, S3DIS, and ScanNet segmentation.
- Sketch and image classification: PointCNN outperforms PointNet++ on TU-Berlin and Quick Draw sketch classification, but trails Sketch-a-Net on TU-Berlin.
- Sketch and image classification: PointCNN performs comparably to other methods on point-cloud MNIST, but has a large gap behind mainstream image CNNs on CIFAR10.
- Ablation experiments: PointCNN significantly outperforms variants without X-transformation, even when parameter count and model depth are controlled.
- Feature visualization: Feature-center classification accuracy rises from 76.83% for Fo and 89.29% for F∗ to 94.72% for FX, indicating stronger feature concentration after X-transformation.
5 Conclusion
PointCNN generalizes CNNs to exploit spatially local correlation in point clouds through the X-Conv operator. Its practical effectiveness is established, while rigorous understanding of deep compositions remains open.
- PointCNN generalizes CNNs for leveraging spatially local correlation in point-cloud data.
- X-Conv weights and permutes input points and features before applying a typical convolution.
- PointCNN is empirically effective in practice, but its rigorous behavior within deep neural networks remains an open problem.
- The paper identifies combining PointCNN with image CNNs for paired point clouds and images as future work.
1 X-Conv Details
The implementation applies X-transformation to features so the subsequent operation remains a standard optimized convolution. Architectural choices reduce parameters, computation, and overfitting risk, while neighborhood extraction assumes roughly uniform point density.
- The MLP for δ uses two fully connected layers with ELU and batch normalization, setting Cδ to C1/4.
- The matrix-producing MLP ends with a reshape to K × K, while replacing an internal fully connected layer avoids O(K4) parameters.
- Separable convolution replaces typical convolution to reduce trainable parameters and computation, using DM = ⌈C2/(C1 + Cδ)⌉.
- MLPg harvests global position information from representative points and concatenates its Cg-dimensional output with the final X-Conv features.
- K-nearest-neighbor extraction assumes a more or less uniform point distribution; radius search can precede random selection for non-uniform clouds.
- X-transformation is applied to features rather than kernels, preserving a standard convolution for the follow-up operation.
2 Dataset Details
The evaluation spans object, indoor-scene, sketch, and image datasets, covering classification, segmentation, semantic voxel labeling, and point-wise part prediction. Dataset preparation includes aligned and unaligned settings, sampled point clouds, and contextual scene blocks.
- The study evaluates PointCNN on datasets of various types and scales, with preprocessing and input-feeding details provided for each benchmark.
- Object datasets: ModelNet40 contains 12,311 mesh models across 40 categories with a 9,843/2,468 train/test split.
- Object datasets: ModelNet40 is evaluated in pre-aligned and unaligned settings, with 2,048 points sampled from each mesh before further sampling.
- Object datasets: ShapeNet Parts contains 16,880 models from 16 categories, with each point assigned one of 50 part labels for segmentation.
- Indoor scene datasets: S3DIS contains scans from 6 areas and 271 rooms, with RGB points labeled across 13 semantic categories for segmentation.
- Indoor scene datasets: ScanNet provides 1,513 indoor scenes for semantic voxel labeling and also supplies object instances for classification, including RGB point information.
- Sketch and image datasets: TU-Berlin and Quick Draw provide 2D sketch point clouds, while MNIST and CIFAR10 are converted from sampled pixels into point-cloud representations.
3 PointCNN Model Zoo
The PointCNN model zoo provides task- and dataset-specific architectures for classification and segmentation. Channel widths increase for more abstract features, while smaller neighborhood sizes reduce memory demands where representative-point counts are high.
- The model zoo lists PointCNN architectures for classification and segmentation across multiple benchmark datasets.
- Larger C values are used in layers carrying more abstract or semantic information, including top classification layers and middle Conv-DeConv segmentation layers.
- Smaller K values are used where representative-point counts are large to relax memory demands.
4 Detailed Segmentation Results
PointCNN achieves the best overall segmentation performance on both ShapeNet Parts and S3DIS, leading across multiple individual categories.
- PointCNN achieves the best overall performance on ShapeNet Parts and leads on 7 of 16 categories.
- PointCNN achieves the best overall performance on S3DIS and leads on 6 of 13 categories.