Source-linked AI summary
SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, Yu Qiao
TL;DR
CNNs are difficult to apply directly to irregular 3D point clouds, which lack regular-grid structure. SpiderCNN addresses this with SpiderConv, whose parameterized filters combine step functions and Taylor expansions; it achieves state-of-the-art ModelNet40 classification and competitive segmentation performance.
Problem
Irregular 3D point clouds lack canonical order and translation invariance, making direct use of conventional CNNs challenging.
Method
SpiderCNN discretizes convolution on point sets and uses SpiderConv layers with parameterized filters combining step functions for local geodesic information and order-3 Taylor expansions for expressive geometric modeling.
Results
SpiderCNN achieves state-of-the-art classification performance on ModelNet40 and competitive performance on ShapeNet-Part segmentation.
Takeaways & Limitations
SpiderCNN provides a convolutional neural network that directly processes 3D point clouds while extracting local geometric features through parameterized filters.
Abstract
from arXiv · showhide
Deep neural networks have enjoyed remarkable success for various vision tasks, however it remains challenging to apply CNNs to domains lacking a regular underlying structures such as 3D point clouds. Towards this we propose a novel convolutional architecture, termed SpiderCNN, to efficiently extract geometric features from point clouds. SpiderCNN is comprised of units called SpiderConv, which extend convolutional operations from regular grids to irregular point sets that can be embedded in R^n, by parametrizing a family of convolutional filters. We design the filter as a product of a simple step function that captures local geodesic information and a Taylor polynomial that ensures the expressiveness. SpiderCNN inherits the multi-scale hierarchical architecture from classical CNNs, which allows it to extract semantic deep features. Experiments on ModelNet40 demonstrate that SpiderCNN achieves state-of-the-art accuracy 92.4% on standard benchmarks, and shows competitive performance on segmentation task.
1 Introduction
SpiderCNN addresses the difficulty of applying CNNs directly to irregular 3D point clouds by introducing SpiderConv, a parameterized convolutional layer designed for point sets. Its filters combine coarse local-geodesic structure with expressive Taylor expansions, and the resulting network performs strongly on classification and segmentation benchmarks.
- Architecture: SpiderCNN uses a multi-scale hierarchical architecture to learn deep semantic features from irregular point sets.The architecture extends convolutional learning beyond regular grids while retaining the hierarchical pattern associated with classical CNNs.
- Motivation: 3D point clouds lack canonical order and translation invariance, making direct application of conventional CNNs difficult.Voxelization offers one workaround, but is computationally inefficient for sparse point clouds representing usually 2D surfaces.
- Approach: SpiderCNN directly extracts features from point clouds by discretizing the convolution integral and selecting filters from a parameterized family on R3.This design introduces SpiderConv as the network's point-cloud convolutional layer.
- Filter Design: SpiderConv filters combine step functions for local geodesic information with order-3 Taylor expansions for intricate local geometric variations.The step functions also remain relatively feasible to optimize using SGD.
- Evaluation: SpiderCNN achieves state-of-the-art classification performance on ModelNet40 and competitive segmentation performance on ShapeNet-Part.The introduction reports these outcomes for a relatively simple network architecture.
2 Related Work
Prior work applies deep networks to point sets, voxelized clouds, and non-Euclidean domains through different strategies for handling unordered inputs and defining convolution. SpiderCNN instead operates directly in the ambient Euclidean space, avoiding voxelization and local manifold parametrization while using data augmentation for rigid-transformation robustness.
- Point Clouds as Input: PointNet directly processes unordered point sets with a shared MLP and symmetric max-pooling, but its design limits local-structure encoding.PointNet++ and other methods add grouping strategies to mimic hierarchical local-feature learning.
- Voxels as Input: Voxel-based methods apply 3D convolutions but incur high computational and memory costs, motivating sparse voxel representations.SpiderCNN instead accepts point clouds directly and can handle very sparse input.
- SpiderCNN Positioning: SpiderCNN's output is permutation-invariant by design, and data augmentation helps its representations remain robust to spatial rigid transformations.These properties address unordered point inputs and spatial transformations without requiring a canonical spectral domain.
- Convolution on Non-Euclidean Domains: Spatial non-Euclidean methods define local patches through manifold coordinates, whereas SpiderCNN designs filters in the ambient Euclidean space Rn.The paper presents this as an alternative intended to be efficient for point-cloud processing.
- Spectral Methods: Spectral methods rely on Laplacian eigenfunctions or graph spectra, but different graphs or shapes lack canonically identifiable spectral domains.SyncSpecCNN addresses this issue through spectral-domain alignment using functional maps.
3 SpiderConv
SpiderConv extends convolution from regular grids to point sets by using localized, parametrized filters. Its filter family combines coarse geometric localization with expressive Taylor terms and is implemented with K-neighbor approximations.
- Convolution on point sets: SpiderConv is the fundamental building block of SpiderCNN for defining convolutional layers on point-set features in R^n.
- Convolution on point sets: SpiderConv generalizes classical convolution by applying a filter supported in a radius-r ball around each point.On a regular grid, the formulation reduces to classical convolution.
- A special family of filters: The step component captures coarse local geometry through local geodesic distance, while the Taylor component captures richer angular and spatial variation.Taylor terms can interpolate arbitrary values at cube vertices and capture rich spatial information.
- Implementation: SpiderCNN uses K-nearest neighbors instead of radius neighborhoods and approximates the step-function operation with a permutation-based matrix product.The approximations exploit the uniform sampling process used to construct point clouds.
- Implementation: SpiderConv maps c1 input channels on a point cloud to c2 output channels using ordered neighbors and multiple Taylor terms.For each point, neighbors are ordered and Taylor filters are indexed by term, input channel, and output channel.
4 Experiments
Experiments evaluate SpiderCNN on classification, retrieval, and segmentation benchmarks, while probing architecture and pooling choices. The results show strong classification performance, competitive segmentation, and benefits from geometric feature extraction and top-2 pooling.
- Hyperparameters: 20 nearest neighbors is optimal among 12, 16, 20, and 24 choices for SpiderConv in the tested 3-layer ModelNet40 model.The number of nearest neighbors functions analogously to filter size in standard convolution.
- Pooling: Top-2 pooling preserves representative critical points, including edge points and points with non-zero curvature.The visualization compares original point clouds with points contributing to top-2 pooled features.
- Feature learning comparison: SpiderCNN’s FPFH prediction training loss is one-fourth of PointNet’s in the reported comparison.The authors interpret the models as complementary because SpiderCNN captures local geometry while PointNet captures global features.
- ModelNet40 classification: A 4-layer SpiderCNN achieves 92.4% accuracy on ModelNet40, improving over the best reported result among models using 1,024 points and normals.Across five runs, its mean accuracy is 92.0%.
- Feature complementarity: Combining 128-dimensional SpiderCNN and PointNet features improves classification accuracy to 92.2%.The authors describe SpiderCNN as learning local geometric features and PointNet as capturing global features.
- Pooling: Top-2 pooling yields 92.4% accuracy for a 4-layer SpiderCNN and 91.5% for a 3-layer model, compared with 92.0% and 90.4% using max-pooling.The reported comparison links richer geometric information with preservation of more non-zero-curvature points.
- SHREC15 classification: On SHREC15, SpiderCNN outperforms the other methods in the reported classification comparison.SHREC15 contains more complicated local geometry and non-rigid deformation than ModelNet40.
- ShapeNet Part segmentation: SpiderCNN achieves 85.24 mean mean IoU over four runs on ShapeNet Part with a relatively simple 4-layer architecture.The model uses points and normal vectors as input while assuming category labels are known.
5 Analysis
SpiderCNN remains effective with substantially fewer input points, and its learned filters exhibit geometric patterns analogous to 2D image filters.
- Robustness: 87.7% accuracy is achieved by SpiderCNN with only 32 input points on ModelNet40.The robustness experiment compares inputs of 512, 248, 128, 64, and 32 points, together with normals.
- Robustness: The analysis evaluates robustness to missing points by training SpiderCNN and PointNet++ with progressively smaller point sets.The tested input sizes are 512, 248, 128, 64, and 32 points.
- Visualization: The first-layer filters are visualized as scatter plots whose point colors represent filter values.The visualization concerns the convolutional filters g_w(x, y, z) learned in SpiderCNN's first layer.
- Visualization: Projecting learned 3D filters onto planes reveals patterns resembling 2D image filters and provides hints about learned geometric features.The first row corresponds to 2D image filters that can capture boundary information.
6 Conclusions
The paper concludes by proposing SpiderCNN, a convolutional network that directly processes 3D point clouds with parameterized filters. It identifies more complex architectures and applications as future directions.
- SpiderCNN directly processes 3D point clouds using parameterized convolutional filters.The conclusion presents SpiderCNN as a new convolutional neural network for this setting.
- More complex network architectures and additional applications of SpiderCNN remain open directions.
7 Appendix
The appendix examines filter choices and a hybrid architecture through additional ModelNet40 experiments. It compares Taylor expansions with MLP and alternative polynomial parameterizations, then evaluates combining SpiderCNN with PointNet.
- Filter analysis: The SpiderCNN filter decomposes into a step function multiplied by a Taylor component.The decomposition is written as g(x, y, z) = gStep(x, y, z) · gTaylor(x, y, z).
- Filter analysis: Taylor outperforms MLP for ModelNet40 classification even when MLP uses more parameters.The comparison is conducted in a 4-layer SpiderCNN.
- Filter analysis: SpiderCNN uses an order-3 Taylor expansion, with trilinear interpolation yielding a simpler polynomial expansion.The listed expansion includes linear, pairwise interaction, and xyz terms.
- Filter analysis: The appendix evaluates fwT, order-2 Taylor, and linear Taylor alternatives by varying the number of parameters in the expansion.These alternatives are tested in a 4-layer SpiderCNN on ModelNet40 classification.
- Filter analysis: Rigid transformations can generate additional polynomial terms, motivating the use of order-3 Taylor features instead of fwT.For x′ = x + y, x′2 expands to x2 + 2xy + y2.
- SpiderCNN + PointNet: A combined 3-layer SpiderCNN and PointNet model outperforms both the standalone 3-layer SpiderCNN and PointNet on ModelNet40 classification.The architecture and classification results are presented in Figure 13 and Table 6.