Source-linked AI summary
KPConv: Flexible and Deformable Convolution for Point Clouds
Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, Leonidas J. Guibas
TL;DR
Point clouds lack grid structure, limiting conventional convolutional approaches. KPConv places learnable weights at spatial kernel points and achieves strong classification and segmentation performance, with rigid and deformable variants suited to different task complexities.
Problem
Point clouds are non-grid structured data, motivating convolution operators that process them without intermediate representations.
Method
KPConv applies weights spatially located by kernel points to radius neighborhoods and learns local shifts that deform kernels to fit point-cloud geometry.
Results
Rigid and deformable KPConv networks top competing algorithms on several datasets, with rigid variants better for simpler tasks and deformable variants for more diverse, difficult segmentation.
Takeaways & Limitations
Rigid and deformable KPConv are both valuable across datasets and configurations, while deformable kernels provide greater ability to adapt to scene-object geometry.
Takeaways & Limitations
Deformable KPConv requires fitting and repulsive regularization because unconstrained kernel shifts can pull kernel points away from sparse input points or collapse them together.
Abstract
from arXiv · showhide
We present Kernel Point Convolution (KPConv), a new design of point convolution, i.e. that operates on point clouds without any intermediate representation. The convolution weights of KPConv are located in Euclidean space by kernel points, and applied to the input points close to them. Its capacity to use any number of kernel points gives KPConv more flexibility than fixed grid convolutions. Furthermore, these locations are continuous in space and can be learned by the network. Therefore, KPConv can be extended to deformable convolutions that learn to adapt kernel points to local geometry. Thanks to a regular subsampling strategy, KPConv is also efficient and robust to varying densities. Whether they use deformable KPConv for complex tasks, or rigid KPconv for simpler tasks, our networks outperform state-of-the-art classification and segmentation approaches on several datasets. We also offer ablation studies and visualizations to provide understanding of what has been learned by KPConv and to validate the descriptive power of deformable KPConv.
1. Introduction
The paper introduces KPConv, a point convolution using spatially located kernel points for unordered point clouds, and extends it with deformable kernels that adapt to local geometry. It reports robustness to varying densities and strong classification and segmentation performance across datasets.
- KPConv: KPConv defines local 3D filters by attaching convolution weights to kernel points that determine where each weight is applied.The operator is inspired by image convolution but replaces kernel pixels with kernel points.
- Deformable KPConv: Deformable KPConv learns local shifts for kernel points, allowing kernel shapes to adapt across different input-cloud regions.The deformable version also uses regularization to help deformed kernels fit the data.
- Efficiency and robustness: Radius neighborhoods combined with regular subsampling make KPConv robust to varying point densities while reducing the computational cost associated with normalization strategies.The paper contrasts this approach with KNN neighborhoods, which are not robust under non-uniform sampling.
- Results: Rigid and deformable KPConv both perform very well, topping competing algorithms on several datasets in classification and segmentation.The paper also reports that KPConv supports very deep architectures while keeping fast training and inference times.
- Results: Deformable kernels improve the network’s ability to adapt to the geometry of scene objects, according to the qualitative effective-receptive-field study.The study is presented as qualitative evidence about what deformable kernels learn.
2. Related Work
Prior point-cloud methods project points onto grids, approximate kernels with intermediate functions, or use spatially constrained and computationally costly designs. KPConv instead uses explicit spatially located weights, flexible kernel-point counts, order-invariant correlations, and deformability, and is reported to outperform comparable networks.
- Projection networks: Projection methods map point clouds to multi-view images or 3D voxel grids, introducing occlusion, density, or kernel-size constraints.Sparse octrees and hash-maps improve voxel scalability, but voxel kernels remain constrained to 27 or 125 positions; permutohedral lattices reduce this to 15.
- Point convolutions: Unlike MLP-based point convolutions, KPConv directly learns an explicit convolution kernel without an intermediate representation.The paper associates MLP representations with greater operator complexity and harder convergence, while describing KPConv’s explicit kernel as image-convolution-like.
- Point convolutions: Pointwise CNN uses voxel bins and costly normalization, whereas KPConv’s subsampling strategy addresses varying densities and computational cost.The comparison specifically identifies voxel-bin placement as limiting flexibility like grid networks.
- Point convolutions: SpiderCNN, Flex-convolution, and PCNN have limitations involving spatial inconsistency, representative power, density robustness, or scalability.SpiderCNN orders neighbors by distance; Flex-convolution uses linear kernels and KNN; PCNN has quadratic computation and Gaussian correlation.
- Positioning: KPConv networks outperform all comparable networks in the experiments, and the authors report no prior experiments with spatially deformable point convolutions.The claim is stated as a summary of the paper’s experimental comparison and prior-work review.
3. Kernel Point Convolution
KPConv defines point convolution over radius-based neighborhoods using kernel points with associated weights and distance-based correlations. Its deformable variant learns geometry-adaptive kernel positions, supported by fitting regularization and density-controlling subsampling.
- Neighborhoods: Radius neighborhoods provide a consistent spherical domain intended to improve robustness to varying point densities and support meaningful representations.The neighborhood is defined by points within a chosen radius r rather than by a fixed number of nearest neighbors.
- Kernel function: KPConv defines the kernel function over a radius-neighborhood ball using kernel points that assign distinct weights to different spatial regions.Kernel points are associated with weight matrices mapping features from D_in to D_out.
- Kernel function: The kernel function weights each neighbor through a linear correlation that increases as the neighbor approaches its kernel point.The influence distance σ is chosen according to input density, and the simpler correlation facilitates gradient backpropagation during kernel deformation learning.
- Deformable KPConv: Deformable KPConv learns input-dependent kernel-point shifts from a rigid KPConv, while fitting regularization prevents kernel points from being lost when they have no neighbors in range.The learned shifts are intended to fit the local geometry of the input point cloud.
- Implementation: Grid subsampling controls density across layers by selecting barycenters of points in non-empty grid cells, enabling spatially consistent support locations.The method also defines KP-CNN for classification and KP-FCNN for segmentation, with deformable kernels used only in the last 5 KPConv blocks for classification results.
4. Experiments
Experiments on shape benchmarks and diverse real-world scenes show that KPConv outperforms point-based state-of-the-art methods, with rigid or deformable kernels favored by task and dataset characteristics. Ablations and visualizations indicate that deformable KPConv retains descriptive power with few kernel points and adapts to scene geometry.
- Shape classification and segmentation: KPConv outperforms other state-of-the-art methods using only point inputs on ModelNet40 classification and ShapeNetPart part segmentation.ModelNet40 is evaluated with overall accuracy, while ShapeNetPart uses class-average and instance-average IoU.
- Shape classification and segmentation: Rigid KPConv performs slightly better than deformable KPConv on the simpler ModelNet40 classification task.The paper attributes this to deformable kernels increasing network complexity, potentially disturbing convergence or causing overfitting.
- Scene segmentation: Across four real-scene datasets, deformable kernels improve Paris-Lille-3D and S3DIS, whereas rigid kernels perform better on Scannet and Semantic3D.The datasets cover indoor cluttered scenes, large indoor spaces, fixed outdoor scans, and mobile outdoor scans; S3DIS evaluation uses Area-5.
- Ablation study: Deformable KPConv loses only 1.5% mIoU when restricted to four kernel points on the Scannet validation set.Validation experiments also show deformable KPConv outperforming rigid KPConv with 15 kernel points and performing better across several validation sets.
- Learned features: Visualizations show that KPConv learns low-level planes, lines, and corners in early layers and more complex shapes such as buttresses, balls, cones, and stairs later.The feature visualization was performed with rigid KPConv on ModelNet40, and it did not clearly distinguish rigid from deformable kernels.
- Learned features: Deformable KPConv adapts to scene-object geometry, supporting its stronger performance on diverse indoor datasets.The paper links this adaptive behavior to improved network adaptation and notes that deformable kernels are useful on large and diverse datasets.
5. Conclusion
KPConv is a point-cloud convolution that processes radius neighborhoods using weights spatially located by kernel points. Its deformable variant learns local shifts to adapt convolution kernels to point-cloud geometry, while rigid and deformable versions suit different dataset diversity and network configurations.
- Core contribution: KPConv processes point-cloud radius neighborhoods with weights spatially located by a small set of kernel points.The operator works directly on point clouds.
- Deformable operator: Deformable KPConv learns local shifts that deform convolution kernels to fit point-cloud geometry.The learned shifts adapt the kernels to local geometric structure.
- Model variants: Rigid and deformable KPConv are selected according to dataset diversity or network configuration.The passage presents both variants as appropriate under different conditions.
Supplementary Material for KPConv: Flexible and Deformable Convolution for Point Clouds · Abstract
The supplementary document expands on KPConv’s architectures, initialization, regularization, segmentation results, and explanatory videos. It also compares model sizes and speeds and illustrates learned kernel deformations and scene segmentation.
- Abstract: Section A details network architectures, training parameters, and comparisons of model sizes and speeds.
- Abstract: Section B presents the kernel point initialization method.
- Abstract: Section C describes regularization for addressing the “lost” kernel point phenomenon.
- Abstract: Section D provides additional segmentation results with class scores.
- Abstract: The KPConv Method video illustrates the KPConv principle with animated diagrams and shows learned kernel deformations.
- Abstract: The KPConv Results video shows indoor and outdoor scenes segmented by KP-FCNN.
A. Network Architectures and Parameters
The architectures use bottleneck-ResNet-style convolutional blocks with rigid or deformable KPConv, while separate segmentation and classification networks process variable-size point clouds through independent batch elements. Their blocks combine residual connections, batch normalization, and leaky ReLU, with optional shortcut operations for strided or dimension-changing convolutions.
- Architecture blocks: Both rigid and deformable KPConv architectures are built from bottleneck-ResNet-style convolutional blocks.The same block design applies to normal or strided KPConv and to rigid or deformable kernels.
- Batching: Variable-size point clouds are concatenated along the point dimension and processed independently because neighbor and pooling indices do not cross batch elements.Global pooling tracks batch-element indices, and batches are formed by adding examples until a target total-point limit is reached.
- Architecture blocks: The blocks use residual connections, batch normalization, and leaky ReLU for both rigid and deformable KPConv.Shortcut max pooling is optional for strided KPConv, while shortcut 1x1 convolution is needed when D_in ≠ 2D.
- Network architectures: The paper defines separate network architectures for 3D point-cloud segmentation and classification, with points serving as support structures for feature transformations.Figure 9 illustrates consecutive operations applied to features while the point clouds guide each layer.
- Model sizes and speeds: KP-FCNN and KP-CNN have similar parameter counts because the KP-FCNN decoder uses only lightweight 1x1 convolutions.The passage also notes that Table 3 reports model statistics across ModelNet40, ShapeNetPart, Scannet, and Semantic3D.
B. Kernel Points Initialization
KPConv initializes kernel points by optimizing a repulsive energy within a sphere, producing stable, regular polyhedral arrangements for selected kernel sizes. These dispositions are then adapted per layer through rescaling and random rotation, while the energy can regularize learned locations.
- Energy-based initialization: Kernel points are initialized by maximizing separation within a sphere through a repulsive potential, with an attractive center potential preventing indefinite divergence.The global energy combines these potentials over the kernel points.
- Optimization: Gradient descent starts from random point locations and may include optional constraints, including fixing one point at the sphere center.For some K values, fixing the center point leads to a unique stable disposition.
- Stable dispositions: For some values of K, the optimized points converge to unique stable dispositions that form regular polyhedrons.These polyhedrons can be described by groups of points sharing planes perpendicular to their symmetry axes.
- Layer-wise placement: Every KP-CNN and KP-FCNN layer rescales a chosen stable disposition to its appropriate radius and randomly rotates it.This applies the same initialized geometric arrangement across layers while adapting its scale and orientation.
- Learned positions: The total energy can also serve as a regularization loss when kernel point positions are trained in KP-CNN.This extends the initialization objective to learned kernel locations.
C. Effect of the Kernel Point Regularization
Deformable KPConv can lose kernel points because sparse point clouds leave shifted points without gradients. The proposed regularization prevents these lost points and increases the number of active kernel points.
- Sparse point clouds can pull deformable kernel points away from input points, causing poor performance.Empty space around points allows network-predicted shifts to move kernel points beyond the input support.
- Shifted kernel points receive null shift gradients and remain lost across similar input shapes.The stochastic optimizer can cause this loss for many input shapes during convergence.
- Without regularization, many deformed kernel points lie far from the input floor plane and become lost.Figure 11 contrasts the rigid kernel with a deformation in which most purple points are far from the floor.
- The regularization strategy prevents lost kernel points and maximizes the number of active kernel points.Almost every yellow point remains close to the floor plane and has input points within range.
D. More Segmentation Results
This section reports additional segmentation experiments for benchmarking future work, including class-level results on ShapeNetPart and S3DIS. It also points to online benchmark scores for Scannet, Semantic3D, and NPM3D.
- Benchmarking: The section provides class scores for segmentation experiments on ShapeNetPart and S3DIS.ShapeNetPart results appear in Table 5, while S3DIS results appear in Tables 6 and 7.
- Online benchmarks: Class scores for Scannet, Semantic3D, and NPM3D are available on their respective online benchmark websites.These datasets are identified as online benchmarks rather than results detailed in the section tables.
- ShapeNetPart: Table 5 reports segmentation mIoUs on ShapeNetPart.The passage identifies this table as the ShapeNetPart segmentation benchmark.
- S3DIS: Tables 6 and 7 report semantic segmentation IoU scores on S3DIS Area-5 and S3DIS k-fold.Both tables additionally report mean class recall, also called mean class accuracy in some previous works.