Source-linked AI summary
Interpolated Convolutional Networks for 3D Point Cloud Understanding
Jiageng Mao, Xiaogang Wang, Hongsheng Li
TL;DR
Point-cloud convolutions must accommodate sparse, irregular, and unordered data. The paper introduces InterpConv, which combines discrete kernel weights with feature interpolation and neighborhood normalization, and builds InterpCNNs for recognition and segmentation. Across ModelNet40, ShapeNet Parts, and S3DIS, the networks achieve state-of-the-art performance.
Problem
Sparse, irregular, and unordered point-cloud structure makes directly applying convolutions and learning efficiently challenging.
Method
InterpConv uses spatially discrete kernel weights, interpolates point features to neighboring kernel-weight coordinates, and normalizes neighborhoods for sparsity invariance; InterpCNNs build on these layers.
Results
InterpCNNs achieve state-of-the-art performance on ModelNet40, ShapeNet Parts, and S3DIS across shape classification, part segmentation, and indoor semantic parsing.
Takeaways & Limitations
The approach directly handles irregular point clouds while capturing fine-grained local structures and global shape context information.
Takeaways & Limitations
Existing graph-based methods remain sensitive to point-cloud density, while MLP-based continuous-kernel approaches can be inefficient and unstable in practice.
Abstract
from arXiv · showhide
Point cloud is an important type of 3D representation. However, directly applying convolutions on point clouds is challenging due to the sparse, irregular and unordered data structure. In this paper, we propose a novel Interpolated Convolution operation, InterpConv, to tackle the point cloud feature learning and understanding problem. The key idea is to utilize a set of discrete kernel weights and interpolate point features to neighboring kernel-weight coordinates by an interpolation function for convolution. A normalization term is introduced to handle neighborhoods of different sparsity levels. Our InterpConv is shown to be permutation and sparsity invariant, and can directly handle irregular inputs. We further design Interpolated Convolutional Neural Networks (InterpCNNs) based on InterpConv layers to handle point cloud recognition tasks including shape classification, object part segmentation and indoor scene semantic parsing. Experiments show that the networks can capture both fine-grained local structures and global shape context information effectively. The proposed approach achieves state-of-the-art performance on public benchmarks including ModelNet40, ShapeNet Parts and S3DIS.
1. Introduction
Point-cloud learning must handle sparse, irregular, unordered inputs without losing geometric information or incurring dense-grid costs. InterpConv addresses these challenges with interpolated discrete kernels, and InterpCNNs achieve state-of-the-art performance across several benchmarks.
- Point clouds are important for 3D sensing applications, but their irregular, unordered, and sparse structure makes efficient learning challenging.
- Rasterizing point clouds into voxel grids can lose geometric information and impose heavy computational costs through dense 3D convolutions.
- Graph-based approaches process irregular inputs directly, but local graph construction is not sparsity invariant and MLP-based continuous kernels may perform poorly in practice.
- InterpConv uses discrete kernel weights, interpolates point features to neighboring kernel-weight coordinates, and normalizes neighborhoods to support irregular and varying-density inputs.
- InterpCNNs capture fine-grained geometric structures and contextual information, achieving state-of-the-art performance on ModelNet40, ShapeNet Parts, and S3DIS.
- InterpConv is permutation and sparsity invariant, while InterpCNNs are designed for point-cloud recognition and segmentation.
2. Related Work
Prior point-cloud methods either regularize inputs into grids or process irregular inputs with point grouping and graph operations. The paper positions its approach as direct irregular-input processing that avoids rasterization and addresses density sensitivity and inefficient continuous kernels.
- Learning from point clouds by regular grids: Regular-grid methods use 2D projections or voxelization, but projection and rasterization can discard fine-grained geometry while increasing computational or memory costs.
- Learning from point clouds by regular grids: The proposed approach directly processes irregular point clouds without rasterization, which the paper describes as time-saving and accurate.
- Learning from point clouds by irregular inputs: PointNet provides permutation-invariant global representations, while PointNet++ groups and samples points to incorporate local structures.
- Learning from point clouds by irregular inputs: Graph-based methods build local neighborhoods and apply MLP-based continuous kernels before aggregating local features.
- Learning from point clouds by irregular inputs: The paper identifies KNN graph construction as density-sensitive and MLP coordinate processing as inefficient because it ignores explicitly defined geometric relations.
3. Method
InterpConv adapts discrete convolution to irregular point clouds by interpolating features onto kernel-weight coordinates and normalizing for varying sparsity. InterpCNN architectures use flexible receptive fields and multi-scale branches to capture local geometry and broader shape context.
- InterpConv: Unlike voxel convolutions and graph-based continuous convolutions, InterpConv directly processes irregular point clouds without rasterization or learned continuous kernels.Voxelization can lose geometric information, while MLP-based continuous kernels may be inefficient or unstable.
- InterpConv: InterpConv decouples convolution into spatially discrete kernel weights, an interpolation function, and a density normalization term.The interpolation function relates neighboring points to kernel-weight coordinates, while normalization supports sparsity invariance.
- Discrete kernel weights: Kernel length controls receptive field: shorter lengths capture local structures, whereas longer lengths encode more global context.Kernel-weight coordinates may be fixed or learned, and kernel length determines distances between adjacent weights in Euclidean space.
- Interpolation functions: Interpolation assigns input-point features to nearby kernel-weight coordinates, preserving fine-grained structures that naive rasterization can lose.The paper considers trilinear and Gaussian interpolation; trilinear interpolation uses eight adjacent kernel-weight coordinates and self-normalizing weights.
- Normalization: Feature aggregation can normalize either by the number of neighboring points or by the sum of interpolation weights.For point features f_i and interpolation weights t_i, the paper presents both normalization choices.
- Network architectures: PointInception concatenates branches with different kernel lengths to combine multi-scale geometric features, and stacked modules capture local structures and shape context.The classification architecture uses multireceptive-field InterpConv blocks, while deeper models stack PointInception modules.
4. Experiments
Experiments evaluate InterpCNNs on shape classification, part segmentation, and indoor scene parsing, alongside architectural and component ablations. The results show strong benchmark performance, effective multi-scale feature learning, and favorable parameter and runtime comparisons.
- Benchmark evaluation: InterpCNNs are evaluated on ModelNet40, ShapeNet Parts, and S3DIS using classification, part segmentation, and indoor semantic parsing tasks.The experiments use benchmark datasets and task-specific architectures, including classification and encoder-decoder segmentation networks.
- Shape classification: 0.8% improvement in overall accuracy over DGCNN is reported on ModelNet40, with InterpCNN outperforming graph-based and voxel-based counterparts.The comparison reports overall accuracy and also states that InterpCNN performs better than Point2Seq and 3DCapsule.
- Object part segmentation: InterpCNN performs better than compared methods on ShapeNet Parts mean IoU over instances.The study reports mean IoU over both categories and instances, identifying instance mean IoU as the more realistic measure.
- Indoor scene segmentation: InterpCNN significantly outperforms state-of-the-art methods on both S3DIS overall accuracy and mean IoU over categories.The evaluation follows 6-fold validation across six areas.
- Ablation study: 3 × 3 × 3 InterpConvs improve performance by 3% over replacing blocks with 1 × 1 × 1 kernels, while 5 × 5 × 5 kernels provide no further improvement.The ablation links larger-than-1 kernels to local-structure capture and reports that 3 × 3 × 3 kernels reduce parameters relative to 5 × 5 × 5 counterparts.
- Ablation study: Gaussian interpolation performs better in classification, whereas trilinear interpolation performs better in segmentation; normalization methods are comparable, with interpolation-weight normalization preferred in extreme sparse neighborhoods.The preferred normalization handles cases with few close points and many far points around a kernel-weight coordinate.
- Efficiency analysis: InterpCNNs use fewer parameters than other 3D convolution methods and are much faster than DGCNN, though slightly slower than PointNet++.The runtime comparison uses batch size 16 and 1,024 points on an NVIDIA TITAN X GPU.
5. Conclusion
InterpConv and InterpCNNs support 3D classification and segmentation, with experiments on ModelNet40, ShapeNet Parts, and S3DIS showing promising results against existing methods.
- InterpConv and InterpCNNs are proposed for 3D classification and segmentation.