PaperlayerView exact revision on arXiv

Source-linked AI summary

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

Mutian Xu, Runyu Ding, Hengshuang Zhao, Xiaojuan Qi

arXiv:2103.14635v2cs.CV

TL;DR

PAConv targets the difficulty of learning spatial relationships in sparse, irregular, and unordered point clouds. It dynamically assembles Weight Bank matrices with position-based ScoreNet coefficients and integrates them into unchanged MLP-based pipelines. The method approaches or surpasses state-of-the-art performance while significantly improving classification and segmentation baselines with decent efficiency.

  • Problem

    Sparse, irregular, and unordered point clouds make conventional position-to-kernel mappings and effective processing difficult.

  • Method

    PAConv constructs dynamic kernels by combining Weight Bank matrices with coefficients learned from relative point positions through ScoreNet.

  • Results

    PAConv approaches or surpasses state-of-the-art models and significantly improves baselines on classification and segmentation tasks with decent model efficiency.

  • Takeaways & Limitations

    PAConv can replace MLPs in simple point-cloud networks without changing other network configurations while retaining strong performance and efficiency.

  • Takeaways & Limitations

    Using too many Weight Bank matrices can introduce redundancy and heavy memory or computation overheads, with M=8 or 16 found appropriate.

Abstract

from arXiv · show

We introduce Position Adaptive Convolution (PAConv), a generic convolution operation for 3D point cloud processing. The key of PAConv is to construct the convolution kernel by dynamically assembling basic weight matrices stored in Weight Bank, where the coefficients of these weight matrices are self-adaptively learned from point positions through ScoreNet. In this way, the kernel is built in a data-driven manner, endowing PAConv with more flexibility than 2D convolutions to better handle the irregular and unordered point cloud data. Besides, the complexity of the learning process is reduced by combining weight matrices instead of brutally predicting kernels from point positions. Furthermore, different from the existing point convolution operators whose network architectures are often heavily engineered, we integrate our PAConv into classical MLP-based point cloud pipelines without changing network configurations. Even built on simple networks, our method still approaches or even surpasses the state-of-the-art models, and significantly improves baseline performance on both classification and segmentation tasks, yet with decent efficiency. Thorough ablation studies and visualizations are provided to understand PAConv. Code is released on https://github.com/CVMI-Lab/PAConv.

1. Introduction

3D point clouds are important but difficult to process because their sparse, irregular, and unordered structure challenges conventional approaches. PAConv addresses these issues with position-adaptive kernels that improve simple point-cloud pipelines while retaining efficiency.

  • Sparse, irregular, and unordered point-cloud structure makes effective and efficient 3D processing challenging.
  • Voxelization enables grid convolutions but can lose geometric information through quantization and add memory and computational costs.
  • Shared MLP processing limits representation of spatial-variant relationships because every point uses the same MLP.
  • Point convolutions that predict spatial-variant kernels can incur heavy computation and memory costs, while efficient implementations may sacrifice flexibility and performance.
  • PAConv dynamically assembles kernels from Weight Bank matrices using relative-position coefficients learned by ScoreNet, avoiding brute-force kernel prediction.
  • Without changing network configurations, PAConv reaches state-of-the-art performance on ModelNet40 and improves ShapeNet Part by 2.3% and S3DIS by 9.31%.

2. Related Work

Prior point-cloud methods either project or quantize geometry into regular grids, use fixed kernel points, or employ conditional kernels without position adaptation. PAConv instead learns position-adaptive combinations of weight matrices.

  • Projection and voxelization methods can lose 3D geometry, while voxel representations also incur computational and memory costs.
  • Kernel-point methods depend on predefined kernel positions and hand-crafted combinations, limiting flexibility across datasets and backbones.
  • CondConv combines filters through routing but is not position-adaptive, whereas point clouds require weights that adapt to different point locations.

3. Method

PAConv addresses the mismatch between image-style fixed kernel mappings and irregular 3D point clouds by dynamically assembling position-adaptive kernels from a Weight Bank using ScoreNet. Weight regularization promotes diverse basis matrices, while the operator remains flexible for modeling spatial relationships.

  • General Point Convolution: General point convolution aggregates neighboring features after applying weights generated from the positional relation between center and neighbor points.The aggregation function can use MAX, SUM, or AVG; 2D convolution is a special case with fixed neighborhood-position mappings.
  • Motivation: Continuous, unordered 3D point clouds admit infinitely many relative offsets, so finite one-to-one kernel mappings used by images are unsuitable.PAConv therefore redesigns the kernel function as a position-adaptive mapping.
  • Dynamic Kernel Assembling: PAConv dynamically assembles each convolution kernel from basic Weight Bank matrices using coefficients learned from relative point positions by ScoreNet.This avoids directly predicting an entire kernel while retaining data-driven adaptation to geometric structure.
  • Weight Bank: The Weight Bank stores M learnable Cin×Cout matrices, with M controlling the number of available bases for kernel assembly.The paper reports that M=8 or 16 is appropriate, while larger banks can introduce redundancy and memory or computation overhead.
  • ScoreNet: ScoreNet maps a position-relation input through MLPs to normalized coefficients, where each coefficient represents one matrix’s contribution to the generated kernel.Softmax scores lie in (0, 1), and higher scores indicate stronger relations between a weight matrix and the position input.
  • Weight Regularization: Weight regularization penalizes correlations between Weight Bank matrices so generated kernels retain greater diversity.Without this constraint, randomly initialized matrices may converge to similar solutions.

4. Backbone Network Architectures

The authors evaluate PAConv by inserting it into simple classical MLP-based backbones while leaving other network configurations unchanged. Different task scales use lightweight object-level networks or encoder-decoder scene-level networks.

  • Backbone Selection: Three classical MLP-based backbones—PointNet, PointNet++, and DGCNN—are used to assess PAConv while minimizing architectural confounds.PAConv replaces MLPs without further network-configuration modifications.
  • Object-Level Tasks: Object-level tasks use lightweight networks without down-sampling layers because they process individual 3D objects.These networks are designed for object-level recognition settings.
  • Object-Level Tasks: DGCNN neighborhoods are formed by searching k-nearest neighbors in 3D coordinate space instead of feature space.This choice avoids the huge computational cost and memory usage associated with feature-space pairwise distances.
  • Scene-Level Tasks: Scene-level segmentation uses a PointNet++-style encoder with farthest-point sampling and a decoder for upsampling.The implementation uses k-nearest-neighbor neighborhoods and the simpler single-scale grouping approach rather than MSG or MRG.
  • Scene-Level Tasks: In PointNet++, PAConv directly replaces the MLPs in encoding layers, while the decoder remains unchanged.Features are propagated directly to the next layer without feature-fusion tricks.

5. Experiments

PAConv is integrated into simple point-cloud backbones and evaluated on classification and segmentation benchmarks, improving baselines while retaining computational efficiency.

  • Experimental setup: PAConv is implemented as a CUDA layer and integrated into point-cloud networks without modifying other network configurations.The experiments use PAConv across PointNet, PointNet++, and DGCNN backbones.
  • Object classification: 9,843 ModelNet40 shapes are used for training and 2,468 for testing in the object-classification evaluation.The dataset contains 40 categories of 3D meshed models.
  • Object classification: 4.0% and 1.0% accuracy gains are obtained over PointNet and DGCNN, respectively, on ModelNet40.DGCNN+PAConv reaches 93.9% accuracy with voting and 93.6% without voting.
  • Shape part segmentation: ShapeNet Parts visualizations compare ground-truth labels in the first row with PAConv predictions in the second across five object categories.The displayed categories are motorbike, lamp, aeroplane, chair, and pistol.
  • Indoor scene segmentation: 9.31% improvement is achieved over PointNet++ on S3DIS, while PAConv uses 38.6% fewer computational FLOPs than the compared methods.With Lcorr, PAConv obtains the best mIoU among methods using block sampling; Area-5 is used for testing.

6. Ablation Studies

Ablations examine ScoreNet inputs and normalization, weight-bank size and regularization, and robustness under transformations. The results link richer positional inputs, Softmax scoring, diverse weight matrices, and stable transformed performance to PAConv’s behavior.

  • 6.1. ScoreNet: Using all three coordinate axes as ScoreNet input achieves the best segmentation performance on S3DIS Area-5.The representation includes neighbor and center coordinates and their Euclidean distance.
  • 6.1. ScoreNet: Softmax normalization outperforms Sigmoid and Tanh by predicting scores for all weight matrices jointly.The comparison concerns how ScoreNet distributes scores and assembles weight matrices.
  • 6.2. The Number of Weight Matrices: 65.05% performance with 2 weight matrices is only 0.58% below the 16-matrix setting, while 16 matrices provide the best and most stable performance.Larger matrix counts can experience fluctuating gains because of optimization issues.
  • 6.3. Weight Bank Regularization: Correlation regularization improves scene-segmentation mIoU by 0.95 points while substantially lowering Pearson’s R between weight matrices.The regularizer encourages low correlations and greater diversity among the matrices.
  • 6.4. Robustness: PAConv remains stable under random permutation, rotations, translations, scaling, and Gaussian jittering.Its symmetric neighborhood aggregation supports permutation invariance, while position-based score assembly contributes to transformation robustness.

7. Conclusion

PAConv dynamically assembles convolution kernels from Weight Bank matrices using ScoreNet-learned position coefficients. Integrated into simple MLP-based networks without configuration changes, it approaches or surpasses state-of-the-art performance and improves baselines efficiently.

  • 7. Conclusion: PAConv combines Weight Bank matrices into convolution kernels, with coefficients learned from point positions through ScoreNet.The operator is designed for point cloud processing and dynamic kernel assembling.
  • 7. Conclusion: Embedded in simple MLP-based networks without configuration changes, PAConv approaches or surpasses state-of-the-art results and significantly outperforms baselines with decent efficiency.The conclusion attributes these findings to extensive experiments and ablation studies.

Outline

The supplementary document benchmarks recent point convolutions, studies ScoreNet, documents task-specific configurations and implementation strategies, provides a CUDA implementation, and lists detailed segmentation results.

  • Supplementary overview: Section A compares recent point convolution operators under matched backbones and data augmentation.The comparison focuses on performance under controlled settings.
  • Supplementary overview: Section B investigates ScoreNet’s effects on PAConv.This includes analysis of ScoreNet behavior and settings.
  • Supplementary overview: Section C details network configurations and implementation strategies for different downstream tasks.The section addresses how PAConv is integrated across tasks.
  • Supplementary overview: Section D presents a CUDA implementation of PAConv, while Section E lists detailed semantic-segmentation results.These sections provide implementation and expanded evaluation material.

A. Comparison of Point Convolutional Operators

Controlled S3DIS Area-5 comparisons replace PointNet++ MLPs with point convolution operators without changing network architecture or feature dimensions. PAConv yields the strongest reported improvement among the compared operators, while KPConv reproduction is not comparable.

  • Experimental setup: The comparison integrates PointConv and KPConv into PointNet++ by directly replacing MLPs while keeping the architecture fixed.All experiments use S3DIS Area-5 and the same data augmentation strategies.
  • Results: 9.31%↑ mIoU is reported for PAConv over PointNet++ with decent efficiency, compared with PointConv’s 2.7%↑ and time-consuming inference.The PointConv comparison also reports tremendous FLOPs.
  • Results: KPConv is not reported because the reproduced result was not comparable with its original version.The table note attributes this to implementation-specific kernel-point radius tuning across point-cloud scales.
  • Comparison: PAConv avoids complicated architecture design and hand-crafted kernel-point-space adjustment, supporting flexibility and efficiency across applications.This is presented as a comparison with KPConv.

B. More Explorations on ScoreNet

The appendix examines how ScoreNet depth and spatial score distributions affect PAConv, while documenting its integration into unchanged MLP-based point-cloud backbones and training setups.

  • ScoreNet depth: Deeper ScoreNets improve performance but reduce efficiency on S3DIS without correlation loss.Table B.1 reports floating-point operations per sample for balancing performance and efficiency.
  • Score distribution in the network: Score coefficients for different weight matrices are diversely distributed across network layers, indicating that nearly all matrices are utilized.The visualization supports flexible dynamic kernel assembly rather than concentration on a single matrix.
  • Score distribution in 3D space: Different weight matrices capture different position relations in 3D space through their learned score distributions.The spatial visualization maps larger scores to points with higher plotted values or colors closer to yellow.
  • Network configurations: PAConv is embedded into PointNet, PointNet++, and DGCNN without modifying their network architectures or feature dimensions.For PointNet++, PAConv replaces PointNet modules in encoding layers; DGCNN replaces EdgeConv while retaining the official architecture.
  • Training and implementation: The documented implementations use task-specific neighborhood sizes, optimizers, learning-rate schedules, batch sizes, and training durations.DGCNN-based classification uses 20 neighbors, part segmentation 30, and both converge in 350 epochs; PointNet++ scene segmentation uses 32 neighbors for 100 epochs.

D. CUDA Implementation

The CUDA implementation assembles transformed neighbor features on the fly to reduce memory use, while the appendix reports segmentation comparisons and a version-dependent S3DIS caveat.

  • CUDA implementation: PAConv can assemble neighbor features after transforming them with weight matrices, avoiding storage of a large intermediate matrix.The CUDA layer queries neighbor indices on the fly during assembly.
  • CUDA implementation: Memory usage decreases from 10G+ to 5600M with 65,536 points in the CUDA implementation.The reduction is attributed to on-the-fly neighbor-index querying without storing large intermediate matrices.
  • Implementation caveat: CUDA and original PAConv versions produce slightly different S3DIS semantic-segmentation performance because they aggregate neighboring features at different pipeline stages.The CUDA version aggregates after each PAConv layer, whereas the original version refines local features through three PAConv layers before aggregation.
  • Segmentation results: PAConv with Lcorr achieves the best mAcc among listed S3DIS approaches and compares on par with deformable KPConv without grid sampling.The comparison concerns category-wise segmentation results on S3DIS Area-5.
  • Segmentation visualizations: PointNet++ equipped with PAConv shows conspicuously stronger segmentation than original PointNet++ across various S3DIS scenes and areas.The figure compares original inputs, ground truth, PointNet++ outputs, and PAConv-enhanced outputs by scene row.
Loading 2103.14635v2…