PaperlayerView exact revision on arXiv

Source-linked AI summary

PointConv: Deep Convolutional Networks on 3D Point Clouds

Wenxuan Wu, Zhongang Qi, Li Fuxin

arXiv:1811.07246v3cs.CV

TL;DR

Irregular, unordered point clouds make conventional convolution difficult, while volumetric processing is costly at high resolution. PointConv models continuous, density-adjusted filters from local coordinates and uses a memory-efficient reformulation to scale deep networks. The resulting networks achieve strong point-cloud segmentation performance and match similar-structure image CNNs on converted CIFAR-10.

  • Problem

    Irregular, unordered point clouds lack regular grids, making conventional CNNs difficult to apply and high-resolution volumetric alternatives computationally intractable.

  • Method

    PointConv learns coordinate-dependent continuous weight functions with MLPs, applies inverse-density reweighting, and uses a summation-order reformulation for memory-efficient computation.

  • Results

    PointConv networks achieve state-of-the-art results on part and indoor semantic segmentation benchmarks and match similar-structure image CNNs on converted CIFAR-10.

  • Takeaways & Limitations

    PointConv enables deep convolutional networks to operate directly on point clouds with translation and permutation invariance.

  • Takeaways & Limitations

    The naive PointConv implementation is memory-inefficient for large output channel sizes, and approximating each filter with a network is inefficient.

Abstract

from arXiv · show

Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and density functions through kernel density estimation. The most important contribution of this work is a novel reformulation proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure.

1. Introduction

PointConv addresses the difficulty of applying convolution to unordered, irregular point clouds by modeling continuous, density-adjusted filters and introducing a memory-efficient implementation. Its PointDeconv extension supports finer-resolution segmentation, while experiments report competitive or state-of-the-art results across point-cloud benchmarks and comparable CIFAR-10 accuracy to similar image CNNs.

  • Point clouds are unordered and lack regular lattice grids, making conventional CNNs difficult to apply; volumetric alternatives become sparse and computationally intractable at high resolution.
  • PointConv approximates continuous 3D convolution by learning coordinate-dependent weight functions with MLPs and reweighting them with inverse density to address non-uniform sampling.
  • A summation-order reformulation greatly improves memory efficiency, enabling deeper point-cloud convolutional networks with translation and point-order permutation invariance.
  • PointDeconv propagates coarse-layer information to finer layers, supporting improved semantic segmentation performance.
  • PointConv networks achieve state-of-the-art results in part segmentation and indoor semantic segmentation, while matching similar-structure image CNNs on converted CIFAR-10.

2. Related Work

Prior 3D point-cloud methods commonly project data to images, voxelize it, or process raw points with shared MLPs and pooling. These approaches face resolution, computational, local-feature, or permutation-invariance constraints, while continuous-filter methods pursue direct convolutional alternatives.

  • Projection methods convert point clouds or shapes into multiple 2D images before applying 2D CNNs, but extending them to high-resolution scene segmentation is nontrivial.
  • Voxel-based methods quantize point clouds into volumetric grids, but their resolution and 3D convolution cost constrain performance.
  • PointNet-style shared MLPs with global max pooling make local-feature capture difficult, while PointNet++ adds hierarchical local-to-global feature extraction.
  • PointCNN uses learned transformations to weight and permute point features, but the cited comparison states it does not achieve permutation invariance.
  • Continuous-filter approaches approximate convolutional weight functions, while some efficient variants approximate only depth-wise rather than full convolution.

3. PointConv

PointConv extends continuous convolution to irregular, non-uniform point clouds by learning coordinate-dependent weights and density corrections. Its efficient implementation supports hierarchical convolutional networks, deconvolution, and invariance to point ordering and translation.

  • Convolution on 3D Point Clouds: PointConv extends traditional convolution to point clouds by approximating a continuous 3D convolution operator.It uses local relative coordinates rather than fixed raster-grid positions.
  • Convolution on 3D Point Clouds: Because point-cloud coordinates are arbitrary and local point arrangements vary, conventional raster convolution cannot be applied directly.Point clouds are non-uniform samples, so local point density and relative positions differ across regions.
  • Convolution on 3D Point Clouds: PointConv learns coordinate-dependent weight functions with MLPs and estimates inverse density with KDE followed by a nonlinear transform.The density correction reduces the effect of non-uniform sampling, including clusters where nearby points should contribute less individually.
  • Convolution on 3D Point Clouds: Shared MLPs generate weights from each point’s local coordinates, while density-scaled features are aggregated into output channels for each local region.The operation uses local positions and input features, with a density scale applied to the learned weights.
  • Convolution on 3D Point Clouds: PointConv is permutation-invariant and translation-invariant because its weight functions use relative positions and shared parameters.On a regular image grid, the same construction reduces to conventional discretized convolution.
  • Convolution on 3D Point Clouds: A summation-order reformulation improves PointConv’s memory efficiency, addressing the inefficiency of approximating each filter with a separate network.This reformulation enables scaling to deeper, larger point-cloud convolutional networks.
  • Convolution on 3D Point Clouds: The method includes PointDeconv for propagating coarse features to finer resolutions and uses hierarchical modules to combine local features across spatial scales.PointDeconv combines interpolation, skip-linked features, and PointConv.

4. Efficient PointConv

The efficient PointConv reformulation replaces memory-intensive filter storage with matrix multiplication and 1 × 1 convolution, enabling scalable point-cloud networks while preserving PointConv’s operation.

  • Reformulation: The reformulation expresses PointConv using standard matrix multiplication and convolution operations, improving GPU parallelism and framework compatibility.The original operation is equivalently reduced to matrix multiplication and a 1 × 1 convolution.
  • PointConv operation: PointConv operates on local coordinates and neighboring features, producing an output feature at the center point of each local region.The local region uses K nearest neighbors, with coordinates transformed from global to local coordinates.
  • Memory challenge: The naive PointConv implementation stores filters of size B × N × K × (Cin × Cout), creating severe memory demands as channel sizes grow.With B = 32, N = 512, K = 32, Cin = 64, and Cout = 64, one layer requires 8GB for single-precision filters.
  • Reformulation: The method rewrites the MLP-generated weight function as W = Conv1×1(H, M), separating intermediate features M from the final kernel H.This factorization changes the summation order while retaining the PointConv computation.
  • Memory efficiency: Instead of storing generated filters, the efficient version stores intermediate result M and convolution kernel H, reducing memory consumption to Cmid K×Cout of the original.For Cmid = 32 under the Figure 3 setup, memory falls to 0.1255GB, about 1/64 of the original PointConv.

5. Experiments

Experiments evaluate PointConv across object classification, part segmentation, indoor semantic segmentation, and CIFAR-10 converted to point clouds. The reported results are competitive or state-of-the-art across these settings.

  • Experimental setup: PointConv is evaluated on ModelNet40, ShapeNet, ScanNet, and CIFAR-10 converted into point clouds using coordinates and features.The experiments use TensorFlow on a GTX 1080Ti with Adam optimization, ReLU, and batch normalization.
  • ShapeNet part segmentation: 82.8% class-average mIoU and 85.7% instance-average mIoU are reported for PointConv on ShapeNet part segmentation.These results are described as on par with state-of-the-art methods using only point-cloud inputs.
  • Semantic scene segmentation: PointConv outperforms other compared algorithms by a significant margin on ScanNet semantic scene segmentation.The evaluation compares Tangent Convolutions, SPLAT Net, PointNet++, and ScanNet using mIoU.
  • CIFAR-10: 89.13% accuracy is achieved by a 5-layer PointConv network on CIFAR-10, similar to the corresponding image-convolution network.CIFAR-10 pixels are represented as 2D points with xy coordinates and RGB features.

6. Ablation Experiments and Visualizations

The ablations examine PointConv’s density scaling, MLP configuration, sliding-window settings, and learned filters, while ScanNet results show strong performance against baselines.

  • MLP Configuration: Larger Cmid values do not necessarily improve classification, and varying the MLP layer count produces little difference in results.Each configuration was evaluated across three random trials on a 1,024-point ScanNet classification subset.
  • Inverse Density Scale: Inverse density scaling improves PointConv’s ScanNet performance by about 1% compared with omitting the scale.The comparison uses ScanNet with and without the inverse density scale.
  • ScanNet Segmentation: PointConv outperforms other approaches by a large margin on the ScanNet benchmark.The study performs additional ablations on the public validation set because only one final result can be submitted to the benchmark server.
  • ScanNet Segmentation: Smaller sliding-window strides improve segmentation results, while adding RGB information does not significantly improve ScanNet performance.PointConv still outperforms baselines by a large margin without these additional improvements.
  • Learned Filters: Visualized filters sampled on the z = 0 plane exhibit patterns in the learned continuous convolutional filters.The filters are generated by the MLPs in PointConv.

7. Conclusion

PointConv performs convolution directly on 3D point clouds by learning continuous filters from local coordinates and using an efficient implementation. The resulting networks are translation- and permutation-invariant, scalable, and competitive across challenging benchmarks.

  • Conclusion: PointConv uses MLPs on local point coordinates to approximate continuous weight and density functions in convolutional filters.This design supports convolution directly on 3D point clouds.
  • Conclusion: An efficient implementation greatly improves PointConv’s scalability for building deep convolutional networks on point clouds.The paper presents this implementation as a key part of making PointConv practical at larger scale.
  • Conclusion: The learned convolution is translation-invariant and permutation-invariant, allowing deep networks to operate on unordered 3D point sets.These invariances parallel key capabilities of grid-based image convolution while addressing point-cloud ordering.
  • Conclusion: PointConv demonstrates strong performance on multiple challenging benchmarks and matches the performance of a grid-based convolutional network on 2D images.The paper identifies adapting architectures such as ResNet and DenseNet to point clouds as future work.
Loading 1811.07246v3…