PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvPoint: Continuous Convolutions for Point Cloud Processing

Alexandre Boulch

arXiv:1904.02375v5cs.CV

TL;DR

Point clouds are unordered and lack the grid structure required by standard discrete CNNs. ConvPoint replaces discrete kernels with continuous convolutions and builds hierarchical networks for point-cloud tasks, achieving competitive state-of-the-art results across classification and segmentation. The framework is designed to handle varying point-cloud sizes, but practical training choices and raw-input features constrain its current scope.

  • Problem

    Point clouds are unordered and generally lack the regular spatial structure required by image CNNs’ discrete convolutions.

  • Method

    ConvPoint generalizes discrete convolution with a learned continuous geometric weighting function and a search-tree-based hierarchical representation for point-cloud networks.

  • Results

    ConvPoint is competitive with the state of the art across shape classification, part segmentation, and large-scale semantic segmentation.

  • Takeaways & Limitations

    The formulation supports flexible network architectures for classification, part segmentation, and large-scale semantic segmentation of sparse, unstructured point clouds.

  • Takeaways & Limitations

    The study uses only raw inputs—RGB when available and otherwise constant features—and leaves normals or curvatures for future work.

Abstract

from arXiv · show

Point clouds are unstructured and unordered data, as opposed to images. Thus, most machine learning approach developed for image cannot be directly transferred to point clouds. In this paper, we propose a generalization of discrete convolutional neural networks (CNNs) in order to deal with point clouds by replacing discrete kernels by continuous ones. This formulation is simple, allows arbitrary point cloud sizes and can easily be used for designing neural networks similarly to 2D CNNs. We present experimental results with various architectures, highlighting the flexibility of the proposed approach. We obtain competitive results compared to the state-of-the-art on shape classification, part segmentation and semantic segmentation for large-scale point clouds.

1 Introduction

Point clouds lack the regular grid and ordering that image CNNs require, motivating a direct CNN generalization for unstructured data. ConvPoint introduces continuous convolutions and uses them to build networks for classification and segmentation, with competitive results across tasks.

  • Motivation: Point clouds are generally unordered, lack spatial structure, and may contain only relative positions rather than color features.They are sparse samplings that cannot be placed on a regular grid like image pixels.
  • Motivation: Image CNNs cannot be transferred directly because discrete convolutions depend extensively on grid structure.The paper identifies voxel projection and CNN reformulation as two common ways to address this mismatch.
  • Contributions: ConvPoint introduces a continuous convolution designed for unstructured data as a straightforward extension of discrete convolution.The formulation is intended to replace the grid-dependent discrete kernel.
  • Contributions: The authors use continuous convolutions with a search-tree-based hierarchical representation to design networks similarly to image-processing architectures.This framework supports the paper’s classification and segmentation experiments.
  • Results: ConvPoint is applied to classification and segmentation, including large-scale indoor and outdoor semantic segmentation, with competitive state-of-the-art performance.The paper reports applicability across various tasks rather than a single benchmark.

2 Related work

Prior point-cloud methods either structure data for conventional CNNs or operate directly on unstructured points, graphs, or meshes. ConvPoint belongs to the direct-processing family and differs from related approaches in how it represents geometry and learns convolutional weights.

  • Approaches: Point-cloud learning methods include handcrafted descriptors, voxel or image projections, geometric deep learning, and direct processing of raw point clouds.The paper’s method belongs to the direct raw-point-cloud category.
  • Grid-based methods: Voxel-based 3D CNNs can be time-consuming, require relatively low resolution, and introduce information loss or grid-axis bias.Sparse convolutions and spatial scanning are described as alternatives for reducing these drawbacks.
  • Grid-based methods: Multi-view methods apply 2D CNNs to projected shape representations but require an efficient and robust strategy for choosing viewpoints.They avoid direct 3D convolutions by projecting results back to 3D.
  • Geometric deep learning: Graph and manifold methods often target graphs, meshes, or surface analysis rather than raw point clouds.Building a mesh from raw point clouds is difficult and requires priors about the surface being reconstructed.
  • Direct point processing: Unlike PointNet and PointNet++, ConvPoint does not use spatial coordinates as input features; geometry instead weights relations between kernel and input points.Its geometry-feature interaction differs from methods that explicitly feed input geometry as features.
  • Direct point processing: Compared with a related MLP-weighted convolution, ConvPoint learns a dense whole-kernel weighting relation using an explicit point-associated kernel.The kernel-to-input relation is learned with an MLP rather than specified by an explicit Gaussian RBF.

3 Convolution for point processing

ConvPoint generalizes discrete convolution to unordered point sets by replacing exact location matching with a learned geometric weighting function. Relative coordinates, permutation-invariant aggregation, normalization, and learnable kernel locations provide the point-cloud formulation.

  • Discrete convolution: The discrete formulation assumes kernel and input elements share locations and uses an indicator function to establish one-to-one correspondence.This assumption is suitable for grid-sampled data such as images.
  • Point-set convolution: For arbitrary point locations, exact indicator matching is almost always zero, so it is too restrictive for point-cloud processing.The paper therefore replaces exact matching with a more general relation between input points and kernel elements.
  • Point-set convolution: The geometric weighting function distributes input features onto kernel elements and is applied independently to points to preserve permutation invariance.The function uses relative positions between input points and kernel elements.
  • Point-set convolution: The point-set convolution mixes spatial and feature domains, while spatial coordinates are used to weight relations rather than treated as input features.A normalization by input-set size improves robustness to varying input sizes.
  • Learned weighting: A simple MLP learns the geometric weighting function instead of imposing a handcrafted Gaussian form and its parameters.This avoids a specific assumption about the function’s behavior.
  • Learned weighting: Kernel spatial locations are sampled in the unit sphere and optimized jointly with kernel weights and MLP parameters by gradient descent.The kernel has separate spatial and feature components, producing a vector output for multiple kernels.
  • Invariances: Summation over input points makes the output permutation invariant, while relative coordinates provide translation invariance.Normalizing input points to the unit ball improves insensitivity to scale.
  • Invariances: Dividing the output by input-set size reduces sensitivity to point-cloud size, so duplicating the input set does not change the result.This is the stated effect of the normalization.

4 Hierarchical representation and neighborhood computation

The convolution layer maps an input point cloud to an output point cloud by selecting output locations, finding local neighborhoods, and computing features for each neighborhood. Its output cardinality can preserve, reduce, or increase the input size.

  • Layer mapping: The layer projects input point cloud P onto output locations {q} and computes features {y} for the resulting pairs Q = {(q, y)}.The operation is local on subsets of the input point cloud.
  • Cardinality: Equal input and output cardinalities correspond to convolution without stride, while |P| > |Q| performs spatial size reduction.The reduction parallels a discrete convolution with stride greater than one.
  • Cardinality: When |P| < |Q|, the layer produces an upsampled input version analogous to an up-convolutional layer.This is the third cardinality behavior described for the layer.
  • Point selection: If output locations are not supplied, the method selects them iteratively using neighborhood-aware selection scores.A selected point receives a large score increase, while points in its neighborhood receive smaller increases.
  • Neighborhood computation: For each output point, local neighborhoods are computed with a kd-tree before applying the convolution and assembling the output pairs.The kd-tree is built from the input points.

5 Convolutional layer

The convolutional layer first selects local neighborhoods and then applies the convolution to produce output features. Its parameters control output channels, output point-cloud size, kernel size, and neighborhood size.

  • The layer computes local neighborhoods for each selected point using a k-d tree before applying convolution.Inputs are P and optionally {q}; if {q} is absent, it is selected as a subset of P.
  • The convolution operation is applied independently to each computed neighborhood, producing output features.The output Q is the union of pairs {(q, y)}.
  • The number of output channels C determines the output feature dimension of y.
  • The output point-cloud size |Q| specifies how many points are passed to the next layer.
  • The kernel size |K| is the number of kernel elements used for convolution, while k specifies neighborhood size.Neighborhood size k is the number of points in {p} considered for each point in {q}.

6 Experiments

The experiments evaluate ConvPoint across multiple tasks using repeated spatial samplings because spatial structure generation is stochastic. The paper presents convolutional architectures whose spatial structure and point-cloud size evolve across layers.

  • Spatial structure generation is stochastic, so multiple network runs can produce different outputs.Results are aggregated by averaging outputs across runs.
  • The number of spatial samplings denotes the number of runs whose outputs are averaged.For classification and part segmentation, this number appears in parentheses in the tables.
  • Figure 3 separates each convolutional layer into spatial structure computation and convolution on local neighborhoods.
  • The classification network uses five convolutional layers with progressive point-cloud size reduction and a final fully connected layer.All layers use |K| = 16.

6.1 Classification

Classification experiments cover both 2D MNIST point clouds and 3D ModelNet40 point clouds. ConvPoint is competitive with image CNNs and point-based methods while learning from either grayscale values or geometry alone.

  • The classification experiments use both 2D and 3D point-cloud datasets.The datasets are MNIST and ModelNet40.
  • The classification network progressively reduces the point cloud to one point while increasing channels, then feeds its features to a linear layer.This architecture is similar to networks used for image processing, such as LeNet.
  • MNIST point clouds use pixel coordinates as point coordinates and are evaluated in grayscale and black-points-only variants.The first variant retains grayscale values; the second retains only black points with features set to 1.
  • MNIST scores averaged over 16 spatial samplings are competitive with image CNNs and point-based methods.The comparison includes LeNet, Network in Network, PointNet++, and PointCNN.
  • ModelNet40 experiments use point clouds randomly sampled from mesh faces with input sizes of 1024 or 2048 points.Results are reported for 3D classification and are competitive with state-of-the-art point-based classification approaches.

6.2 Segmentation

ConvPoint uses encoder-decoder networks for part and large-scale semantic segmentation, adapting network size and point selection to different point-cloud settings. Across ShapeNet and large indoor/outdoor benchmarks, it achieves competitive or leading results.

  • Segmentation network: The segmentation architecture uses an encoder that reduces point-cloud cardinality and a decoder that restores the original size with skip connections.Skip connections provide decoder point locations and concatenate encoder and decoder features at corresponding scales.
  • Segmentation network: The part-segmentation network is used on ShapeNet, while a larger variant adds three convolutions for larger input point clouds.Both variants include dropout before the final linear layer.
  • Part segmentation: ShapeNet part segmentation treats 50 annotated part classes as a semantic-segmentation problem and scores predictions at shape level.The dataset contains 16,680 models across 16 shape categories, with 2–6 parts per category.
  • Semantic segmentation: For large-scale semantic segmentation, the method samples 2,500 points and predicts labels for each selected input point, using nearest-neighbor labels for unlabeled points.Input features are set to one because the point clouds lack color features.
  • Part segmentation: ConvPoint ranks among the top five methods for both mcIoU and mIoU on ShapeNet part segmentation.Scores are aggregated across multiple stochastic network runs.
  • Semantic segmentation: Fusion improves S3DIS segmentation on 10 of 13 categories and raises average IoU by 3.5%.On Semantic8, fusion reaches 61% on artifacts versus 43–44% for either mono-modal model.
  • Semantic segmentation: On large-scale benchmarks, ConvPoint is competitive with the state of the art, placing second on S3DIS and NPM3D and first on Semantic8.The S3DIS and NPM3D second-place results are behind KPConv.

6.3 Emprical properties of the convolutional layer.

The experiments examine learned filters, stochastic spatial sampling, and robustness to changes in input and neighborhood sizes. ConvPoint supports variable point-cloud conditions, though performance is generally strongest near the training configuration.

  • Learned filters: The first convolutional layer learns weighting functions that target different regions of interest for different kernel elements.The resulting filters show varied orientations and shapes, analogously to filters in discrete CNNs.
  • Spatial sampling: Aggregating stochastic outputs across up to 16 spatial samplings improves performance, while larger sampling counts provide no significant additional benefit.Different runs can produce different sampled output point sets and predictions.
  • Input-size robustness: 85% performance is reached with 32 input points on ModelNet40, although most models perform best at their native training input size.The experiment varies training sizes from 16 through 2,048 points and tests each model across input sizes.
  • Neighborhood-size robustness: The neighborhood size remains adjustable after training, and equation (5) averages over |X| to improve robustness to neighborhood-size variation.Changing the neighborhood size also simulates point-density variation for the layer.

7 Computation times and implementation details

ConvPoint’s implementation combines PyTorch, NanoFLANN, and OpenMP, and its reported timings compare desktop and low-end hardware. Against PointCNN, it is faster for training and more memory efficient in the tested settings.

  • Implementation: The pipeline uses PyTorch for convolutional computation, NanoFLANN for neighborhoods, and OpenMP to parallelize neighborhood computation over CPU cores.
  • Timing comparison: On ModelNet40, ConvPoint training is about 30% faster than PointCNN, while inference times are similar.The comparison uses the Config. 1 workstation and PointCNN’s recommended architecture.
  • Timing comparison: On ShapeNet with batch size 4, ConvPoint training is more than 5 times faster and testing is 3 times faster than PointCNN.
  • Memory efficiency: ConvPoint uses less GPU memory than the PointCNN implementation in the reported configurations.For example, ConvPoint supports 2,048 points with batch size 128 on a GTX 1070, where PointCNN exceeds GPU memory.
  • Large-scale inference: The fusion architecture can run on the low-end configuration for large-scale segmentation with point-cloud size 8,192.The reported timings cover inference, neighborhood computation, and convolutions, excluding data loading.

8 Discussion and limitations

ConvPoint’s design choices address differing scale assumptions, neighborhood sampling, input features, and network architectures. The authors identify fixed-radius neighborhoods, richer precomputed features, and broader architecture exploration as future directions.

  • Convolutional layer: Neighborhood normalization to the unit ball makes the convolution agnostic to object scales, benefiting CAD models and photogrammetric point clouds.For metric scans, however, object sizes can provide valuable information, so normalization may be undesirable.
  • Convolutional layer: Fixed-radius neighborhoods could make features more geometric and less dependent on sampling, but variable neighbor counts would prevent current batch-training optimizations.The paper contrasts this alternative with using a fixed number of neighbors.
  • Input features: The study uses only raw inputs—RGB when available and otherwise constant-one features—leaving normals and curvatures for future work.The authors explicitly frame precomputed input features as an unexplored direction.
  • Network architecture: The two proposed architectures are widely inspired by computer-vision models, leaving further variations such as residual networks to explore.The formulation’s generalization of discrete convolution makes transposing additional CNN architectures possible.

9 Conclusion

The paper presents a flexible CNN framework that generalizes discrete convolution to sparse, unstructured point clouds. Experiments show efficient, state-of-the-art performance across classification, part segmentation, and large-scale semantic segmentation benchmarks.

  • Conclusion: ConvPoint generalizes discrete convolution for sparse and unstructured point-cloud data.The framework is presented as a new CNN approach for point-cloud processing.
  • Conclusion: The formulation is flexible and computationally efficient, supporting varied architectures for classification, part segmentation, and large-scale semantic segmentation.The paper reports experiments across real and simulated benchmarks.
  • Conclusion: ConvPoint achieves efficient, state-of-the-art performance across several point-cloud benchmarks.The conclusion summarizes results on classification, part segmentation, and large-scale semantic segmentation.
Loading 1904.02375v5…