PaperlayerView exact revision on arXiv

Source-linked AI summary

Flex-Convolution (Million-Scale Point-Cloud Learning Beyond Grid-Worlds)

Fabian Groh, Patrick Wieschollek, Hendrik P. A. Lensch

arXiv:1803.07289v4cs.CV

TL;DR

Irregular point clouds do not fit the regular-grid assumptions behind conventional convolutions, limiting the direct use of image-learning architectures. The paper generalizes convolution for irregular points and reports improved large-scale segmentation alongside competitive small-benchmark results.

  • Problem

    Conventional convolutions rely on regular grids, whereas point clouds have irregular neighborhoods and do not readily support the same image-learning design choices.

    Source: §1 · p. 1

  • Method

    Flex-convolution generalizes convolution to irregular point sets using sparse local-neighborhood operations, with an efficient GPU implementation.

    Source: §3.1 · p. 6, §6 · p. 13

  • Results

    The approach improves large-scale real-world point-cloud segmentation and is competitive on smaller benchmarks using fewer parameters and less memory.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The layers behave similarly to image convolutions, allowing established network architectures to be applied to irregular point sets.

    Source: §6 · p. 13

    The method focuses on static scans with neighborhoods computed during data pre-fetching; dynamic point clouds require an approximate nearest-neighborhood layer.

    Source: §5.2 · p. 13

Abstract

from arXiv · show

Traditional convolution layers are specifically designed to exploit the natural data representation of images -- a fixed and regular grid. However, unstructured data like 3D point clouds containing irregular neighborhoods constantly breaks the grid-based data assumption. Therefore applying best-practices and design choices from 2D-image learning methods towards processing point clouds are not readily possible. In this work, we introduce a natural generalization flex-convolution of the conventional convolution layer along with an efficient GPU implementation. We demonstrate competitive performance on rather small benchmark sets using fewer parameters and lower memory consumption and obtain significant improvements on a million-scale real-world dataset. Ours is the first which allows to efficiently process 7 million points concurrently.

1 Introduction

Grid-based convolutions are poorly suited to irregular point clouds, while existing point-cloud methods struggle with million-scale inputs. The paper introduces flex-convolution and a GPU implementation, reporting gains on large-scale segmentation and competitive results on smaller benchmarks.

1 Introduction

  • Grid-based methods face a resolution trade-off between discretization artifacts and memory use, which grows exponentially with dimensionality.Source: §1 · p. 1
  • Voxel networks remain limited to 256^3 voxels, while irregular point-cloud methods have lacked support for million-scale inputs.Source: §1 · pp. 1–2
  • Up to 7 million points can be processed in one forward pass at the same accuracy, compared with previous methods’ limit of 1 million.Source: Figure 1 · p. 2
  • The authors hypothesize that a simple convolution with few learnable parameters suits deeper networks and introduce a generalized convolution with a tuned GPU implementation.Source: §1 · p. 2
  • The approach improves large-scale point-cloud segmentation and offers competitive small-benchmark results with fewer parameters and less memory.Source: §1 · p. 3

2 Related Work

Prior point-cloud learning approaches handle irregular data through voxelization, graph structure, or direct processing in Euclidean space, each with distinct constraints. Flex-convolution is presented as a scalable Euclidean-space approach that evaluates points and features together.

2 Related Work

  • Voxel methods either lose spatial information through discretization or require substantial computation, and higher-dimensional extensions face the curse of dimensionality.Source: §2 · p. 3
  • Graph-based sparse convolution uses adjacency to mask intermediate outputs, mimicking information diffusion across layers.Source: §2 · p. 3
  • Euclidean-space methods retain absolute positions; PointNet aggregates point features globally, while PointNet++ [19] adds local neighborhoods across resolutions.Source: §2 · p. 3
  • Flex-convolution belongs to the Euclidean-space category and is described as a scalable special case of that evaluates points and features in one go.Source: §2 · p. 4

3 Method

The method extends grid-based convolution to irregular data with real-valued locations, using local neighborhoods to define operations on point sets. It also introduces a scalable sub-sampling strategy for irregular data.

3 Method

  • The method generalizes convolution to point sets by applying an operation over each point’s k nearest neighbors.Each point carries an explicit location and feature value.Source: §3.1 · p. 5

3.1 Convolution Layer

  • A learnable linear kernel supports arbitrary relative positions, is differentiable in point locations, and reduces trainable parameters while retaining translation invariance.The formulation is presented as a linear approximation of the grid-based lookup table.Source: §3.1 · pp. 5–6

3.2 Extending Sub-Sampling to Irregular Data

  • IDISS samples points in O(n) time, using summed distances to k-neighbors as an inverse-density proxy.Compared with random sampling, it produces more uniformly distributed coarser points and is more likely to preserve important areas and overall structure.Source: §3.2 · p. 7

4 Implementation

The implementation supplies TensorFlow layers for flex-convolution and flex-max-pooling, using fixed k-nearest-neighbor indices and tuned CUDA operations. Its large-scale semantic-segmentation network adapts an encoder-decoder design with residual blocks, subsampling, and skip connections.

4.1 Neighborhood Processing

  • For each fixed point set, an efficient kD-tree computes each point’s k nearest-neighbor indices once, and the indices are passed to layers as a tensor.Neighborhoods for subsampled resolutions can be computed during parallel data pre-fetching at negligible cost relative to a forward-and-backward pass under optimal GPU utilization.Source: §4.1 · pp. 7–8
  • Flex-convolution looks up neighboring feature vectors and locations by index, attaches point locations to features, and parallelizes the lookup because points have equal neighbor counts.Flex-max-pooling takes a neighborhood-wise maximum without subsampling; subsampling is applied after pooling, and flex-upsampling copies selected-point features into a larger layer while zero-initializing other points.Source: §4.1 · pp. 7–8

4.2 Efficient Implementation of Layer Primitives

  • A hand-tuned CUDA implementation provides the required GPU operations; the TensorFlow-only implementation required data duplication and was spread across 8 GPUs for one layer.Profiling compared implementations for batches of 8 × 4096 points, C′ = C = 64, and 9 neighbors; the authors note cuDNN uses more memory but has faster back-propagation than flex-max-pooling.Source: §4 · p. 7, §4.2 · pp. 8–9, Table 1 · p. 8

4.3 Network Architecture for large-scale Semantic Segmentation

  • The network follows SegNet-Basic with U-net skip connections and a final point-wise softmax, while omitting batch normalization, weighted softmax, and costly pre- or post-processing.The authors state those omitted techniques could further improve the reported predictions.Source: §4.3 · p. 9
  • The six-stage encoder processes multiscale point-cloud features with two residual blocks per stage, while the mirrored decoder uses skip connections and flex-upsampling.Each encoder stage reduces point resolution by a factor of four and doubles feature length as resolution decreases.Source: §4.3 · pp. 9–10, Figure 4 · p. 9
  • A maximum neighborhood size of k = 8 was chosen for speed; on 2D-3D-S, k = 16 showed no accuracy decrease but ran 2.2 times slower.The authors report no significant improvement from preliminary trainable flex-transposed-convolution experiments and prefer flex-max-pooling as a lighter operation.Source: §4.3 · p. 10

5 Experiments

Experiments evaluate flex-convolution on synthetic classification and segmentation benchmarks and a large real-world scan. The reported results include competitive synthetic performance and improved large-scale segmentation accuracy and inference speed.

5.2 Real-World Semantic Point Cloud Segmentation

  • The real-world evaluation uses the 2D-3D-S scan dataset, covering 270 Mio. points.The dataset comprises scans from six square kilometers of building complexes.Source: §5 · p. 10, §5.2 · p. 12

5.1 Synthetic Data

  • ModelNet40 classification is competitive with fewer parameters, and remains competitive using one quarter of the points and a smaller network.The paper also reports a human study based on 2,682 classification votes and notes that some dataset categories can be difficult to distinguish.Source: §5.1 · pp. 10–11, Table 2 · p. 10
  • ShapeNet part segmentation improves average mIoU and processes a magnitude more shapes per second than compared methods.The experiments use 2,048 points per object; the authors say the results support simple filters with few learnable parameters in deeper networks.Source: §5.1 · p. 11, Table 3 · p. 11

5.2 Real-World Semantic Point Cloud Segmentation

  • 0.31, 0.39, and 0.50 mAP result from constant features, normalized position, and position plus color, respectively.Raw single-pass predictions outperform previous approaches with the same available information, but lack precision for categories including beam, column, and door; full-resolution processing benefits smaller objects.Source: §5.2 · p. 12, Table 4 · p. 13
  • 7 Mio. points run concurrently in 4.7 seconds at the same inference performance; processes at most 1 Mio. points in 7.1 seconds.The authors report exponential runtime growth for the PointNet family and linear runtime growth for their method; accuracy from competing methods was not meaningful above 4,096 points.Source: §5.2 · p. 13
  • The method focuses on static point-cloud scans; dynamic completion or generation requires an approximate nearest-neighborhood layer.A prototype takes around 1 second for 2 million points, reported as eight times faster than the kd-tree used in parallel pre-fetching.Source: §5.2 · p. 13

6 Conclusion

The paper extends convolution, transposed convolution, and max-pooling to irregular point sets, with state-of-the-art results on small synthetic and large real-world datasets. Its layers use local neighborhoods and process millions of points concurrently and efficiently.

6 Conclusion

  • The method achieves state-of-the-art results on small synthetic and large real-world datasets while processing millions of points concurrently and efficiently.Source: §6 · p. 13
  • The sparse operations use each point’s local neighborhood, specified by its k nearest neighbors, to process irregular point sets.Source: §6 · p. 13
  • Because the layers behave similarly to 2D image convolutions, successful image-network architectures can be leveraged without relying on hand-crafted input features or heavy pre- and post-processing.Source: §6 · p. 13
  • Figure 6 shows raw semantic-segmentation outputs on the held-out validation set, with surfaces not necessarily rendered as opaque.Source: Figure 6 · p. 14

Related papers

Loading 1803.07289v4…