Source-linked AI summary
Deep Parametric Continuous Convolutional Neural Networks
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, Raquel Urtasun
TL;DR
Standard CNNs are limited to grid-structured inputs, leaving many non-grid applications unsupported. The paper introduces Parametric Continuous Convolution, which learns continuous kernel functions for arbitrary structures with computable support relationships, and reports state-of-the-art results across point-cloud tasks.
Problem
Standard convolutional operators depend on dense grid structure and cannot be directly applied to many non-grid data applications such as point clouds.
Method
Parametric Continuous Convolution models continuous kernels with MLPs, supports arbitrary output points, and uses locality and factorization to improve efficiency.
Results
The proposed network achieves state-of-the-art performance across point-cloud segmentation and motion-estimation tasks and datasets.
Takeaways & Limitations
The operator extends deep convolutional learning to arbitrary structured data when support relationships between elements are computable.
Takeaways & Limitations
Indoor qualitative results include errors caused by visually ambiguous textures and labels, while very deep networks can face GPU memory limitations.
Abstract
from arXiv · showhide
Standard convolutional neural networks assume a grid structured input is available and exploit discrete convolutions as their fundamental building blocks. This limits their applicability to many real-world applications. In this paper we propose Parametric Continuous Convolution, a new learnable operator that operates over non-grid structured data. The key idea is to exploit parameterized kernel functions that span the full continuous vector space. This generalization allows us to learn over arbitrary data structures as long as their support relationship is computable. Our experiments show significant improvement over the state-of-the-art in point cloud segmentation of indoor and outdoor scenes, and lidar motion estimation of driving scenes.
1. Introduction
The paper addresses the mismatch between grid-based convolutions and non-grid data such as point clouds, proposing a continuous operator for arbitrary structures. Experiments report strong results across segmentation and motion estimation.
- Non-grid applications including 3D point-cloud perception, mesh registration, and non-rigid shape correspondence cannot directly use standard convolutional operators.
- Voxelization enables discrete convolutions but wastes memory and computation because most of the volume is typically empty.
- The proposed parametric continuous convolution uses parameterized kernel functions spanning the continuous vector space to handle arbitrary data structures with computable support relationships.
- The resulting networks are described as expressive and memory efficient, while enabling very deep networks to be learned end-to-end over raw point clouds.
- The method outperforms the state-of-the-art by a large margin on indoor and outdoor point-cloud segmentation and lidar motion estimation.The experiments include a dataset containing 223 billion points, reported as two orders of magnitude larger than existing benchmarks.
2. Related Work
Related work adapts convolutional or message-passing methods to geometric and graph-structured data, but faces representation, memory, scalability, or generalization challenges. Parametric continuous convolution instead operates over arbitrary continuous-domain points without a fixed graph structure.
- Deep Learning for 3D Geometry: RGB-depth projections and voxelized 3D convolutions simplify geometric processing but can lose true 3D relationships, precision, or memory efficiency.
- Deep Learning for 3D Geometry: PointNet learns independently over points and aggregates global information through pooling, representing an early direct approach to point-cloud learning.
- Graph Neural Networks: Graph neural networks propagate information through nodes and edges, while graph convolution methods use spectral or spatial formulations.
- Graph Convolution Networks: Spectral graph convolutions are not feasible for large-scale data because generic graphs lack an FFT-like computational shortcut.
- Other Approaches: Unlike fixed-graph approaches, parametric continuous convolution can output features at arbitrary continuous-domain points.
- Other Approaches: Compared with a concurrent method using shallow isotropic Gaussian kernels, this approach uses expressive deep networks to parameterize continuous filters.
3. Deep Parametric Continuous CNNs
Parametric continuous convolutions replace grid-bound discrete kernels with finite-parameter functions over continuous support, enabling deep networks for non-grid data. The resulting layers can use differing input and output point sets, locality constraints, and differentiable training components.
- Parametric Continuous Convolutions: Continuous convolutions generalize discrete convolutions by defining features and kernels over continuous domains rather than finite integer grids.
- Parametric Continuous Convolutions: Monte-Carlo integration approximates the convolution at arbitrary output points from finitely sampled input points.
- Parametric Continuous Convolutions: MLPs parameterize continuous kernels, spanning the full support domain with finitely many learnable parameters.
- Parametric Continuous Convolution Layer: Unlike standard convolutions, each layer may take distinct input and output point sets, supporting dense prediction and pooling over redundant observations.
- Deep Parametric Continuous CNNs: Deep networks stack these layers with batch normalization, nonlinearities, residual connections, pooling, and end-to-end back-propagation.
- Discussions: Locality is enforced with a modulating window that retains either K-nearest neighbors or points within radius r, reducing the support used by the layer.
4. Experimental Evaluation
The experiments evaluate continuous convolution networks on indoor and outdoor point-cloud labeling and lidar motion estimation, including large-scale datasets and cross-model comparisons.
- Datasets: 137 billion and 629 million points respectively are contained in the outdoor and indoor datasets used for semantic labeling.
- Indoor semantic segmentation: 9.3% mIOU and 9.6% mACC are the reported improvements over the state of the art on indoor semantic segmentation.
- Driving-scene semantic segmentation: 3D-FCN+PCCN achieves the best driving-scene segmentation performance, correcting several 3D-FCN oversegmentation errors with high confidence.
- Model size: The end-to-end continuous convolution network is eight times smaller than 3D-FCN while achieving comparable results, and 3D-FCN+PCCN is 0.01MB larger with improved mean IOU.
- Runtime: A forward pass takes 33ms, neighbor search takes 28ms, and end-to-end computation takes 61ms for the benchmarked 8-layer PCCN model.
- Lidar flow: 0.351cm lower end-point error and approximately 20% fewer outliers are reported for 3D-FCN+PCCN than 3D-FCN in lidar flow.
5. Conclusions
The paper concludes that parametric continuous convolution enables deep learning on non-grid structured data and achieves state-of-the-art results across point-cloud tasks and datasets.
- The proposed convolution layer operates over non-grid structured data using a parameterized kernel function spanning the full continuous domain.
- The resulting architecture applies to arbitrary structured data when support relationships between elements are computable.
- The network achieves state-of-the-art performance on point-cloud segmentation and motion-estimation tasks across very large-scale datasets.
A. Generalization
The supplementary experiments test models across datasets and road settings without retraining, and report good generalization under all tested configurations.
- Cross-dataset transfer: The semantic labeling network is trained on North American driving scenes and tested on KITTI scenes captured in Europe.
- Cross-setting transfer: The semantic labeling network is trained on non-highway roads and tested on a lidar sequence from a highway-driving truck.
- Cross-dataset transfer: The lidar flow network is trained on North American driving scenes and tested on KITTI without retraining.
- Observed generalization: Under all tested settings, the algorithm generalizes well, including moving, turning, and stationary ego-car scenarios on KITTI.
B. Lidar Flow Data and Analysis
The paper constructs lidar-flow ground truth from vehicle and object motion, analyzes its distribution, and validates labels through cross-frame point alignment.
- Ground-truth Generation: The ground-truth flow at frame k is defined for each point using its ego-car-centered locations and the vehicle pose transform.The formulation connects point positions across frames through the estimated motion.
- Ground-truth Generation: Ground-truth lidar flow combines global vehicle pose transforms with rigid-motion labels for dynamic objects.Dynamic-object labeling accounts for both ego-motion and the objects’ own motion.
- Ground-truth Motion Analysis: Ground-truth motion differs across static background, vehicles, and pedestrians, with forward ego-car motion forming the heaviest distribution along the y-axis.The analysis uses 2D histograms of translation components along the x and y axes.
- Ground-truth Validation and Visualization: Vehicle points align perfectly after warping across frames, while pedestrian discrepancies are near-perfect and attributed to non-rigid posture changes.The labels are validated by overlaying target points with source points warped by ground-truth motion.
C. More Results
Additional qualitative results on the Stanford dataset show that the model usually predicts semantic labels correctly.
- Additional Qualitative Results: Qualitative results on the Stanford dataset show correct semantic-label predictions in most cases.These results are presented as additional evaluations across the paper’s tasks.
C.2. Semantic Segmentation for Driving Scenes
Driving-scene semantic-labeling results capture small dynamic objects and distinguish road from non-road using lidar intensity and subtle geometric structure.
- Semantic Segmentation for Driving Scenes: The model captures small dynamic objects such as pedestrians and bicyclists in driving-scene semantic labeling.The authors identify this as potential for object detection and tracking.
- Semantic Segmentation for Driving Scenes: The model distinguishes road from non-road using lidar intensity and subtle geometry such as road curbs.The authors connect this behavior to potential map automation.
- Semantic Segmentation for Driving Scenes: Most semantic-labeling errors occur at road boundaries.These errors appear as bright curves in the error map.
C.3. Lidar Flow
Additional lidar-flow results show that the model captures most of the flow field, while errors concentrate at object boundaries.
- Lidar Flow: The model captures the majority of the lidar flow field in additional qualitative results.The visualization uses colored vectors to depict motion-vector magnitudes.
- Lidar Flow: Most lidar-flow errors occur at object boundaries.The authors suggest that adding space and intensity features to the support domain could improve performance.
- Lidar Flow: Early PCCN layers capture low-level geometry, while deeper layers capture semantic structures such as road curbs and dynamic objects.Activation maps compare layer 1 with layer 8 of the network.
E. Point Cloud Classification
The paper evaluates parametric continuous convolution on ModelNet40 point-cloud classification using randomly sampled points from CAD meshes. Its performance is comparable with PointNet and slightly below PointNet++, with global max pooling identified as a possible limitation.
- Experimental setup: ModelNet40 contains CAD models from 40 categories and is used to evaluate global point-cloud classification.The authors randomly sampled 2048 points from each training and testing mesh.
- Experimental setup: The method processes 2048 sampled points from each ModelNet40 mesh for training and testing.The point clouds are fed into the proposed neural network.
- Results: Performance is comparable with PointNet and slightly below PointNet++.These representative ModelNet40 algorithms are used as comparison points.
- Results: The method uses naive global max pooling to aggregate global information.The authors expect more comprehensive and hierarchical pooling strategies could improve results.