PaperlayerView exact revision on arXiv

Source-linked AI summary

Point-Voxel CNN for Efficient 3D Deep Learning

Zhijian Liu, Haotian Tang, Yujun Lin, Song Han

arXiv:1907.03739v2cs.CV

TL;DR

Voxel and point-based 3D models face complementary efficiency limitations: voxel methods scale cubically in memory, while point methods incur irregular-access overhead. PVCNN represents data sparsely as points but performs convolutions in regular voxel grids, achieving strong accuracy and efficiency across multiple tasks.

  • Problem

    Voxel models lose information at low resolution and grow cubically in memory, while point models reduce memory but suffer irregular access and computation overhead.

  • Method

    PVCNN uses sparse point representations for input data and regular voxel-domain convolutions, then maps voxel features back to points.

  • Results

    PVCNN consistently improves efficiency and accuracy across segmentation and detection tasks, including 10× lower memory than a voxel baseline and 7× average measured speedup over point-based models.

  • Takeaways & Limitations

    Co-designing point and voxel representations can provide both sparse memory usage and regular convolutional access for fast 3D deep learning.

Abstract

from arXiv · show

We present Point-Voxel CNN (PVCNN) for efficient, fast 3D deep learning. Previous work processes 3D data using either voxel-based or point-based NN models. However, both approaches are computationally inefficient. The computation cost and memory footprints of the voxel-based models grow cubically with the input resolution, making it memory-prohibitive to scale up the resolution. As for point-based networks, up to 80% of the time is wasted on structuring the sparse data which have rather poor memory locality, not on the actual feature extraction. In this paper, we propose PVCNN that represents the 3D input data in points to reduce the memory consumption, while performing the convolutions in voxels to reduce the irregular, sparse data access and improve the locality. Our PVCNN model is both memory and computation efficient. Evaluated on semantic and part segmentation datasets, it achieves much higher accuracy than the voxel-based baseline with 10x GPU memory reduction; it also outperforms the state-of-the-art point-based models with 7x measured speedup on average. Remarkably, the narrower version of PVCNN achieves 2x speedup over PointNet (an extremely efficient model) on part and scene segmentation benchmarks with much higher accuracy. We validate the general effectiveness of PVCNN on 3D object detection: by replacing the primitives in Frustrum PointNet with PVConv, it outperforms Frustrum PointNet++ by 2.4% mAP on average with 1.5x measured speedup and GPU memory reduction.

1 Introduction

Efficient 3D models are needed for real-time applications on resource- and battery-constrained edge devices. PVCNN combines sparse point representation with voxel convolution to reduce memory consumption and improve efficiency.

  • Motivation: Real-time AR/VR and autonomous-driving applications require low-latency 3D deep learning on hardware-constrained edge devices.Mobile phones and VR headsets have limited hardware resources and battery capacity.
  • Voxel-based models: High-resolution voxel processing preserves detail but incurs computational cost and memory requirements that grow cubically with voxel resolution.Low-resolution voxelization merges multiple points and loses information.
  • Point-based models: Point-based models use sparse representations to reduce GPU memory, but irregular memory access, nearest-neighbor search, and dynamic kernels create substantial overheads.Neighbors are not stored contiguously, and dynamic kernels depend on varying relative positions.
  • PVCNN: PVCNN represents inputs as point clouds for sparsity while using voxel-based convolution for contiguous memory access.This design targets both memory footprint and access-pattern inefficiency.
  • Results: 10× lower memory consumption and 7× measured average speedup are reported against voxel-based and state-of-the-art point-based models, respectively.The comparisons are reported across multiple tasks.

2 Related Work

Prior 3D deep-learning research includes voxel-based, point-based, and task-specific models. These approaches differ in representation, neighborhood modeling, and efficiency objectives.

  • Voxel-Based 3D Models: Voxel-based methods process 3D data volumetrically, with later work using octrees and applying volumetric models to segmentation and object detection.The cited voxel-based literature includes vanilla volumetric CNNs and 3D extensions of 2D CNNs.
  • Point-Based 3D Models: PointNet processes unordered point sets with symmetric functions, while later methods stack PointNets hierarchically to model neighborhoods and increase capacity.This line of work focuses on point-cloud inputs and neighborhood structure.
  • Point-Based 3D Models: Other point-based methods use dynamically generated kernels or learned neighborhood permutation functions instead of stacked PointNet blocks.SPLATNet and SONet provide additional approaches for general-purpose point-cloud modeling.
  • Special-Purpose 3D Models: Special-purpose models target semantic or instance segmentation, while methods such as F-PointNet, PointRCNN, PointPillars, and SECOND target 3D object detection or efficiency.The related work spans task-specific architectures rather than one unified model family.

3 Motivation

Voxel and point representations expose complementary efficiency problems: voxels trade detail for rapidly growing memory, while points preserve sparsity but incur irregular access and dynamic-kernel overhead. These issues motivate combining the two representations.

  • Convolution formulation: 3D data consists of coordinates p_k and corresponding features f_k for points or voxel grids.The representation is written as x = {x_k} = {(p_k, f_k)}.
  • Convolution formulation: A convolution indexes neighbors N(x_k), applies kernel K(x_k, x_i) to neighboring features F(x_i), and produces output y_k for each center.This formulation covers both voxel-based and point-based convolution.
  • Voxel-Based Models: Large Memory Footprint: 64-resolution voxel processing loses 42% of points, while preserving more than 90% requires resolution 128 and 82.6 GB, or 7.2× the GPU memory.The affordable resolution is measured on a single 12 GB GPU during training with batch size 16.
  • Point-Based Models: Point-based methods save memory but lose local-context efficiency through irregular memory access and dynamic kernel computation.These overheads arise when neighborhood information is aggregated in the point domain.
  • Irregular Memory Access: Nearest-neighbor identification and neighbor gathering require expensive KNN computation and large amounts of random memory access.Point neighborhoods are scattered in 3D and are not laid out contiguously in memory.
  • Dynamic Kernel Computation: Unpredictable point offsets force point-based convolutions to calculate kernels for each neighbor on the fly.Volumetric convolutions can directly index kernels because their relative neighbor positions are fixed.
  • Efficiency Bottlenecks: 55% to 88% of computation is attributed to combined irregular-access and dynamic-kernel overhead in DGCNN and PointCNN.The reported range indicates substantial computation is spent handling point-representation irregularity.

4 Point-Voxel Convolution

PVConv combines low-resolution voxel-based neighborhood aggregation with high-resolution point-based feature transformation. This design preserves fine-grained point information while improving memory efficiency and data locality.

  • PVConv design: PVConv combines point-based feature transformation with voxel-based neighborhood aggregation to exploit their complementary strengths.The voxel branch processes coarse-grained neighborhood information, while the point branch retains fine-grained individual features.
  • Voxel-based feature aggregation: PVConv normalizes point coordinates, voxelizes point features by averaging points within each grid, and applies volumetric convolutions for feature aggregation.The voxelized representation uses a low resolution, avoiding a very large memory footprint.
  • Voxel-based feature aggregation: Trilinear devoxelization maps voxel features back to points while keeping features distinct for points sharing a voxel.The voxelization and devoxelization operations are differentiable, enabling end-to-end optimization of the aggregation branch.
  • Efficiency: 16× to 64× fewer noncontiguous memory accesses are empirically achieved than conventional point-based methods through one-pass voxelization and devoxelization.PVConv requires O(n) random memory accesses, compared with at least O(kn) for gathering k neighbors, and avoids KNN and dynamic kernel computation.
  • Results: 5.5× measured speedup and 3× memory reduction are achieved over point-based models on ShapeNet Part, while the voxel-based baseline is surpassed by 59× speedup and 11× memory reduction.These are average results for object part segmentation reported in Table 1.
  • Effectiveness: 25.7 ms latency and 1.0 GB memory consumption are required by PVConv versus 75.2 ms and 3.6 GB for PointNet++ under the stated setup.At comparable latency, PointNet++ must downsample from 2048 to 685 points, whereas PVConv models the full point cloud.

5 Experiments

Experiments across part segmentation, indoor scene segmentation, and 3D object detection show PVCNN consistently improves accuracy while reducing latency and GPU memory consumption. The evaluations also examine edge-device speed, voxel-resolution scaling, complementary branches, and ablations.

  • Object Part Segmentation: PVCNN outperforms previous models on object part segmentation, improving PointNet accuracy by 2.5% with smaller overhead than PointNet++.
  • Object Part Segmentation: 10× lower GPU memory accompanies better accuracy than 3D-UNet, while PVCNN is 15× faster than SpiderCNN and 2.7× faster than PointCNN at similar accuracy.
  • Object Part Segmentation: 2× speedup over PointNet and PointCNN is measured across three edge devices, reaching 19.9 objects per second on Jetson Nano and 20.2 on Jetson Xavier.
  • Analysis: PVCNN memory rises only 1.03× when voxel resolution increases from 16 to 32, while retaining individual-point information through its high-resolution point branch.
  • Visualization: The voxel branch captures large continuous structures, whereas the point branch captures isolated discontinuous details, providing complementary information.
  • Indoor Scene Segmentation: PVCNN++ exceeds PointCNN by 1.7% mIoU with 4× lower latency and exceeds 3D-UNet by 4% mIoU with more than 8× lower latency and GPU memory consumption.
  • Indoor Scene Segmentation: Narrower PVCNN achieves 4% higher mIoU than PointNet while running 2.5× faster, and delivers 15× and 9× measured speedups over DGCNN and RSNet.
  • 3D Object Detection: PVCNN-based models improve F-PointNet baselines on KITTI, with up to 8% mAP improvement for pedestrians and 3.5–6.8% for cyclists.

6 Conclusion

The conclusion presents PVCNN as a fast, efficient 3D deep-learning approach that combines sparse point representation with regular voxel convolution. Across multiple tasks, the method consistently demonstrates effectiveness and efficiency.

  • PVCNN combines sparse point representation for lower memory footprint with dense voxel convolution for more regular memory access.
  • Extensive experiments across multiple tasks consistently demonstrate PVCNN’s effectiveness and efficiency.
Loading 1907.03739v2…