PaperlayerView exact revision on arXiv

Source-linked AI summary

Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networks

Martin Engelcke, Dushyant Rao, Dominic Zeng Wang, Chi Hay Tong, Ingmar Posner

arXiv:1609.06666v2cs.ROcs.AIcs.CVcs.LGcs.NE

TL;DR

Large-scale 3D point-cloud object detection is computationally burdensome because the third dimension makes naive dense CNNs inefficient. Vote3Deep uses feature-centric voting and sparsity-promoting convolutional layers, achieving state-of-the-art KITTI performance across classes and difficulty levels, including an almost 40% cyclist AP margin.

  • Problem

    Naive dense CNNs are computationally burdensome for large-scale native 3D point-cloud object detection because most operations involve zeros and the third dimension increases cost.

  • Method

    Vote3Deep uses feature-centric voting for sparse 3D convolutions and an L1 penalty to encourage sparsity in intermediate representations.

  • Results

    Vote3Deep establishes state-of-the-art performance across all classes and difficulty levels on KITTI, with an almost 40% cyclist average-precision margin in the easy test case.

  • Takeaways & Limitations

    Sparse voting-based CNNs can perform fast native 3D point-cloud detection while achieving state-of-the-art KITTI results and outperforming multimodal methods in most test cases.

Abstract

from arXiv · show

This paper proposes a computationally efficient approach to detecting objects natively in 3D point clouds using convolutional neural networks (CNNs). In particular, this is achieved by leveraging a feature-centric voting scheme to implement novel convolutional layers which explicitly exploit the sparsity encountered in the input. To this end, we examine the trade-off between accuracy and speed for different architectures and additionally propose to use an L1 penalty on the filter activations to further encourage sparsity in the intermediate representations. To the best of our knowledge, this is the first work to propose sparse convolutional layers and L1 regularisation for efficient large-scale processing of 3D data. We demonstrate the efficacy of our approach on the KITTI object detection benchmark and show that Vote3Deep models with as few as three layers outperform the previous state of the art in both laser and laser-vision based approaches by margins of up to 40% while remaining highly competitive in terms of processing time.

I. INTRODUCTION

Vote3Deep addresses the computational burden of CNN-based 3D point-cloud processing by using feature-centric voting for sparse convolutions directly in 3D. It further promotes sparsity throughout the network with ReLU activations and an L1 regularization penalty.

  • Motivation: The method targets efficient, robust object detection for spatially sparse point clouds, where most regions are unoccupied and dense 3D processing is computationally burdensome.This setting is especially relevant to mobile robotics applications such as autonomous driving, where detection supports planning and decision making.
  • Contribution: Vote3Deep builds efficient CNN convolutional layers that use feature-centric voting to exploit sparsity in native 3D point-cloud processing.The approach avoids projecting inputs into lower-dimensional spaces or constraining the detector’s search space.
  • Contribution: ReLU activations and an L1 sparsity penalty encourage sparse intermediate representations, extending sparse computation throughout the CNN stack.The regularizer is imposed during training to enhance computational benefits from sparse inputs at intermediate layers.
  • Novelty: Vote3Deep is presented as the first approach combining voting-based sparse convolutional layers with L1 regularization for efficient large-scale processing of full 3D point clouds using CNNs.The authors aim to learn high-capacity, nonlinear models with constant-time test-time evaluation, unlike non-parametric methods.
  • Contribution: Vote3Deep models with as few as three layers are reported to achieve state-of-the-art performance, demonstrating the effectiveness of the proposed sparse CNN design.The supplied passage introduces this result but does not provide the comparison values or benchmark details.

II. RELATED WORK · III. METHODS

Prior work applies CNNs to projected, dense, or sparse 3D representations but is limited by information loss, computational cost, or small input sizes. Vote3Deep instead processes variable-size sparse 3D grids with voting-based convolutions, orientation handling, 3D NMS, and class-specific receptive fields.

  • II. RELATED WORK: CNNs applied to 2D projections can match on KITTI car detection, but viewpoint projection discards valuable 3D information.The approach in [7] predicts detection scores and regresses bounding boxes after adding point height as a channel.
  • II. RELATED WORK: 90s per frame follows from reported dense-grid processing rates when scaling 5ms per m3 to a 60m × 60m × 5m point-cloud volume.Related dense occupancy-grid CNNs report 6ms for a 32×32×32 crop and 5ms per m3 for landing-zone detection.
  • II. RELATED WORK: Prior sparse-convolution methods still process zero or bias-valued neighbours and target comparatively small crops or inputs.The cited methods use sparse convolutions on comparatively small 2D or 3D crops, while permutohedral-lattice convolutions likewise consider comparatively small inputs.
  • III. METHODS: Vote3Deep discretises point clouds into variable-size sparse 3D grids and extracts per-cell features from occupancy, reflectance statistics, and shape information.Each occupied cell receives a feature vector based on point statistics, including binary occupancy, mean and variance of reflectance, and three shape features.
  • III. METHODS: The network stacks voting convolutions and ReLU nonlinearities, with its output layer predicting detection scores from the resulting sparse 3D representation.This process can be repeated as in a traditional CNN.
  • III. METHODS: Parallel CNN evaluations at N angular orientations address different object orientations, while 3D NMS prunes duplicate detections and better handles overlapping objects in depth.The method uses N parallel threads and applies NMS in 3D space, where 3D bounding boxes can overlap less than their 2D projections.
  • III. METHODS: Class-specific networks use different total receptive-field sizes matched to object classes, and can run in parallel at test time.The receptive field should cover an object’s bounding box without being excessively large; one network for multiple classes is left for future work.
  • III. METHODS: Voting convolutions apply flipped filter weights only at non-zero locations, producing equivalent sparse convolution outputs while avoiding dense evaluation of empty locations.The voting procedure is extended from a sparse 2D illustration to 3D inputs with several feature maps, followed by ReLU and repeatable CNN stacking.

A. Sparse Convolutions via Voting

The section replaces wasteful dense 3D convolutions with feature-centric voting that computes only around non-zero input features while preserving convolutional equivalence. Non-positive outputs and non-positive biases maintain sparsity in the resulting feature grid.

  • A. Sparse Convolutions via Voting: Dense 3D convolution wastes computation on multiplications by zero, motivating feature-centric voting over non-zero input features.The extra spatial dimension makes dense 3D convolution more expensive than the 2D convolutions underlying image-based CNNs.
  • A. Sparse Convolutions via Voting: Each non-zero input feature vector casts filter-weighted votes to neighboring output cells within the filter’s receptive field, using the spatially flipped kernel.The voting operation is repeated over all non-zero input-cell indices and their corresponding kernel offsets.
  • A. Sparse Convolutions via Voting: A ReLU discards non-positive voting outputs, while non-positive biases are added only to non-empty output cells to preserve sparsity.A positive bias would populate almost every output cell and eliminate the computational advantage of sparse processing.
  • A. Sparse Convolutions via Voting: The sparse scheme applies filters only to occupied input cells rather than the entire grid and is formally equivalent to exhaustive convolution.Dense convolution evaluates all L × M × N locations, whereas voting uses the set of non-zero cell indices.

B. Maintaining Sparsity with ReLUs

Because successive convolutions dilate non-empty regions, Vote3Deep uses ReLUs after sparse convolutions to preserve sparsity while retaining nonlinear representational capacity.

  • B. Maintaining Sparsity with ReLUs: Successive convolutions expand non-empty regions by approximately their filters’ receptive-field sizes, making sparse inputs to every layer a critical assumption for fast voting.The activation function must therefore help maintain sparsity between convolutional layers.
  • B. Maintaining Sparsity with ReLUs: ReLUs are applied after each sparse convolution, as advocated in, producing hidden activations for the subsequent sparse representation.The ReLU operates on the sparse convolution output in each layer.
  • B. Maintaining Sparsity with ReLUs: Only positive features cast votes in the next sparse convolution, because ReLUs discard negative values through thresholding and thereby preserve intermediate sparsity.This thresholding also supports nonlinear function approximation and increases representational capacity.

IV. TRAINING · A. Linear Hinge Loss

The networks are trained as class-specific binary classifiers on fixed-size 3D crops, using periodically mined hard negatives and a linear hinge loss. This loss enforces a maximum-margin separation by pushing scores outside the interval [−1, 1] and can be backpropagated through the CNN.

  • IV. TRAINING: Training uses positive and negative 3D crops whose dimensions match the architecture’s specified receptive-field size.
  • IV. TRAINING: Class-specific networks use a linear hinge loss because it provides a maximum-margin training objective.The networks are binary classifiers distinguishing positive from negative samples.
  • IV. TRAINING: Negative examples are obtained through hard negative mining performed periodically after a fixed number of training epochs.
  • A. Linear Hinge Loss: The hinge loss is defined for detection scores ˆy ∈ R, labels y ∈ {−1, 1}, and network parameters θ.
  • A. Linear Hinge Loss: Table I specifies kernel dimensions for the compared architectures, while Fig. 3 illustrates Model D’s sparse 3D grids and native sparse convolutions.
  • A. Linear Hinge Loss: Scores incur zero loss above 1 for positive samples and below −1 for negative samples, so training drives them outside the margin interval [−1, 1].

B. L1 Sparsity Penalty · V. EXPERIMENTS

The method adds an L1 penalty on intermediate feature activations to discard uninformative features and promote sparsity throughout the CNN, with normalization reducing dependence on input size.

  • B. L1 Sparsity Penalty: The proposed regulariser encourages the network to discard uninformative features and increase sparsity throughout the entire CNN stack.
  • B. L1 Sparsity Penalty: The L1 penalty targets intermediate layers by applying the L1 norm to each feature activation, producing representations that can contain exactly zero values [17].Output-layer sparsity remains tunable through the detection threshold.
  • B. L1 Sparsity Penalty: The L1 loss is normalized by each layer’s feature-map spatial dimensions, making the penalty’s influence less dependent on input size for a fixed parameter setting.

A. Dataset · B. Evaluation · C. Training

The study trains and evaluates 3D point-cloud detectors on KITTI, using calibrated projection and moderate-category AP for official ranking. Training combines balanced crops, discretization-aware augmentation, hard-negative mining, and specified optimization and model-selection procedures.

  • A. Dataset: KITTI provides synchronised stereo-camera and lidar frames, but the models use only 3D point clouds; evaluation focuses on cars, pedestrians, and cyclists.The labelled training set contains 7,481 frames split 80%/20% for training and validation, while the test set contains 7,518 frames with hidden labels.
  • B. Evaluation: Official KITTI evaluation projects 3D detections into calibrated 2D image space, discards out-of-image detections, and ranks models by moderate-category average precision.Difficulty levels reflect bounding-box size, truncation, and occlusion; the hard category includes the most positives, while the most difficult examples are ignored for moderate and easy categories.
  • C. Training: Training uses initially balanced positive and negative 3D crops, with negatives sampled from locations that do not overlap positive examples.The networks are trained on 3D crops extracted from the point-cloud training data.
  • C. Training: Sub-cell translations and sub-bin rotations augment front-facing positives to improve generalisation despite spatial and angular discretisation.Translations are smaller than grid-cell dimensions, and rotations are smaller than angular-bin resolution.
  • C. Training: Every ten epochs, hard-negative mining adds the ten highest-scoring false positives per training frame after scanning full point clouds with the current model.This iteratively expands the training set with difficult negative examples.
  • C. Training: On internal validation, two- and three-layer nonlinear models outperform the linear baseline for all three classes, while increasing hidden-layer filters yields only incremental gains.Figure 4 reports average precision at moderate difficulty for the architecture in Table I.
  • C. Training: Networks train for 100 epochs using SGD with momentum 0.9, learning rate 10^-3, batch size 16, and L2 decay 10^-4; the highest-validation-AP epoch selects comparison and submission models.Weights are initialised as in [18]. Timing instead uses fully trained 100-epoch models because validation-AP selection favours lower intermediate sparsity.
  • C. Training: A custom C++ library supports training and testing, with the largest models requiring about three days on a 16-core cluster CPU node.Each batch example is processed in a separate thread; the official submission uses Model B for cars and Model D for pedestrians and cyclists, each with eight hidden-layer filters and no sparsity penalty.

D. Model Comparison

The section compares five sparse 3D CNN architectures on the validation set to assess the trade-off between model capacity and detection performance. Multi-layer nonlinear models outperform the linear baseline, while added hidden filters or altered kernel receptive fields provide limited further gains.

  • D. Model Comparison: Five architectures with up to three layers and different filter configurations are benchmarked on the validation set to examine capacity–performance trade-offs.Fast detection is important for robotics because larger, more expressive models incur higher computational costs and run more slowly.
  • D. Model Comparison: Nonlinear, multi-layer networks clearly outperform the linear baseline comparable to, showing that greater model complexity improves point-cloud object detection.This result is reported in Fig. 4.
  • D. Model Comparison: Increasing hidden-layer filter counts yields only moderate gains beyond the large improvement achieved with eight filters.
  • D. Model Comparison: Changing kernel receptive fields while keeping the network’s total receptive field fixed does not indicate a significant performance improvement.The architectures use small 3×3×3 and 5×5×5 lower-layer kernels, followed by ReLU, with outputs computed by class-specific convolutional filters.
  • D. Model Comparison: The limited benefit of larger models may reflect insufficient regularisation or the ability of relatively small models to capture most task-relevant variation in interpretable 3D inputs.

E. Test Results

Vote3Deep achieves state-of-the-art point-cloud detection across all classes and difficulty levels, including strong gains for cyclists. It also outperforms multimodal methods in most cases while operating at considerably faster detection speed, although CPU execution remains slower than.

  • E. Test Results: Increasing filter count or kernel size does not significantly improve accuracy but reduces detection speed, motivating eight 3×3×3 filters per hidden layer.The test submission uses this configuration to balance accuracy and speed.
  • E. Test Results: Vote3Deep establishes state-of-the-art performance for all three object classes and difficulty levels in point-cloud detection.Cyclist performance improves by almost 40% in the easy test case and more than doubles AP in the other two test cases.
  • E. Test Results: On CPU, Vote3Deep is about two times slower than and almost as fast as GPU-accelerated [7].The authors expect GPU sparse-convolution implementations to improve detection speed.
  • E. Test Results: Despite using only point clouds, Vote3Deep outperforms multimodal methods, in most test cases and is considerably faster.It is only slightly worse in the remaining cases.
  • E. Test Results: Vote3Deep achieves the highest AP for all three object classes on the hard test cases.These cases contain the largest number of positive ground-truth objects.
  • E. Test Results: Relatively shallow networks trained without recent vision tricks are expressive enough to achieve significant performance gains, especially for cyclist detection.The authors conjecture that cyclists’ distinctive 3D shape helps compensate for their fewer training examples.

F. Timing and Sparsity

The L1 sparsity penalty improves detection speed, especially for cars, with effects on accuracy varying by object class and training behavior. Larger penalties can collapse pedestrian and cyclist activations, while smaller penalties improve their speed and may regularize accuracy.

  • Timing and Sparsity: Pedestrians detect fastest because their networks have smaller receptive fields, and two-layer car Model B is faster than three-layer cyclist Model D.This speed ordering holds independently of whether sparsity penalties are used.
  • Timing and Sparsity: About 15% faster pedestrian and cyclist detection occurs with smaller penalties, whereas a 10^-1 penalty collapses their activations to zero during training.The largest usable penalty is more effective for cars than for the other classes.
  • Timing and Sparsity: Almost 40% faster car detection results from L1 sparsity with negligible accuracy loss, while Table IV evaluates speed and average precision across penalty values.Detection times are measured over 200 validation frames.
  • Timing and Sparsity: Cars benefit most from sparsity because fewer intermediate layers and a larger receptive field may produce sparser but informative representations.The authors present this mechanism as a conjecture explaining why larger penalties can be applied to cars.
  • Timing and Sparsity: The authors caution that the stochastic training algorithm warrants a more rigorous investigation of the speed gains, left for future work.The current results establish a beneficial effect on detection speed but not its full statistical characterization.

VI. CONCLUSIONS

The work achieves fast object detection in point clouds using CNNs with voting-based sparse convolutional layers, establishing a new state of the art on KITTI and outperforming multimodal methods in most test cases.

  • VI. CONCLUSIONS: The method establishes a new state of the art on the KITTI benchmark for detecting objects in point clouds.It performs object detection at fast speeds using CNNs constructed from sparse convolutional layers.
  • VI. CONCLUSIONS: Its CNNs use sparse convolutional layers based on the voting scheme introduced in.These networks learn hierarchical representations and non-linear decision boundaries.
  • VI. CONCLUSIONS: Vote3Deep outperforms methods using both point clouds and images in most test cases.
Loading 1609.06666v2…