Source-linked AI summary
StarNet: Targeted Computation for Object Detection in Point Clouds
Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, Yin Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens, Vijay Vasudevan
TL;DR
LiDAR provides high-resolution 3D information for self-driving cars, but prior point-cloud detection methods have reused camera-oriented architectures. StarNet uses sparse sampling and local point-based processing, achieving competitive or superior benchmark performance while enabling targeted computation. Prior-frame detections can guide sampling for improved mAP.
Problem
LiDAR provides high-resolution 3D data, but leading point-cloud detectors have repurposed camera-based architectures despite LiDAR’s sparsity and changing scene priorities.
Source: §1 · p. 1Method
StarNet samples locations in the point cloud, processes local neighboring points independently, and predicts region classes and bounding boxes from native point features.
Source: §3 · p. 3, §3.1 · p. 4Results
StarNet is competitive with state-of-the-art systems on KITTI and can outperform a competitive convolutional baseline on Waymo; prior-frame targeting improves mAP by up to 40% relative.
Source: §5 · p. 8Takeaways & Limitations
A single trained model can vary its inference cost and use prior-frame detections to target current-frame computation without additional computational cost.
Source: §5 · p. 8The work focuses on simple sampling methods for proposals; more expensive or learned proposal methods may further improve the system.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Detecting objects from LiDAR point clouds is an important component of self-driving car technology as LiDAR provides high resolution spatial information. Previous work on point-cloud 3D object detection has re-purposed convolutional approaches from traditional camera imagery. In this work, we present an object detection system called StarNet designed specifically to take advantage of the sparse and 3D nature of point cloud data. StarNet is entirely point-based, uses no global information, has data dependent anchors, and uses sampling instead of learned region proposals. We demonstrate how this design leads to competitive or superior performance on the large Waymo Open Dataset and the KITTI detection dataset, as compared to convolutional baselines. In particular, we show how our detector can outperform a competitive baseline on Pedestrian detection on the Waymo Open Dataset by more than 7 absolute mAP while being more computationally efficient. We show how our redesign---namely using only local information and using sampling instead of learned proposals---leads to a significantly more flexible and adaptable system: we demonstrate how we can vary the computational cost of a single trained StarNet without retraining, and how we can target proposals towards areas of interest with priors and heuristics. Finally, we show how our design allows for incorporating temporal context by using detections from previous frames to target computation of the detector, which leads to further improvements in performance without additional computational cost.
1. Introduction
LiDAR provides accurate 3D data for autonomous driving, but point-cloud detectors have largely reused camera-oriented designs despite differences in sparsity and scene priorities. StarNet instead uses sampled, local point processing to target computation flexibly.
1. Introduction
- LiDAR’s sparse 3D data and shifting scene priorities differ from camera imagery, yet leading 3D detectors have repurposed camera-based architectures.The paper frames its detector as a redesign for LiDAR and self-driving-car perception.Source: §1 · pp. 1–2
- Sampling schemes with zero learned parameters generate region proposals while matching the point-cloud scene distribution and exploiting its sparsity.StarNet processes proposed regions independently and uses native point-cloud features to classify regions and regress box locations.Source: §1 · p. 2
- StarNet is as accurate as the state of the art at lower inference cost, and more accurate at similar inference costs.Its sampling exploits point-cloud sparsity, and a single trained model can vary proposal and point counts at inference.Source: §1 · p. 2
- A single trained pedestrian model can exceed a baseline convolutional model by ∼48% at similar computational demand, or match its performance at ∼20% of the computational demand.Source: §1 · p. 2
- Using prior-frame detections to target current-frame computation improves detection mAP scores by ∼40%.Source: §1 · p. 2
2. Background
Image detectors trade computational cost against prediction performance, while point clouds are unordered and vary in size. Existing point-cloud detectors adapt image-based methods in different ways, with fixed grids or dense perspective views imposing specific constraints.
2.1. Object detection in images
- Two-stage image detectors propose candidate locations and then classify them, but dense CNN-based proposal generation and per-proposal processing are computationally demanding.One-stage detectors reduce inference time but dense convolutions treat all locations equally, limiting adaptation to scene or latency requirements.Source: §2.1 · p. 2
2.2. Point cloud featurization
- Point clouds are unordered and can contain different numbers of points; ideal learned representations are permutation invariant and agnostic to point count.Source: §2.2 · pp. 2–3
2.3. Object detection with point clouds
- Voxel-based methods enable CNN detection, but PointPillars’ fixed grid cannot adapt to a scene’s point distribution and becomes notably more costly on larger scenes.VoxelNet partitions 3D space and encodes points within each partition; PointPillars builds grid-based per-pillar features for classification and regression.Source: §2.3 · p. 3
- LaserNet applies convolutions to dense LiDAR range images, where perspective effects produce large variation in object scale.Source: §2.3 · p. 3
- PointRCNN uses an expensive per-point segmentation network for proposals before a second stage classifies and localizes cropped features; other methods use paired camera images for learned proposals.The camera-paired methods aim to improve predictive performance by seeding proposals to maximize recall.Source: §2.3 · p. 3
3. Methods
StarNet uses data-dependent sampling to propose regions and processes each local point cloud independently. A point-based featurizer produces features for classification and box regression, with anchors placed relative to sampled centers.
3. Methods
- StarNet samples locations in the point cloud, featurizes nearby points, and independently classifies each region and regresses its bounding box using local information.Independent proposals can be processed in parallel, and contextual information can inform proposal importance.Source: §3 · p. 3
3.1. Center selection
- Proposals use sampled point coordinates rather than fixed-grid locations; random sampling favors dense regions, while farthest point sampling (FPS) seeks even spatial coverage.Sampling excludes points outside a dataset-specific z range, but those points remain available later; Figure 5 plots proposal coverage against sample count on KITTI and Waymo.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
3.2. Featurizing local point clouds
- Each proposal gathers K points within radius R, recenters them, and featurizes them with stacked StarNet blocks using max aggregation and mean readouts.The five-layer featurizer outputs a 384-dimensional feature and can accept varying numbers of input points.Source: §3.2 · pp. 4–5, Figure 4 · p. 4
3.3. Constructing final predictions from bounding box proposals.
- Each cell center receives a data-dependent G × G grid of anchor offsets, whose features produce classification and bounding-box regression predictions.Regression predicts location, size, and heading residuals; Figure 6 evaluates one model trained with 1024 proposals using 64 to 1024 proposals.Source: §3.3 · p. 5, Figure 6 · p. 5
- Anchors are labeled foreground above IoU 0.6, background below 0.45, or ignored between those thresholds; final predictions use oriented multi-class NMS.If an object has no foreground anchor, the highest-matching eligible anchor is force-matched; classification uses focal cross-entropy and regression uses smoothed-L1 losses.Source: §3.3 · p. 5
4. Results
StarNet is competitive on KITTI and outperforms PointPillars on Waymo pedestrian detection. Its sampling, inference-time flexibility, and use of previous-frame detections shape the reported results.
4.1. Sampling strategies for point cloud detections
- Farthest point sampling provides uniformly better proposal coverage than random sampling at a fixed proposal count; KITTI coverage exceeds 98% within 256 samples.Coverage counts annotated objects with 5+ points whose sampled anchor boxes have IoU > 0.5; farthest point sampling is used for the remaining experiments.Source: §4.1 · p. 6
4.2. KITTI Dataset
- On KITTI, standard point-cloud and bounding-box augmentation produced gains of up to +18.0, +16.9, and +30.5 mAP for car, pedestrian, and cyclist, respectively; StarNet was competitive on the 3-D test benchmark.All detection results and comparisons were based only on LIDAR data; mAP used IoU thresholds of 0.7 for vehicles and 0.5 for cyclists and pedestrians.Source: §4.2 · p. 6, Table 1 · p. 6
4.3. Waymo Open Dataset
- Higher PointPillars spatial resolution improves precision for pedestrians and vehicles, but its computational cost grows quadratically.The comparison evaluates five separately trained PointPillars models, while StarNet’s curves use one set of saved weights.Source: §4.3 · p. 7, Figure 7 · p. 7
- At a similar ~100GFlops budget, StarNet-128 with 1024 centers gains ~48% relative pedestrian predictive performance over the baseline; it can match the most accurate Pillars model at ~20% of its computational budget.The StarNet performance-cost curves come from a single trained model, with inference varied across proposal counts and points per region.Source: §4.3 · p. 7, Figure 7 · p. 7
- 7.8 mAP and 12.6 mAPH: StarNet exceeds PointPillars on Waymo pedestrian detection with directional loss; with directionless loss, it exceeds the baseline by 10.1 mAP.These are Waymo Test set results for LEVEL 1 objects with at least five points.Source: Table 2 · p. 7, §4.3 · p. 8
4.4. Targeting computation with temporal context
- Using 192 prior-frame detections among 384 centers improves validation pedestrian mAP by nearly 17 absolute points, or 40%; 32 prior detections add over 10 points.With 1024 total centers, using the best 512 previous detections improves mAP by about 3 points and test mAP by 2 points.Source: §4.4 · p. 8, Table 4 · p. 8
5. Discussion
StarNet operates on native point clouds and is competitive on KITTI and Waymo. Its targeted computation supports temporal context and inference-time trade-offs, while the authors identify several avenues for improving fidelity.
5. Discussion
- StarNet is competitive on KITTI and can outperform a convolutional baseline on the large-scale Waymo Open Dataset.The system operates on native point cloud data and is designed to target computation across different computational priorities.Source: §5 · p. 8
- A single trained pedestrian model can gain ~48% predictive performance over a baseline at similar FLOPS, or match its performance at ~20% of the FLOPS.The paper also reports up to a 40% relative mAP improvement when prior-frame detections guide current-frame computation.Source: §5 · p. 8
- The authors identify camera fusion, map-guided targeting, restored proposal-level global context, and more expensive or learned proposal methods as ways to improve system fidelity.They also express interest in studying the system’s suitability for object tracking.Source: §5 · pp. 8–9
A. Full Waymo Open Dataset Results
The appendix identifies a Waymo Open Dataset test-set results table comparing StarNet with a PointPillars baseline.
A. Full Waymo Open Dataset Results
- Table 5 reports Waymo Open Dataset test results for StarNet versus a PointPillars baseline.The table covers LEVEL 1 and LEVEL 2 mAP or heading-weighted mAP (mAPH).Source: Table 5 · p. 12
- The reported metrics are mean average precision (mAP) and heading-weighted mean average precision (mAPH).Source: Table 5 · p. 12
- The caption identifies the comparison scope and metrics but does not provide result values.Source: Table 5 · p. 12