Source-linked AI summary
RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection
Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, Dragomir Anguelov
TL;DR
Longer-range LiDAR creates a need for accurate, efficient 3D detection over larger regions. RSN combines high-resolution range-image foreground segmentation with sparse 3D convolutions on selected points, and experiments report state-of-the-art accuracy and efficiency on WOD. Its scope includes limitations of voxel representations, whose complexity and memory consumption scale quadratically or cubically with range.
Problem
Longer-range LiDAR sensing places strong demands on efficient and accurate 3D detection, while voxel representations face range-dependent complexity, memory, and resolution constraints.
Method
RSN extracts range-image features, segments foreground points, and applies sparse convolutions to the selected points for 3D box detection.
Results
Experiments on WOD demonstrate state-of-the-art accuracy and efficiency for vehicle and pedestrian detection, while an internal dataset demonstrates scalability for long-range detection.
Takeaways & Limitations
RSN combines dense range-image processing with sparse 3D processing to support efficient and accurate long-range LiDAR detection.
Takeaways & Limitations
Voxel-based representations remain constrained at long range because their complexity and memory consumption scale quadratically or cubically with range.
Abstract
from arXiv · showhide
The detection of 3D objects from LiDAR data is a critical component in most autonomous driving systems. Safe, high speed driving needs larger detection ranges, which are enabled by new LiDARs. These larger detection ranges require more efficient and accurate detection models. Towards this goal, we propose Range Sparse Net (RSN), a simple, efficient, and accurate 3D object detector in order to tackle real time 3D object detection in this extended detection regime. RSN predicts foreground points from range images and applies sparse convolutions on the selected foreground points to detect objects. The lightweight 2D convolutions on dense range images results in significantly fewer selected foreground points, thus enabling the later sparse convolutions in RSN to efficiently operate. Combining features from the range image further enhance detection accuracy. RSN runs at more than 60 frames per second on a 150m x 150m detection region on Waymo Open Dataset (WOD) while being more accurate than previously published detectors. As of 11/2020, RSN is ranked first in the WOD leaderboard based on the APH/LEVEL 1 metrics for LiDAR-based pedestrian and vehicle detection, while being several times faster than alternatives.
1. Introduction
Longer-range LiDAR sensing increases the need for accurate and efficient 3D detection. RSN combines dense range-image processing with sparse 3D convolutions on selected foreground points to improve both efficiency and accuracy.
- Up to 250m and 300m sensing ranges create strong demands for efficient and accurate 3D detection methods.
- Grid methods become inefficient on large detection grids, while sparse convolutions scale better but remain slow when applied to all points.
- RSN applies lightweight 2D convolutions to range images, segments foreground points, then uses sparse convolutions and learned range features to regress 3D boxes.
- RSN transfers information from high-resolution range images to sparse 3D processing, preserving resolution and restricting later computation to selected foreground points.
- The model uses foreground segmentation to reduce downstream computation and eliminates non-maximum suppression through its sparse detection design.
- On WOD, RSN reports state-of-the-art vehicle and pedestrian accuracy and efficiency, with ablations examining features, selection thresholds, and training choices.
2. Related Work
LiDAR 3D detection uses voxel grids, point sets, range images, and hybrid representations, each balancing scalability, resolution, and detection quality. Prior approaches expose limitations that motivate alternative representations and architectures.
- Voxel methods divide 3D space into voxels and apply feature extraction followed by dense or sparse convolutions.
- Voxel grids use constant voxel sizes and face complexity, memory, and resolution constraints as detection range increases.
- Point-set methods can scale better with range but generally lag behind voxel methods in detection quality and require additional processing stages.
- Range-image methods process native dense point representations with 2D convolutions, while related systems add dilation or second-stage refinement.
- Two-stage detectors classify sparse proposed regions, whereas single-stage methods directly predict objects using architectures adapted to point-cloud representations.
3. Range Sparse Net
RSN combines dense range-image processing with sparse 3D processing: it segments likely foreground points, extracts sparse features, and regresses boxes efficiently. Its temporal extension aligns selected points from multiple frames before shared sparse processing.
- Architecture: RSN processes raw LiDAR range images with a 2D convolution backbone, segments foreground points, voxelizes selected points, and applies sparse convolutions before box regression.Foreground segmentation emphasizes recall because later stages can remove false positives, whereas missed foreground points cannot be recovered.
- Range Image Feature Extraction: The range-image feature extractor uses a lightweight U-Net to learn semantic features and perform foreground segmentation from range, intensity, and elongation channels.The input channels are clipped and rescaled to [0, 1].
- Sparse Point Feature Extraction: Selected foreground points are dynamically voxelized and processed by 2D or 3D sparse convolution networks, depending on the voxelization style.The voxel features can include point offsets from voxel means and centers, covariance, and optional PointNet processing.
- Box Regression: The modified CenterNet head predicts heatmaps and box parameters from sparse voxel features, using local maxima above a threshold to select boxes.The heatmap is based on distances between feature-map points and centers of containing ground-truth boxes, while box parameters include center offsets, dimensions, and heading.
- Temporal Fusion: Temporal RSN applies range-image feature extraction and foreground selection to multiple frames, transforms selected points into the latest frame, and appends frame-time information before sparse processing.The design avoids directly stacking moving-frame range images and avoids mixing points from different frames during voxelization.
4. Experiments
Experiments on Waymo and an internal long-range dataset evaluate RSN’s accuracy, efficiency, ablations, and scalability. RSN improves accuracy–latency trade-offs for vehicles and pedestrians, with foreground selection and range-image features supporting its performance.
- Experimental setup: RSN is evaluated primarily on Waymo Open Dataset using official BEV/3D AP and APH metrics at L1 and L2 difficulty levels.Latency is measured on Tesla V100 GPUs, except PVRCNN results obtained on Titan RTX.
- Vehicle detection: The efficient CarS 1f model is at least 3x more efficient than baselines while exceeding all single-stage methods in accuracy.Its temporal version further improves accuracy at negligible additional inference cost.
- Vehicle detection: CarXL 3f significantly outperforms published methods, including PVRCNN-WOD, the most accurate LiDAR-only challenge submission.RSN ensemble results also outperform the PVRCNN WOD challenge submission.
- Pedestrian detection: Pedestrian detection shows larger efficiency and accuracy gains: PedS 1f exceeds all published single-stage baselines, while PedL 3f surpasses PVRCNN-WOD with higher efficiency.The gains are attributed in the experiments to fewer pedestrian foreground points and high-resolution range-image features.
- Foreground selection: A foreground threshold exists that significantly reduces latency with negligible accuracy impact, provided selection maintains high recall.Reported foreground precision/recall is 77.5%/99.6% for CarS 3f and 15.3%/97.6% for PedS 3f.
- Ablation studies: Range-image features improve both foreground segmentation efficiency and detection accuracy, with larger accuracy improvement for pedestrians and effects especially impacting long-range detection.End-to-end gradients from SPFE to RIFE affect accuracy, while disabling heatmap normalization reduces detection accuracy.
- Long-range scalability: On an internal long-range dataset with a 500m × 500m detection range, RSN scales with good accuracy and efficiency, while PointPillars exceeds Tesla V100 memory capacity.The experiments report that foreground sampling and range-image features remain effective at the larger range.
A. Additional details on SPFE
SPFE uses different sparse-convolution configurations across pedestrian and vehicle models, with larger models switching to 3D sparse convolutions. The architectures are detailed in Figure 8 for CarS, PedS, and CarXL.
- PedS, PedL, CarS, and CarL use 2D sparse convolutions with 96 channels throughout.
- CarXL uses 3D sparse convolutions with 64 channels and omits PointNet within each 3D voxel.
- Figure 8 presents SPFE architectures for CarS, PedS, and CarXL.
B. More Details on Temporal Fusion
Temporal RSN processes multiple frames through shared-weight segmentation branches, augments and merges their foreground points, then applies the single-frame detection pipeline.
- Temporal RSN duplicates RIFE and foreground point selection for each temporal frame, using shared weights across branches.
- Each selected point is augmented with voxel statistics, its time difference, and features learned from the corresponding RIFE stage.
- Augmented points from all frames are merged before normal voxelization and PointNet processing.
- During inference, only the latest frame is computed because earlier time-step results are reused.
- A k + 1-frame temporal model is trained by regrouping input sequences with frame i as the target output.
C. Ensemble Details
The reported ensemble combines models trained on disjoint data subsets with predictions from multiple point-cloud augmentations, then aggregates the resulting 3D boxes.
- Five model copies are trained, each on a disjoint subset containing 80% of the original training data.
- Each model predicts boxes under five random point-cloud augmentations, producing 25 result sets per sample.
- The ensemble uses data-level and test-time augmentation-based voting schemes.
- Figure 9 depicts the expanded temporal RSN architecture before SPFE.
- Box aggregation extends the cited Weighted Boxes Fusion strategy to 3D boxes with yaw heading.